[ https://issues.apache.org/jira/browse/SPARK-15062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael Armbrust resolved SPARK-15062. -------------------------------------- Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request 12849 [https://github.com/apache/spark/pull/12849] > Show on DataFrame causes OutOfMemoryError, NegativeArraySizeException or > segfault > ---------------------------------------------------------------------------------- > > Key: SPARK-15062 > URL: https://issues.apache.org/jira/browse/SPARK-15062 > Project: Spark > Issue Type: Bug > Components: SQL > Environment: spark-2.0.0-SNAPSHOT using commit hash > 90787de864b58a1079c23e6581381ca8ffe7685f and Java 1.7.0_67 > Reporter: koert kuipers > Priority: Blocker > Fix For: 2.0.0 > > > {noformat} > scala> val dfComplicated = sc.parallelize(List((Map("1" -> "a"), List("b", > "c")), (Map("2" -> "b"), List("d", "e")))).toDF > ... > dfComplicated: org.apache.spark.sql.DataFrame = [_1: map<string,string>, _2: > array<string>] > scala> dfComplicated.show > java.lang.OutOfMemoryError: Java heap space > at org.apache.spark.unsafe.types.UTF8String.getBytes(UTF8String.java:229) > at org.apache.spark.unsafe.types.UTF8String.toString(UTF8String.java:821) > at > org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificSafeProjection.apply(Unknown > Source) > at > org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.fromRow(ExpressionEncoder.scala:241) > at > org.apache.spark.sql.Dataset$$anonfun$org$apache$spark$sql$Dataset$$execute$1$1$$anonfun$apply$13.apply(Dataset.scala:2121) > at > org.apache.spark.sql.Dataset$$anonfun$org$apache$spark$sql$Dataset$$execute$1$1$$anonfun$apply$13.apply(Dataset.scala:2121) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) > at > scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) > at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186) > at scala.collection.TraversableLike$class.map(TraversableLike.scala:234) > at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:186) > at > org.apache.spark.sql.Dataset$$anonfun$org$apache$spark$sql$Dataset$$execute$1$1.apply(Dataset.scala:2121) > at > org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:54) > at org.apache.spark.sql.Dataset.withNewExecutionId(Dataset.scala:2408) > at > org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$execute$1(Dataset.scala:2120) > at > org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$collect(Dataset.scala:2127) > at org.apache.spark.sql.Dataset$$anonfun$head$1.apply(Dataset.scala:1861) > at org.apache.spark.sql.Dataset$$anonfun$head$1.apply(Dataset.scala:1860) > at org.apache.spark.sql.Dataset.withTypedCallback(Dataset.scala:2438) > at org.apache.spark.sql.Dataset.head(Dataset.scala:1860) > at org.apache.spark.sql.Dataset.take(Dataset.scala:2077) > at org.apache.spark.sql.Dataset.showString(Dataset.scala:238) > at org.apache.spark.sql.Dataset.show(Dataset.scala:529) > at org.apache.spark.sql.Dataset.show(Dataset.scala:489) > at org.apache.spark.sql.Dataset.show(Dataset.scala:498) > ... 6 elided > scala> > {noformat} > By increasing memory to 8G one will instead get a NegativeArraySizeException > or a segfault. > See here for original discussion: > http://apache-spark-developers-list.1001551.n3.nabble.com/spark-2-segfault-td17381.html -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org