[ https://issues.apache.org/jira/browse/SPARK-11498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15015060#comment-15015060 ]
Antony Mayi commented on SPARK-11498: ------------------------------------- can't reproduce on 1.5.2, seems to be fixed. > TreeNodeException under very special condition > ---------------------------------------------- > > Key: SPARK-11498 > URL: https://issues.apache.org/jira/browse/SPARK-11498 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 1.5.1 > Environment: spark 1.5.1 (official binary distribution) running on > hadoop yarn 2.6 with parquet 1.5.0 (both from cdh5.4.8) > Reporter: Antony Mayi > > {code:title=/tmp/bug.py} > from pyspark import SparkContext > from pyspark.sql import SQLContext, Row > sc = SparkContext() > sqlc = SQLContext(sc) > R = Row('id', 'foo') > r = sqlc.createDataFrame(sc.parallelize([R('abc', 'foo')])) > q = sqlc.createDataFrame(sc.parallelize([R('aaaaaaaaaaaaaaaaaaaaaaaa', > 'bar')])) > q.write.parquet('/tmp/1.parq') > q = sqlc.read.parquet('/tmp/1.parq') > j = r.join(q, r.id == q.id) > print j.count() > {code} > {noformat} > [user@sandbox test]$ spark-submit --executor-memory=32g /tmp/bug.py > [user@sandbox test]$ hadoop fs -rmr /tmp/1.parq > {noformat} > {noformat} > 15/11/04 04:28:38 INFO codegen.GenerateUnsafeProjection: Code generated in > 119.90324 ms > Traceback (most recent call last): > File "/tmp/bug.py", line 13, in <module> > print j.count() > File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/dataframe.py", line > 268, in count > File "/usr/lib/spark/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py", > line 538, in __call__ > File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 36, > in deco > File "/usr/lib/spark/python/lib/py4j-0.8.2.1-src.zip/py4j/protocol.py", > line 300, in get_return_value > py4j.protocol.Py4JJavaError: An error occurred while calling o148.count. > : org.apache.spark.sql.catalyst.errors.package$TreeNodeException: execute, > tree: > TungstenAggregate(key=[], functions=[(count(1),mode=Final,isDistinct=false)], > output=[count#10L]) > TungstenExchange SinglePartition > TungstenAggregate(key=[], > functions=[(count(1),mode=Partial,isDistinct=false)], > output=[currentCount#13L]) > TungstenProject > BroadcastHashJoin [id#0], [id#8], BuildRight > TungstenProject [id#0] > Scan PhysicalRDD[id#0,foo#1] > ConvertToUnsafe > Scan ParquetRelation[hdfs:///tmp/1.parq][id#8] > at > org.apache.spark.sql.catalyst.errors.package$.attachTree(package.scala:49) > at > org.apache.spark.sql.execution.aggregate.TungstenAggregate.doExecute(TungstenAggregate.scala:69) > at > org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.apply(SparkPlan.scala:140) > at > org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.apply(SparkPlan.scala:138) > at > org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147) > at > org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:138) > at > org.apache.spark.sql.execution.SparkPlan.executeCollect(SparkPlan.scala:174) > at > org.apache.spark.sql.DataFrame$$anonfun$collect$1.apply(DataFrame.scala:1385) > at > org.apache.spark.sql.DataFrame$$anonfun$collect$1.apply(DataFrame.scala:1385) > at > org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:56) > at > org.apache.spark.sql.DataFrame.withNewExecutionId(DataFrame.scala:1903) > at org.apache.spark.sql.DataFrame.collect(DataFrame.scala:1384) > at org.apache.spark.sql.DataFrame.count(DataFrame.scala:1402) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:497) > at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231) > at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379) > at py4j.Gateway.invoke(Gateway.java:259) > at > py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133) > at py4j.commands.CallCommand.execute(CallCommand.java:79) > at py4j.GatewayConnection.run(GatewayConnection.java:207) > at java.lang.Thread.run(Thread.java:745) > {noformat} > Note this happens only under following condition: > # executor memory >= 32GB (doesn't fail with up to 31 GB) > # the ID in the q dataframe has exactly 24 chars (doesn't fail with less or > more then 24 chars) > # q is read from parquet -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org