[ https://issues.apache.org/jira/browse/SPARK-16918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15411287#comment-15411287 ]
Dongjoon Hyun commented on SPARK-16918: --------------------------------------- Hi, [~phihungle]. I met the same error in bot 1.6.1 and 1.6.2 with your sample code. And, the problem seems to be resolved in Apache Spark 2.0 release. {code} >>> N = 101 >>> df = sqlContext.createDataFrame([{'value': 0}]) >>> udf_columns = [pyspark.sql.functions.udf(lambda x: 0)('value') for _ in >>> range(N)] >>> df.select(udf_columns).take(1) [Row(<lambda>(value)=u'0', <lambda>(value)=u'0', ... {code} > Weird error when selecting more than 100 spark udf columns > ---------------------------------------------------------- > > Key: SPARK-16918 > URL: https://issues.apache.org/jira/browse/SPARK-16918 > Project: Spark > Issue Type: Bug > Components: PySpark, SQL > Affects Versions: 1.6.1 > Reporter: Phi Hung LE > > Starting with a simple spark dataframe with only one value, I create N simple > udf columns. > {code} > N = 100 > df = sqlContext.createDataFrame([{'value': 0}]) > udf_columns = [pyspark.sql.functions.udf(lambda x: 0)('value') for _ in > range(N)] > df.select(udf_columns).take(1) > {code} > For N <= 100 this code works perfectly. But as soon as N >= 101, I found the > following error > {code} > Py4JJavaError: An error occurred while calling > z:org.apache.spark.sql.execution.EvaluatePython.takeAndServe. > : org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 > in stage 34.0 failed 1 times, most recent failure: Lost task 0.0 in stage > 34.0 (TID 50, localhost): java.lang.UnsupportedOperationException: Cannot > evaluate expression: PythonUDF#<lambda>(input[0, LongType]) > at > org.apache.spark.sql.catalyst.expressions.Unevaluable$class.genCode(Expression.scala:239) > at org.apache.spark.sql.execution.PythonUDF.genCode(python.scala:44) > at > org.apache.spark.sql.catalyst.expressions.Expression$$anonfun$gen$2.apply(Expression. > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org