Repository: spark Updated Branches: refs/heads/branch-2.0 b82abde06 -> 3c3865d0b
[SPARK-15811][SQL] fix the Python UDF in Scala 2.10 ## What changes were proposed in this pull request? Iterator can't be serialized in Scala 2.10, we should force it into a array to make sure that . ## How was this patch tested? Build with Scala 2.10 and ran all the Python unit tests manually (will be covered by a jenkins build). Author: Davies Liu <dav...@databricks.com> Closes #13717 from davies/fix_udf_210. (cherry picked from commit ef43b4ed87894982678fcc6f2c61cf1487ee9e14) Signed-off-by: Reynold Xin <r...@databricks.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/3c3865d0 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/3c3865d0 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/3c3865d0 Branch: refs/heads/branch-2.0 Commit: 3c3865d0b42413435f8cc9044da842cfb4458afc Parents: b82abde Author: Davies Liu <dav...@databricks.com> Authored: Fri Jun 17 00:34:33 2016 -0700 Committer: Reynold Xin <r...@databricks.com> Committed: Fri Jun 17 00:34:38 2016 -0700 ---------------------------------------------------------------------- .../org/apache/spark/sql/execution/python/ExtractPythonUDFs.scala | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/3c3865d0/sql/core/src/main/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFs.scala ---------------------------------------------------------------------- diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFs.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFs.scala index 668470e..87583c8 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFs.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFs.scala @@ -132,7 +132,7 @@ private[spark] object ExtractPythonUDFs extends Rule[SparkPlan] { val validUdfs = udfs.filter { case udf => // Check to make sure that the UDF can be evaluated with only the input of this child. udf.references.subsetOf(child.outputSet) - } + }.toArray // Turn it into an array since iterators cannot be serialized in Scala 2.10 if (validUdfs.nonEmpty) { val resultAttrs = udfs.zipWithIndex.map { case (u, i) => AttributeReference(s"pythonUDF$i", u.dataType)() --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
