Repository: spark Updated Branches: refs/heads/branch-1.6 d7223bb9f -> b7acc1b71
[SPARK-16173] [SQL] Can't join describe() of DataFrame in Scala 2.10 ## What changes were proposed in this pull request? This PR fixes `DataFrame.describe()` by forcing materialization to make the `Seq` serializable. Currently, `describe()` of `DataFrame` throws `Task not serializable` Spark exceptions when joining in Scala 2.10. ## How was this patch tested? Manual. (After building with Scala 2.10, test on bin/spark-shell and bin/pyspark.) Author: Dongjoon Hyun <dongj...@apache.org> Closes #13902 from dongjoon-hyun/SPARK-16173-branch-1.6. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/b7acc1b7 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/b7acc1b7 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/b7acc1b7 Branch: refs/heads/branch-1.6 Commit: b7acc1b71c5d4b163a7451e8c6430afe920a04e0 Parents: d7223bb Author: Dongjoon Hyun <dongj...@apache.org> Authored: Fri Jun 24 22:30:52 2016 -0700 Committer: Davies Liu <davies....@gmail.com> Committed: Fri Jun 24 22:30:52 2016 -0700 ---------------------------------------------------------------------- sql/core/src/main/scala/org/apache/spark/sql/DataFrame.scala | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/b7acc1b7/sql/core/src/main/scala/org/apache/spark/sql/DataFrame.scala ---------------------------------------------------------------------- diff --git a/sql/core/src/main/scala/org/apache/spark/sql/DataFrame.scala b/sql/core/src/main/scala/org/apache/spark/sql/DataFrame.scala index 1beb080..8884daa 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/DataFrame.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/DataFrame.scala @@ -1363,7 +1363,8 @@ class DataFrame private[sql]( // All columns are string type val schema = StructType( StructField("summary", StringType) :: outputCols.map(StructField(_, StringType))).toAttributes - LocalRelation.fromExternalRows(schema, ret) + // `toArray` forces materialization to make the seq serializable + LocalRelation.fromExternalRows(schema, ret.toArray.toSeq) } /** --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
