Cédric Perriard created SPARK-16995:
---------------------------------------
Summary: TreeNodeException when flat mapping
RelationalGroupedDataset created from DataFrame containing a column created
with lit/expr
Key: SPARK-16995
URL: https://issues.apache.org/jira/browse/SPARK-16995
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 2.0.0
Reporter: Cédric Perriard
A TreeNodeException is thrown when executing the following minimal example in
Spark 2.0. Crucial is that the column q is generated with lit/expr.
{code}
import spark.implicits._
case class test (x: Int, q: Int)
val d = Seq(1).toDF("x")
d.withColumn("q", lit(0)).as[test].groupByKey(_.x).flatMapGroups{case (x, iter)
=> List()}.show
d.withColumn("q", expr("0")).as[test].groupByKey(_.x).flatMapGroups{case (x,
iter) => List()}.show
// this works fine
d.withColumn("q", lit(0)).as[test].groupByKey(_.x).count()
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
