[
https://issues.apache.org/jira/browse/SPARK-22725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Owen resolved SPARK-22725.
-------------------------------
Resolution: Won't Fix
No follow up and not clear this is expected to work
> df.select on a Stream is broken, vs a List
> ------------------------------------------
>
> Key: SPARK-22725
> URL: https://issues.apache.org/jira/browse/SPARK-22725
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.3.0
> Reporter: Andrew Ash
> Priority: Minor
>
> See failing test at https://github.com/apache/spark/pull/19917
> Failing:
> {noformat}
> test("SPARK-ABC123: support select with a splatted stream") {
> val df = spark.createDataFrame(sparkContext.emptyRDD[Row],
> StructType(List("bar", "foo").map {
> StructField(_, StringType, false)
> }))
> val allColumns = Stream(df.col("bar"), col("foo"))
> val result = df.select(allColumns : _*)
> }
> {noformat}
> Succeeds:
> {noformat}
> test("SPARK-ABC123: support select with a splatted stream") {
> val df = spark.createDataFrame(sparkContext.emptyRDD[Row],
> StructType(List("bar", "foo").map {
> StructField(_, StringType, false)
> }))
> val allColumns = Seq(df.col("bar"), col("foo"))
> val result = df.select(allColumns : _*)
> }
> {noformat}
> After stepping through in a debugger, the difference manifests at
> https://github.com/apache/spark/blob/8ae004b4602266d1f210e4c1564246d590412c06/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/QueryPlan.scala#L120
> Changing {{seq.map}} to {{seq.toList.map}} causes the test to pass.
> I think there's a very subtle bug here where the {{Seq}} of column names
> passed into {{select}} is expected to eagerly evaluate when {{.map}} is
> called on it, even though that's not part of the {{Seq}} contract.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
