[
https://issues.apache.org/jira/browse/SPARK-24781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takuya Ueshin updated SPARK-24781:
----------------------------------
Description:
When we use a reference from {{Dataset}} in {{filter}} or {{sort}}, which was
not used in the prior {{select}}, an {{AnalysisException}} occurs, e.g.,
{code:scala}
val df = Seq(("test1", 0), ("test2", 1)).toDF("name", "id")
df.select(df("name")).filter(df("id") === 0).show()
{code}
{noformat}
org.apache.spark.sql.AnalysisException: Resolved attribute(s) id#6 missing from
name#5 in operator !Filter (id#6 = 0).;;
!Filter (id#6 = 0)
+- AnalysisBarrier
+- Project [name#5]
+- Project [_1#2 AS name#5, _2#3 AS id#6]
+- LocalRelation [_1#2, _2#3]
{noformat}
If we use {{col}} instead, it works:
{code:scala}
val df = Seq(("test1", 0), ("test2", 1)).toDF("name", "id")
df.select(col("name")).filter(col("id") === 0).show()
{code}
was:
When we use a reference from {{Dataset}} in {{filter}} or {{sort}}, which was
not used in the prior {{select}}, the {{AnalysisException}} occurs, e.g.,
{code:scala}
val df = Seq(("test1", 0), ("test2", 1)).toDF("name", "enable")
df.select(df("name")).filter(df("enable") =!= 0).show()
{code}
{noformat}
org.apache.spark.sql.AnalysisException: Resolved attribute(s) enable#6 missing
from name#5 in operator !Filter NOT (enable#6 = 0).;;
!Filter NOT (enable#6 = 0)
+- AnalysisBarrier
+- Project [name#5]
+- Project [_1#2 AS name#5, _2#3 AS enable#6]
+- LocalRelation [_1#2, _2#3]
{noformat}
If we use {{col}} instead, it works:
{code:scala}
val df = Seq(("test1", 0), ("test2", 1)).toDF("name", "enable")
df.select(col("name")).filter(col("enable") =!= 0).show()
{code}
> Using a reference from Dataset in Filter/Sort might not work.
> -------------------------------------------------------------
>
> Key: SPARK-24781
> URL: https://issues.apache.org/jira/browse/SPARK-24781
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.3.1
> Reporter: Takuya Ueshin
> Priority: Blocker
>
> When we use a reference from {{Dataset}} in {{filter}} or {{sort}}, which was
> not used in the prior {{select}}, an {{AnalysisException}} occurs, e.g.,
> {code:scala}
> val df = Seq(("test1", 0), ("test2", 1)).toDF("name", "id")
> df.select(df("name")).filter(df("id") === 0).show()
> {code}
> {noformat}
> org.apache.spark.sql.AnalysisException: Resolved attribute(s) id#6 missing
> from name#5 in operator !Filter (id#6 = 0).;;
> !Filter (id#6 = 0)
> +- AnalysisBarrier
> +- Project [name#5]
> +- Project [_1#2 AS name#5, _2#3 AS id#6]
> +- LocalRelation [_1#2, _2#3]
> {noformat}
> If we use {{col}} instead, it works:
> {code:scala}
> val df = Seq(("test1", 0), ("test2", 1)).toDF("name", "id")
> df.select(col("name")).filter(col("id") === 0).show()
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
