[
https://issues.apache.org/jira/browse/SPARK-30421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17008699#comment-17008699
]
Aman Omer edited comment on SPARK-30421 at 1/6/20 10:07 AM:
------------------------------------------------------------
Plans for
{code:java}
df.drop("bar").where($"bar" === "a")
{code}
{code:java}
== Parsed Logical Plan ==
'Filter ('bar = a)
+- Project [foo#7]
+- Project [_1#2 AS foo#7, _2#3 AS bar#8]
+- LocalRelation [_1#2, _2#3]
== Analyzed Logical Plan ==
foo: int
Project [foo#7]
+- Filter (bar#8 = a)
+- Project [foo#7, bar#8]
+- Project [_1#2 AS foo#7, _2#3 AS bar#8]
+- LocalRelation [_1#2, _2#3]
== Optimized Logical Plan ==
LocalRelation [foo#7]
== Physical Plan ==
LocalTableScan [foo#7]
{code}
was (Author: aman_omer):
Plans for
{code:java}
df.drop("bar").where($"bar" === "a").explain(true)
{code}
{code:java}
== Parsed Logical Plan ==
'Filter ('bar = a)
+- Project [foo#7]
+- Project [_1#2 AS foo#7, _2#3 AS bar#8]
+- LocalRelation [_1#2, _2#3]
== Analyzed Logical Plan ==
foo: int
Project [foo#7]
+- Filter (bar#8 = a)
+- Project [foo#7, bar#8]
+- Project [_1#2 AS foo#7, _2#3 AS bar#8]
+- LocalRelation [_1#2, _2#3]
== Optimized Logical Plan ==
LocalRelation [foo#7]
== Physical Plan ==
LocalTableScan [foo#7]
{code}
> Dropped columns still available for filtering
> ---------------------------------------------
>
> Key: SPARK-30421
> URL: https://issues.apache.org/jira/browse/SPARK-30421
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.4.4
> Reporter: Tobias Hermann
> Priority: Minor
>
> The following minimal example:
> {quote}val df = Seq((0, "a"), (1, "b")).toDF("foo", "bar")
> df.select("foo").where($"bar" === "a").show
> df.drop("bar").where($"bar" === "a").show
> {quote}
> should result in an error like the following:
> {quote}org.apache.spark.sql.AnalysisException: cannot resolve '`bar`' given
> input columns: [foo];
> {quote}
> However, it does not but instead works without error, as if the column "bar"
> would exist.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
