[ https://issues.apache.org/jira/browse/SPARK-31123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mikel San Vicente resolved SPARK-31123. --------------------------------------- Fix Version/s: 3.0.0 Resolution: Fixed > Drop does not work after join with aliases > ------------------------------------------ > > Key: SPARK-31123 > URL: https://issues.apache.org/jira/browse/SPARK-31123 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 2.4.2 > Reporter: Mikel San Vicente > Priority: Major > Fix For: 3.0.0 > > > > Hi, > I am seeing a really strange behaviour in drop method after a join with > aliases. It doesn't seem to find the column when I reference to it using > dataframe("columnName") syntax, but it does work with other combinators like > select > {code:java} > case class Record(a: String, dup: String) > case class Record2(b: String, dup: String) > val df = Seq(Record("a", "dup")).toDF > val df2 = Seq(Record2("a", "dup")).toDF > val joined = df.alias("a").join(df2.alias("b"), df("a") === df2("b")) > val dupCol = df("dup") > joined.drop(dupCol) // Does not drop anything > joined.drop(func.col("a.dup")) // It drops the column > joined.select(dupCol) // It selects the column > {code} > > > -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org