cloud-fan commented on pull request #31287: URL: https://github.com/apache/spark/pull/31287#issuecomment-765183307
Ah actually this fixes a 3.1 regression. Previously we merged a bug fix: https://github.com/apache/spark/pull/30488 , which makes sure ambiguous self-join check is always applied. That said, in 3.0 the ambiguous self-join check is skipped under some cases, which hides the bug this PR is fixing. For the query below, it works in 3.0, but fails in 3.1. After this PR, it works again. ``` sql("create table t1 using json as select 1 key, 1 value") sql("create table t2 using json as select 1 key, 2 value") val t1 = spark.table("t1") val t2 = spark.table("t2") val t3 = t1.join(t2, t1("key") === t2("key")).select(t1("value")) t1.join(t3, t1("key") > 1) ``` cc @HyukjinKwon @dongjoon-hyun I think it's a 3.1.1 blocker. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
