GitHub user gatorsmile opened a pull request:
https://github.com/apache/spark/pull/9548
[SPARK-10838][SPARK-11576][SQL][WIP] Incorrect results or exceptions when
using self-joins
When resolving the attributeReference's ambiguity caused by self joins, the
current solution only handles the conflicting attributes. However, this does
not work when the join conditions use the column names that appear in both
dataFrames since the join conditions are evaluated before resolving the
ambiguity of conflicting attributes. Currently, we did not update the
search-condition. When generating the new expression IDs in the right tree, we
must update the corresponding columns' expression ID in search condition.
Here, I am trying to propose a solution to resolve this issue. When
evaluating the join conditions, we record the dataFrame of the search-condition
columns. Then, when resolving the ambiguity of conflicting attributes, we can
use this information to know which columns are from the right tree, and then
update their expression IDs.
When designing this solution, I am trying to reduce the code changes, and
thus, I am using quantifiers to record this information. Ideally, I think each
column needs to clearly correlate with its original source, but this requires a
lot of code changes, but this will help to optimize the plan in the future.
Thanks for any suggestion!
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/gatorsmile/spark selfJoinConflictingConditions
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/9548.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #9548
----
commit 376691af5f639ca4f7ff07cc9e8f572d53e961bf
Author: xiaoli <[email protected]>
Date: 2015-11-08T18:28:12Z
Spark-10838
commit 7d047136cd710ee0e9ff34aa37c1e6d299165233
Author: xiaoli <[email protected]>
Date: 2015-11-08T21:07:37Z
Merge branch 'selfJoinCondition' into selfJoinConflictingConditions
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
