Github user viirya commented on the issue:
https://github.com/apache/spark/pull/18652
I just checked with Hive's behavior (2.1.0). I tried a query like `select *
from l left outer join r on rand(l.a) > 0.1 and rand(cast(l.b as int)) > 0.2
and rand(r.c) > 0.2 and rand(cast(r.d as int)) > 0.5;`.
The conditions `rand(r.c) > 0.2 and rand(cast(r.d as int)) > 0.5` are
pushed down to Filter operator.
TableScan
alias: r
Statistics: Num rows: 2 Data size: 10 Basic stats: COMPLETE
Column stats: NONE
Select Operator
expressions: c (type: int), d (type: double)
outputColumnNames: _col0, _col1
Statistics: Num rows: 2 Data size: 10 Basic stats: COMPLETE
Column stats: NONE
Filter Operator
predicate: ((rand(UDFToInteger(_col1)) > 0.5) and
(rand(_col0) > 0.2)) (type: boolean)
Statistics: Num rows: 1 Data size: 5 Basic stats: COMPLETE
Column stats: NONE
HashTable Sink Operator
filter predicates:
0 {(rand(_col0) > 0.1)} {(rand(UDFToInteger(_col1)) >
0.2)}
1
keys:
0
The other conditions `rand(l.a) > 0.1 and rand(cast(l.b as int)) > 0.2` are
filter predicates with the Join operator.
Map Join Operator
condition map:
Left Outer Join0 to 1
filter predicates:
0 {(rand(_col0) > 0.1)} {(rand(UDFToInteger(_col1)) >
0.2)}
1
keys:
0
1
A query `select * from l left outer join r on rand(l.a) = rand(r.c);` with
non-deterministic joining keys. There's no push down. Hive simply evaluates the
joining keys.
Map Join Operator
condition map:
Left Outer Join0 to 1
keys:
0 rand(_col0) (type: double)
1 rand(_col0) (type: double)
outputColumnNames: _col0, _col1, _col2, _col3
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
