Github user gatorsmile commented on the pull request:
https://github.com/apache/spark/pull/10626#issuecomment-170008457
Thank you @sameeragarwal for investigating this! Sorry to bring this to you
at midnight.
For helping anyone understand the problem, let me post the logical plan if
we do not collect the data to the local node:
```
Aggregate [id#1], [id#1]
+- Join LeftSemi, None
:- Filter (id#1 <=> id#1)
: +- Sample 0.0, 0.4, false, 1
: +- Sort [id#1 ASC], false
: +- Project [_1#0 AS id#1]
: +- LogicalRDD [_1#0], MapPartitionsRDD[2] at apply at
Transformer.scala:22
+- Sample 0.4, 1.0, false, 1
+- Project
+- Sort [id#1 ASC], false
+- Project [_1#0 AS id#1]
+- LogicalRDD [_1#0], MapPartitionsRDD[2] at apply at
Transformer.scala:22
```
The non-empty result of DF `Intersect` is expected due to either randomness
of data distribution or nondeterministic results of `Sample`.
Since this is one line change, I think I just include it and to let the
tests passed. I will also add a comment to explain it in the test case.
Sometimes, users might read the test case to implement their work.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
