bart-samwel commented on pull request #29181: URL: https://github.com/apache/spark/pull/29181#issuecomment-662483694
Can you double check that the ordering is correct if there are NULLs involved, or outer join conditions? The tricky cases I see: - RIGHT / FULL SHJ. If the streaming / "probe" input is ordered by (some of) the join keys. After consuming the streaming input, the hash join will emit rows for build side rows that didn't have matches. Those rows may actually have values for the join keys, and those will end up in the output. Those will be out of order. - RIGHT / FULL SHJ. If the streaming / "prob" input is ordered by some non-join keys. After consuming the streaming input, the hash join will emit rows with NULL values for the streaming input's columns, which include the ordering keys. This may be correct if Spark's ordering property has "nulls last", but it may not be correct even then. For instance, if the input is ordered by (JOINKEY1, NONJOINKEY1) with NULLS LAST, then a final output ordering may look like: (1, 'a') (2, 'c') (3, 'b') (1, NULL) But the correct ordering for NULLS LAST is (1, 'a') (1, NULL) (2, 'c') (3, 'b') ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
