jose-torres commented on issue #23634: [SPARK-26187][SS] Streaming left/right outer join should not return outer nulls for already matched rows URL: https://github.com/apache/spark/pull/23634#issuecomment-457230959 I think the bug here isn't in the third step, but the second step: > row R1 is evicted at batch B due to join and watermark condition, whereas row L1 is not evicted This isn't valid even with a matched flag. If L1 isn't evicted, that means a new row L1' should still be able to match with R1, and therefore R1 can't be evicted either. The unit test seems to bear this out; the left side of the self-join is supposed to evict records 5 seconds behind the watermark, but it seems to be incorrectly waiting 10 second instead.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
