Repository: spark
Updated Branches:
  refs/heads/branch-2.3 e5e9f9a43 -> 56eb9a310

[SPARK-23064][SS][DOCS] Stream-stream joins Documentation - follow up

## What changes were proposed in this pull request?
Further clarification of caveats in using stream-stream outer joins.

## How was this patch tested?

Author: Tathagata Das <>

Closes #20494 from tdas/SPARK-23064-2.

(cherry picked from commit eaf35de2471fac4337dd2920026836d52b1ec847)
Signed-off-by: Tathagata Das <>


Branch: refs/heads/branch-2.3
Commit: 56eb9a310217a5372bdba1e24e4af0d4de1829ca
Parents: e5e9f9a
Author: Tathagata Das <>
Authored: Fri Feb 2 17:37:51 2018 -0800
Committer: Tathagata Das <>
Committed: Fri Feb 2 17:38:07 2018 -0800

 docs/ | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)
diff --git a/docs/ 
index 62589a6..48d6d0b 100644
--- a/docs/
+++ b/docs/
@@ -1346,10 +1346,20 @@ joined <- join(
-However, note that the outer NULL results will be generated with a delay 
(depends on the specified
-watermark delay and the time range condition) because the engine has to wait 
for that long to ensure
+There are a few points to note regarding outer joins.
+- *The outer NULL results will be generated with a delay that depends on the 
specified watermark
+delay and the time range condition.* This is because the engine has to wait 
for that long to ensure
 there were no matches and there will be no more matches in future.
+- In the current implementation in the micro-batch engine, watermarks are 
advanced at the end of a
+micro-batch, and the next micro-batch uses the updated watermark to clean up 
state and output
+outer results. Since we trigger a micro-batch only when there is new data to 
be processed, the
+generation of the outer result may get delayed if there no new data being 
received in the stream.
+*In short, if any of the two input streams being joined does not receive data 
for a while, the
+outer (both cases, left or right) output may get delayed.*
 ##### Support matrix for joins in streaming queries
 <table class ="table">

To unsubscribe, e-mail:
For additional commands, e-mail:

Reply via email to