Hi,
Given two Spark-Structured streams and using them as
https://spark.apache.org/docs/3.5.6/structured-streaming-programming-guide.html#inner-joins-with-optional-watermarking,
just works.
Now if I want to join three streams using the same technique, Spark
complains about multiple possible watermarks. I have a rough
understanding of what happened, and concluded this works as designed.
But as I am certainly not the only one who tried that, what is the
canonical way of doing this? My first idea was like: I'm going to join
S1,S2 with their corresponding watermarks, then write that result to
disk, possibly into a delta table, read the result with another stream
and join this one with the remaining stream and third watermark.
Is there some other way? Or is this the current canonical way of joining
more than two streams that carry a watermark?
Best Regards
M.
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscr...@spark.apache.org
