I have streams of data coming in from various applications through Kafka.
These streams are converted into dataframes in Spark. I would like to join
these dataframes on a common ID they all contain.
Since joining streaming dataframes is currently not supported, what is the
current recommended way to join two dataFrames in a streaming context.
Is it recommended to keep writing the streaming dataframes into some sink
to convert them into static dataframes which can then be joined? Would
this guarantee end-to-end exactly once and fault tolerant guarantees?