Hello,
I have a structured streaming job that needs to be able to write to
multiple sinks. We are using *Continuous* Trigger *and not* *Microbatch*
Trigger.
1. When we use the foreach method using:
*dataset1.writeStream.foreach(kafka ForEachWriter
logic).trigger(ContinuousMode).start().awaitTermination() *
*dataset1.writeStream.foreach(mongo **ForEachWriter *
*logic).trigger(ContinuousMode).start().awaitTermination() *
The first statement blocks the second one for obvious reasons. So this does
not serve our purpose.
2. The next step for this problem would be to use the *foreachbatch*. That
is not supported in the ContinuousMode.
3. The next step was to use something like this
*dataset1.writeStream.format("kafka").trigger(ContinuousMode).start().awaitTermination()
*
*dataset1.writeStream.format("mongo").trigger(ContinuousMode).start().awaitTermination()*
for both the sinks. This does not work either. Only the 1st query works.
The second one does not.
Is there any solution to the problem of being able to write to multiple
sinks in Continuous Trigger Mode using Structured Streaming?
--
Best Regards,
Sheel Pancholi
