Re: [Structured Streaming] Two watermarks and StreamingQueryListener

Thanks for the explanation. We are doing something like this. The *first* watermark is to eliminate the late events from *kafka* The *second* watermark is to eliminate older aggregated metrics across *sessions* I know I can replace the second one with *window* but I was not able to come up

Re: Recommendation of using StreamSinkProvider for a custom KairosDB Sink

nk by implementing the StreamSinkProvider ? >> >> The interface is marked experimental and in-stable ? >> >> >> >> -- >> Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ >> >> --------- >> To unsubscribe e-mail: user-unsubscr...@spark.apache.org >> >> > -- Thanks & Regards, Girish Subramanian

Kafka streaming maxOffsetsPerTrigger

to process Will setting maxOffsetsPerTrigger help me? How does that map to the previous DStream batch size ? Thanks Girish -- Thanks & Regards, Girish Subramanian