Re: Process each kafka record for structured streaming

[Jacek Laskowski]

Hi,

Can you use console sink and make sure that the pipeline shows some
progress?

Pozdrawiam,
Jacek Laskowski

https://about.me/JacekLaskowski
"The Internals Of" Online Books 
Follow me on https://twitter.com/jaceklaskowski




On Wed, Jan 20, 2021 at 10:44 AM rajat kumar 
wrote:

> Hi,
>
> I want to apply custom logic for each row of data I am getting through
> kafka and want to do it with microbatch.
> When I am running it , it is not progressing.
>
>
> kafka_stream_df \
> .writeStream \
> .foreach(process_records) \
> .outputMode("append") \
> .option("checkpointLocation", "checkpt") \
> .trigger(continuous="5 seconds").start()
>
> Regards
>
> Rajat
>
>
>

Process each kafka record for structured streaming

[rajat kumar]

Hi,

I want to apply custom logic for each row of data I am getting through
kafka and want to do it with microbatch.
When I am running it , it is not progressing.


kafka_stream_df \
.writeStream \
.foreach(process_records) \
.outputMode("append") \
.option("checkpointLocation", "checkpt") \
.trigger(continuous="5 seconds").start()

Regards

Rajat