Thank you, so that would mean spark gets the current latest offset(s) when the trigger fires and then process all available messages in the topic upto and including that offset as long as maxOffsetsPerTrigger is the default of None (or large enought to handle all available messages).
I think the word micro-batch confused me (more like mega-batch in some cases). It makes sense though, this makes Trigger.Once a fixed interval trigger that's only fired once and not repeatedly. On Sun, May 3, 2020 at 3:20 AM Jungtaek Lim <kabhwan.opensou...@gmail.com> wrote: > If I understand correctly, Trigger.once executes only one micro-batch and > terminates, that's all. Your understanding of structured streaming applies > there as well. > > It's like a hybrid approach as bringing incremental processing from > micro-batch but having processing interval as batch. That said, while it > enables to get both sides of benefits, it's basically structured streaming, > inheriting all the limitations on the structured streaming, compared to the > batch query. > > Spark 3.0.0 will bring some change on Trigger.once (SPARK-30669 [1]) - > Trigger.once will "ignore" the read limit per micro-batch on data source > (like maxOffsetsPerTrigger) and process all available input as possible. > (Data sources should migrate to the new API to take effect, but works for > built-in data sources like file and Kafka.) > > 1. https://issues.apache.org/jira/browse/SPARK-30669 > > 2020년 5월 2일 (토) 오후 5:35, Magnus Nilsson <ma...@kth.se>님이 작성: > >> I've always had a question about Trigger.Once that I never got around to >> ask or test for myself. If you have a 24/7 stream to a Kafka topic. >> >> Will Trigger.Once get the last offset(s) when it starts and then quit >> once it hits this offset(s) or will the job run until no new messages is >> added to the topic for a particular amount of time? >> >> br, >> >> Magnus >> >> On Sat, May 2, 2020 at 1:22 AM Burak Yavuz <brk...@gmail.com> wrote: >> >>> Hi Rishi, >>> >>> That is exactly why Trigger.Once was created for Structured Streaming. >>> The way we look at streaming is that it doesn't have to be always real >>> time, or 24-7 always on. We see streaming as a workflow that you have to >>> repeat indefinitely. See this blog post for more details! >>> >>> https://databricks.com/blog/2017/05/22/running-streaming-jobs-day-10x-cost-savings.html >>> >>> Best, >>> Burak >>> >>> On Fri, May 1, 2020 at 2:55 PM Rishi Shah <rishishah.s...@gmail.com> >>> wrote: >>> >>>> Hi All, >>>> >>>> I recently started playing with spark streaming, and checkpoint >>>> location feature looks very promising. I wonder if anyone has an opinion >>>> about using spark streaming with checkpoint location option as a slow batch >>>> processing solution. What would be the pros and cons of utilizing streaming >>>> with checkpoint location feature to achieve fault tolerance in batch >>>> processing application? >>>> >>>> -- >>>> Regards, >>>> >>>> Rishi Shah >>>> >>>
