Re: How does Spark Structured Streaming determine an event has arrived late?

2018-02-27 Thread kant kodali
n the picture. In hindsight, the picture can be >> made much better. :) >> >> Hope this explanation helps! >> >> TD >> >> On Tue, Feb 27, 2018 at 2:26 AM, kant kodali wrote: >> >>> I read through the spark structured streaming documentation

Re: How does Spark Structured Streaming determine an event has arrived late?

2018-02-27 Thread naresh Goud
he picture. In hindsight, the picture can be > made much better. :) > > Hope this explanation helps! > > TD > > On Tue, Feb 27, 2018 at 2:26 AM, kant kodali wrote: > >> I read through the spark structured streaming documentation and I wonder >> how does spar

Re: How does Spark Structured Streaming determine an event has arrived late?

2018-02-27 Thread Tathagata Das
;time" is actually "event-time". My apologies for not making this clear in the picture. In hindsight, the picture can be made much better. :) Hope this explanation helps! TD On Tue, Feb 27, 2018 at 2:26 AM, kant kodali wrote: > I read through the spark structured streaming docu

How does Spark Structured Streaming determine an event has arrived late?

2018-02-27 Thread kant kodali
I read through the spark structured streaming documentation and I wonder how does spark structured streaming determine an event has arrived late? Does it compare the event-time with the processing time? [image: enter image description here] <https://i.stack.imgur.com/CXH4i.png> Taking the