Re: question on spark streaming based on event time

Hi, I am sorry I made a really bad Typo. What I meant in my email was actually structured streaming so I wish I could do s/Spark Streaming/Structured Streaming/g. Thanks for the pointers looks like what I was looking for is actually watermarking since my question is all about what I should do if

Re: question on spark streaming based on event time

Spark Streaming (DStreams) wasnt designed keeping event-time in mind. Instead, we have designed Structured Streaming to naturally deal with event time. You should check that out. Here are the pointers. - Programming guide -

question on spark streaming based on event time

Hi All, I read through the documentation on Spark Streaming based on event time and how spark handles lags w.r.t processing time and so on.. but what if the lag is too long between the event time and processing time? other words what should I do if I am receiving yesterday's data (the timestamp