Re: Spark Streaming withWatermark

2018-02-06 Thread Jiewen Shao
ted, right? however in my test, this one is still counted On Tue, Feb 6, 2018 at 2:05 PM, Vishnu Viswanath < vishnu.viswanat...@gmail.com> wrote: > Yes, that is correct. > > On Tue, Feb 6, 2018 at 4:56 PM, Jiewen Shao > wrote: > >> Vishnu, thanks for the reply >>

Spark Streaming withWatermark

2018-02-06 Thread Jiewen Shao
sample code: Let's say Xyz is POJO with a field called timestamp, regarding code withWatermark("timestamp", "20 seconds") I expect the msg with timestamp 20 seconds or older will be dropped, what does 20 seconds compare to? based on my test nothing was dropped no matter how old the timestamp is,