Re: Why does join use rows that were sent after watermark of 20 seconds?

2018-12-10 Thread Abhijeet Kumar
> If you want the intended behaviour you should add time stamp conditions or > window operator in join condition > > > > On Mon, 10 Dec 2018 at 5:23 PM, Abhijeet Kumar <mailto:abhijeet.ku...@sentienz.com>> wrote: > Hello, > > I’m using watermark to

Why does join use rows that were sent after watermark of 20 seconds?

2018-12-10 Thread Abhijeet Kumar
even after watermark got finished Spark is holding the data in memory. I even tried after 45 seconds and that was getting joined too. I’m sending streams from two Kafka queues and tstamp_trans I’m creating with current timestamp values. This is creating confusion in my mind regarding watermark. Thank you, Abhijeet Kumar

Re: Hive Bucketing Support

2018-06-06 Thread Abhijeet Kumar
I would ask my queries here <https://gitter.im/spark-scala/Lobby>. Thanks, Abhijeet Kumar > On 07-Jun-2018, at 1:03 AM, Chris Martin wrote: > > Hi All, > > > first off apologies if this is not the correct place to ask this! > > I've been following SPARK-