er memory, which you can monitor in the task data in
ApplicationMaster web UI.
Cheers
Iain
From: Ofir Kerker [mailto:ofir.ker...@gmail.com]
Sent: 07 April 2016 17:06
To: Iain Cundy; user@spark.apache.org
Subject: Re: mapWithState not compacting removed state
Hi Iain,
Did you manage to solve
Hi Iain,
Did you manage to solve this issue?
It looks like we have a similar issue with processing time increasing every
micro-batch but only after 30 batches.
Thanks.
On Thu, Mar 3, 2016 at 4:45 PM Iain Cundy wrote:
> Hi All
>
>
>
> I’m aggregating data using mapWithState with a timeout set in
Hi All
I'm aggregating data using mapWithState with a timeout set in 1.6.0. It broadly
works well and by providing access to the key and the time in the callback
allows a much more elegant solution for time based aggregation than the old
updateStateByKey function.
However there seems to be a p