RE: mapWithState not compacting removed state

which you can monitor in the task data in ApplicationMaster web UI. Cheers Iain From: Ofir Kerker [mailto:ofir.ker...@gmail.com] Sent: 07 April 2016 17:06 To: Iain Cundy; user@spark.apache.org Subject: Re: mapWithState not compacting removed state Hi Iain, Did you manage to solve this issue

Re: mapWithState not compacting removed state

Hi Iain, Did you manage to solve this issue? It looks like we have a similar issue with processing time increasing every micro-batch but only after 30 batches. Thanks. On Thu, Mar 3, 2016 at 4:45 PM Iain Cundy wrote: > Hi All > > > > I’m aggregating data using

mapWithState not compacting removed state

Hi All I'm aggregating data using mapWithState with a timeout set in 1.6.0. It broadly works well and by providing access to the key and the time in the callback allows a much more elegant solution for time based aggregation than the old updateStateByKey function. However there seems to be a