Re: mapWithState: remove key
That makes sense. Thanks for your quick response. On Fri, Jan 29, 2016 at 7:01 PM, Shixiong(Ryan) Zhuwrote: > 1. To remove a state, you need to call "state.remove()". If you return a > None in the function, it just means don't output it as the DStream's > output, but the state won't be removed if you don't call "state.remove()". > > 2. For NoSuchElementException, here is the doc for "State.get": > > /** >* Get the state if it exists, otherwise it will throw > `java.util.NoSuchElementException`. >* Check with `exists()` whether the state exists or not before calling > `get()`. >* >* @throws java.util.NoSuchElementException If the state does not exist. >*/ > > > > > On Fri, Jan 29, 2016 at 10:45 AM, Udo Fholl > wrote: > >> Hi, >> >> From the signature of the "mapWithState" method I infer that by returning >> a "None.type" (in Scala) the key is removed from the state. Is that so? >> Sorry if it is in the docs, but it wasn't entirely clear to me. >> >> I'm chaining operations and calling "mapWithState" twice (one to >> consolidate, then I perform some operations that might, or might not >> succeed, and invoke "mapWithState" again). I'm getting this error[1] which >> I suppose is because I'm returning "None" in the "mapWithState" function. >> >> Thank you. >> >> Best regards, >> Udo. >> >> [1]: java.util.NoSuchElementException: State is not set >> at org.apache.spark.streaming.StateImpl.get(State.scala:150) >> > >
Re: mapWithState: remove key
1. To remove a state, you need to call "state.remove()". If you return a None in the function, it just means don't output it as the DStream's output, but the state won't be removed if you don't call "state.remove()". 2. For NoSuchElementException, here is the doc for "State.get": /** * Get the state if it exists, otherwise it will throw `java.util.NoSuchElementException`. * Check with `exists()` whether the state exists or not before calling `get()`. * * @throws java.util.NoSuchElementException If the state does not exist. */ On Fri, Jan 29, 2016 at 10:45 AM, Udo Fhollwrote: > Hi, > > From the signature of the "mapWithState" method I infer that by returning > a "None.type" (in Scala) the key is removed from the state. Is that so? > Sorry if it is in the docs, but it wasn't entirely clear to me. > > I'm chaining operations and calling "mapWithState" twice (one to > consolidate, then I perform some operations that might, or might not > succeed, and invoke "mapWithState" again). I'm getting this error[1] which > I suppose is because I'm returning "None" in the "mapWithState" function. > > Thank you. > > Best regards, > Udo. > > [1]: java.util.NoSuchElementException: State is not set > at org.apache.spark.streaming.StateImpl.get(State.scala:150) >
mapWithState: remove key
Hi, >From the signature of the "mapWithState" method I infer that by returning a "None.type" (in Scala) the key is removed from the state. Is that so? Sorry if it is in the docs, but it wasn't entirely clear to me. I'm chaining operations and calling "mapWithState" twice (one to consolidate, then I perform some operations that might, or might not succeed, and invoke "mapWithState" again). I'm getting this error[1] which I suppose is because I'm returning "None" in the "mapWithState" function. Thank you. Best regards, Udo. [1]: java.util.NoSuchElementException: State is not set at org.apache.spark.streaming.StateImpl.get(State.scala:150)