That makes sense. Thanks for your quick response. On Fri, Jan 29, 2016 at 7:01 PM, Shixiong(Ryan) Zhu <[email protected] > wrote:
> 1. To remove a state, you need to call "state.remove()". If you return a > None in the function, it just means don't output it as the DStream's > output, but the state won't be removed if you don't call "state.remove()". > > 2. For NoSuchElementException, here is the doc for "State.get": > > /** > * Get the state if it exists, otherwise it will throw > `java.util.NoSuchElementException`. > * Check with `exists()` whether the state exists or not before calling > `get()`. > * > * @throws java.util.NoSuchElementException If the state does not exist. > */ > > > > > On Fri, Jan 29, 2016 at 10:45 AM, Udo Fholl <[email protected]> > wrote: > >> Hi, >> >> From the signature of the "mapWithState" method I infer that by returning >> a "None.type" (in Scala) the key is removed from the state. Is that so? >> Sorry if it is in the docs, but it wasn't entirely clear to me. >> >> I'm chaining operations and calling "mapWithState" twice (one to >> consolidate, then I perform some operations that might, or might not >> succeed, and invoke "mapWithState" again). I'm getting this error[1] which >> I suppose is because I'm returning "None" in the "mapWithState" function. >> >> Thank you. >> >> Best regards, >> Udo. >> >> [1]: java.util.NoSuchElementException: State is not set >> at org.apache.spark.streaming.StateImpl.get(State.scala:150) >> > >
