Re: [Spark Streaming] How to clear old data from Stream State?

trackStateByKey API is in branch-1.6 FYI On Wed, Nov 25, 2015 at 6:03 AM, Todd Nist wrote: > Perhaps the new trackStateByKey targeted for very 1.6 may help you here. > I'm not sure if it is part of 1.6 or not for sure as the jira does not > specify a fixed version. The

[Spark Streaming] How to clear old data from Stream State?

Hello, I know how I could clear the old state depending on the input value. If some condition matches to determine that the state is old then set the return null, will invalidate the record. But this is only feasible if a new record arrives that matches the old key. What if no new data arrives

Re: [Spark Streaming] How to clear old data from Stream State?

Perhaps the new trackStateByKey targeted for very 1.6 may help you here. I'm not sure if it is part of 1.6 or not for sure as the jira does not specify a fixed version. The jira describing it is here: https://issues.apache.org/jira/browse/SPARK-2629, and the design doc that discusses the API