Generally, Beam will discard expired data for you (including state). Can you describe more? What is your windowing strategy? What are the edge triggers?
On Wed, Jan 24, 2018 at 10:37 AM, Tim Ross <[email protected]> wrote: > I am just trying to do certain processing on edge triggers, i.e. new or > expired data, to reduce the overall processing of a very large stream. > > > > How would I go about doing that with state? As I understand it, state is > tied to key and window. > > > > Thanks, > > Tim > > *From: *Kenneth Knowles <[email protected]> > *Reply-To: *"[email protected]" <[email protected]> > *Date: *Wednesday, January 24, 2018 at 1:25 PM > *To: *"[email protected]" <[email protected]> > *Subject: *Re: Windowing question > > > > A little clarification: in Beam an element exists in a single window, > mathematically speaking. So when you use SlidingWindows, for example, to > assign multiple windows this "copies" the value for each window, and that > is how you should think of it, from a calculation point of view. Under the > hood, a compressed representation is often used, but not in all situations. > > > > Kenn > > > > On Wed, Jan 24, 2018 at 9:45 AM, Robert Bradshaw <[email protected]> > wrote: > > No, Apache Beam doesn't offer this explicitly. You could accomplish it > using State, but perhaps if you clarified what you were trying to > accomplish by using these mechanisms there'd be another way to do the > same thing. > > > On Wed, Jan 24, 2018 at 7:03 AM, Tim Ross <[email protected]> > wrote: > > Is there anything comparable to Apache Storm’s Window.getNew and > > Window.getExpired in Apache Beam? How would I determine if an element is > > new or expired in consecutive windows? > > > > > > > > Thanks, > > > > Tim > > > > This e-mail message and any attachments to it are intended only for the > > named recipients and may contain legally privileged and/or confidential > > information. If you are not one of the intended recipients, do not > duplicate > > or forward this e-mail message. > > > > This e-mail message and any attachments to it are intended only for the > named recipients and may contain legally privileged and/or confidential > information. If you are not one of the intended recipients, do not > duplicate or forward this e-mail message. >
