I am just trying to do certain processing on edge triggers, i.e. new or expired data, to reduce the overall processing of a very large stream.
How would I go about doing that with state? As I understand it, state is tied to key and window. Thanks, Tim From: Kenneth Knowles <[email protected]> Reply-To: "[email protected]" <[email protected]> Date: Wednesday, January 24, 2018 at 1:25 PM To: "[email protected]" <[email protected]> Subject: Re: Windowing question A little clarification: in Beam an element exists in a single window, mathematically speaking. So when you use SlidingWindows, for example, to assign multiple windows this "copies" the value for each window, and that is how you should think of it, from a calculation point of view. Under the hood, a compressed representation is often used, but not in all situations. Kenn On Wed, Jan 24, 2018 at 9:45 AM, Robert Bradshaw <[email protected]<mailto:[email protected]>> wrote: No, Apache Beam doesn't offer this explicitly. You could accomplish it using State, but perhaps if you clarified what you were trying to accomplish by using these mechanisms there'd be another way to do the same thing. On Wed, Jan 24, 2018 at 7:03 AM, Tim Ross <[email protected]<mailto:[email protected]>> wrote: > Is there anything comparable to Apache Storm’s Window.getNew and > Window.getExpired in Apache Beam? How would I determine if an element is > new or expired in consecutive windows? > > > > Thanks, > > Tim > > This e-mail message and any attachments to it are intended only for the > named recipients and may contain legally privileged and/or confidential > information. If you are not one of the intended recipients, do not duplicate > or forward this e-mail message. This e-mail message and any attachments to it are intended only for the named recipients and may contain legally privileged and/or confidential information. If you are not one of the intended recipients, do not duplicate or forward this e-mail message.
