If your goal is to just process events as they come in, you should be able to do something like:
Source -> DoFn -> DoFn -> Sink You only need to GroupByKey if you want all the elements associated with a specific key to be processed together -- which it sounds like you don't. If you do need a GroupByKey, you could then use a trigger (every element or something like that) to trigger the grouping after every element. On Mon, Apr 3, 2017 at 12:14 PM Antony Mayi <[email protected]> wrote: > I thought group-by-key in streaming mode cannot run without windowing? > This is what I get when removing my Window.into(): > > java.lang.IllegalStateException: GroupByKey cannot be applied to > non-bounded PCollection in the GlobalWindow without a trigger. Use a > Window.into or Window.triggering transform prior to GroupByKey. > > my case is: > receiving streaming events > each event is a list of elements > I need to process elements of each event independently of elements of > other events - hence my approach of putting each event in its own window, > then expanding it (since event = list) using flat map and then running > group-by type of transformation on that window (which would mean it is > still isolated from other event elements). > > thx for ideas, > a. > > On Sunday, 2 April 2017, 23:43, Ben Chambers <[email protected]> wrote: > > > Can you elaborate on your use case? If your goal is to just group things, > you can assign a key to each element and then apply a group by key. You > shouldn't need to use windowing for that. > > On Sun, Apr 2, 2017, 2:34 PM Csaba Kassai <[email protected]> > wrote: > > Hi Antony, > > there is a small custom windowing example in this github repo which can be > useful for you: https://github.com/Doctusoft/ds-dataflow-examples > The code is not documented yet, so let me know if you have any question > about it. > > Regards, > Csabi > > > > On Fri, 31 Mar 2017 at 18:04 Robert Bradshaw <[email protected]> wrote: > > Yes, you can extend BoundedWindow to be your own Window type that has > additional members and different equality semantics (rather than > re-using IntervalWindow). The only requirement is that it have an > endpoint. (You'll also have to write a Coder for your new Window > subclass and return that in your WindowFn. > > > https://beam.apache.org/documentation/sdks/javadoc/0.4.0/org/apache/beam/sdk/transforms/windowing/WindowFn.html > > On Thu, Mar 30, 2017 at 11:19 PM, Antony Mayi <[email protected]> > wrote: > > Hi, > > > > is there a way to implement windowing so that each input event gets into > its > > own exclusive window? > > > > I can see the PartitioningWindowFn can be extended. If I implement the > > assignWindow to return new IntervalWindow with both start and end time > set > > to the even time and in case there are two distinct events arriving at > the > > same time (indistinguishable within Instant granularity), would this be > > processed as two separate windows without interfering the event data > during > > any transformations? > > > > My motivation is to to be able to flatmap individual input events into a > > pcollection of multiple elements that - being a single exclusive window - > > can be grouped/... independently of other events (even if the other event > > has same time). > > > > thanks, > > Antony. > > > >
