JB, reshuffle deprecation is discussed mutliple times, most recent one is from May : https://lists.apache.org/thread.html/820064a81c86a6d44f21f0d6c68ea3f46cec151e5e1a0b52eeed3fbf@%3Cdev.beam.apache.org%3E I filed https://issues.apache.org/jira/browse/BEAM-4372. Summary: It can be used, and perhaps needs a rename. Just mentioned the thread here as a reference, further discussion could be directed to the above dev thread or the jira.
Thanks Robert for more detailed description. Raghu. On Fri, Jul 6, 2018 at 11:11 PM Jean-Baptiste Onofré <[email protected]> wrote: > Hi Raghu, > > AFAIR, Reshuffle is considered as deprecated. Maybe it would be better > to avoid to use it no ? > > Regards > JB > > On 06/07/2018 18:28, Raghu Angadi wrote: > > I would use Reshuffle()[1] with entity id as the key. It internally does > > a GroupByKey and sets up windowing such that it does not buffer anything. > > > > [1] > > : > https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Reshuffle.java#L51 > > > > > On Fri, Jul 6, 2018 at 8:01 AM Niels Basjes <[email protected] > > <mailto:[email protected]>> wrote: > > > > Hi, > > > > I have an unbounded stream of change events each of which has the id > > of the entity that is changed. > > To avoid the need for locking in the persistence layer that is > > needed in part of my processing I want to route all events based on > > this entity id. > > That way I know for sure that all events around a single entity go > > through the same instance of my processing sequentially, hence no > > need for locking or other synchronization regarding this persistence. > > > > At this point my best guess is that I need to use the GroupByKey but > > that seems to need a Window. > > I think I don't want a window because the stream is unbounded and I > > want the lowest possible latency (i.e. a Window of 1 second would be > > ok for this usecase). > > Also I want to be 100% sure that all events for a specific id go to > > only a single instance because I do not want any race conditions. > > > > My simple question is: What does that look like in the Beam Java API? > > > > -- > > Best regards / Met vriendelijke groeten, > > > > Niels Basjes > > > > -- > Jean-Baptiste Onofré > [email protected] > http://blog.nanthrax.net > Talend - http://www.talend.com >
