For now I went with my simple solution of using a window with just duration, and that is working fine, even if it might not be the most optimal or streamlined approach.
Alain On Sat, Jan 5, 2019 at 5:27 AM Alain Picard <pic...@castortech.com> wrote: > Hi, > > We are using push streams to process post-commit events. Those events > originate from different data sources. At the moment we are processing > those individually, but the overhead of having a transaction for each is > too much. Quite often those events come in bursts following an upstream > transaction/ change set. > > The goal is to group events by data source and batch them, i.e. wait a bit > when an event arrives to see if others are also coming. If they keep > coming, keep collecting a bit longer, o/w move on. > > I see that the PushStream has methods coalesce and window. Window seems a > bit more appropriate here, as it offers both duration and maxEvents. But it > seems to operate all the time, and not start a batch upon receiving an > event, which doesn't sound optimal in this case. More concerning to me is > the comment regarding back-pressure. We can't use back pressure (no control > on producer which is implemented via whiteboard. So here the maxEvents is > more a way to limit the batch and not to indicate need for back pressure. > > Still, that doesn't address grouping. See that there is a fork, but that > is made to deal with a fixed number of child streams. > > Would I just be best to use a window with just duration, collect a number > of events, then move on and use a regular stream to group them and if > necessary batch them in smaller groups? > > Cheers, > > Alain > >
_______________________________________________ OSGi Developer Mail List osgi-dev@mail.osgi.org https://mail.osgi.org/mailman/listinfo/osgi-dev
