Hi, We are using push streams to process post-commit events. Those events originate from different data sources. At the moment we are processing those individually, but the overhead of having a transaction for each is too much. Quite often those events come in bursts following an upstream transaction/ change set.
The goal is to group events by data source and batch them, i.e. wait a bit when an event arrives to see if others are also coming. If they keep coming, keep collecting a bit longer, o/w move on. I see that the PushStream has methods coalesce and window. Window seems a bit more appropriate here, as it offers both duration and maxEvents. But it seems to operate all the time, and not start a batch upon receiving an event, which doesn't sound optimal in this case. More concerning to me is the comment regarding back-pressure. We can't use back pressure (no control on producer which is implemented via whiteboard. So here the maxEvents is more a way to limit the batch and not to indicate need for back pressure. Still, that doesn't address grouping. See that there is a fork, but that is made to deal with a fixed number of child streams. Would I just be best to use a window with just duration, collect a number of events, then move on and use a regular stream to group them and if necessary batch them in smaller groups? Cheers, Alain
_______________________________________________ OSGi Developer Mail List osgi-dev@mail.osgi.org https://mail.osgi.org/mailman/listinfo/osgi-dev
