External pub/sub is useful because they can provide better buffering what s4 system can provide. It is particularly helpful when the pipelines have different processing/computing times. Think of a case where you have a stage which is simply transforming data and another stage is actually doing some complex algorithm after accumulating some data. Very soon the second stage buffer will become full and start dropping events.
It may not as efficient as doing a tcp/udp but has its own advantages. Some pub/sub also provides better fault tolerance. I am not advocating one over the other because it depends on the use case and what trade off user is willing to make. We should provide one such implementation to make sure we dont have something in s4 that makes it difficult to support this requirement. thanks, Kishore G On Thu, Dec 8, 2011 at 4:46 PM, Karthik Kambatla <[email protected]>wrote: > Hi Kishore > > Do you see any particular advantages of using pub/sub for inter-node > communication over UDP/TCP? The only advantage I see is an external pub/sub > being efficient than our implementations of UDP/TCP. If there is no > advantage, I don't think it makes any sense. > > Thanks > Karthik > > On Thu, Dec 8, 2011 at 7:26 PM, Leo Neumeyer <[email protected]> > wrote: > > > Sure, if someone really wants it and is willing to design/implement, > > I don't see why not. It's a plugin so it wouldn't require major > > modifications. > > > > -leo > > > > On Thu, Dec 8, 2011 at 2:39 PM, kishore g <[email protected]> wrote: > > > Sorry, too many lists. > > > > > > Comm layer is supporting p2p. I remember some time back there was an > ask > > > for support for Active MQ in s4. The comm layer right now can support > > that > > > but we need to write one implementation to write/read to/from a pub sub > > to > > > make sure we can support it. We can do it for Active MQ. This is post > 0.5 > > > > > > On Thu, Dec 8, 2011 at 2:25 PM, Leo Neumeyer <[email protected]> > > wrote: > > > > > >> Hi Kishore, > > >> > > >> Seems to me that our comm layer is already implementing a type of > > >> pub/sub system but with more granularity based on keys. I'm not very > > >> familiar with pub/sub systems though. Can you tell us more about the > > >> motivation? Is it to integrate with existing systems? In any case, you > > >> are not suggesting this for v0.5, right? > > >> > > >> (BTW, we should move this to s4-dev) > > >> > > >> -leo > > >> > > >> On Thu, Dec 8, 2011 at 1:02 PM, kishore g <[email protected]> > wrote: > > >> > Hi, > > >> > > > >> > In s4, the comm layer currently provides support for udp/tcp. We > > should > > >> add > > >> > support for a pub/sub layer. There are already some pub/sub systems > > out > > >> > there. Any thoughts ? > > >> > > > >> > thanks, > > >> > Kishore G > > >> > > >> > > >> > > >> -- > > >> > > >> Leo Neumeyer (@leoneu) > > >> > > > > > > > > -- > > > > Leo Neumeyer (@leoneu) > > >
