Well, that is very fair. In addition to supporting pub/sub in the comm-layer, I think S4 can actually be used to build a fault-tolerant, efficient content-based pub/sub, but that is for a later date.
Thanks Karthik On Thu, Dec 8, 2011 at 9:02 PM, kishore g <[email protected]> wrote: > External pub/sub is useful because they can provide better buffering what > s4 system can provide. It is particularly helpful when the pipelines have > different processing/computing times. Think of a case where you have a > stage which is simply transforming data and another stage is actually doing > some complex algorithm after accumulating some data. Very soon the second > stage buffer will become full and start dropping events. > > It may not as efficient as doing a tcp/udp but has its own advantages. Some > pub/sub also provides better fault tolerance. > > I am not advocating one over the other because it depends on the use case > and what trade off user is willing to make. We should provide one such > implementation to make sure we dont have something in s4 that makes it > difficult to support this requirement. > > thanks, > Kishore G > > > > On Thu, Dec 8, 2011 at 4:46 PM, Karthik Kambatla <[email protected] > >wrote: > > > Hi Kishore > > > > Do you see any particular advantages of using pub/sub for inter-node > > communication over UDP/TCP? The only advantage I see is an external > pub/sub > > being efficient than our implementations of UDP/TCP. If there is no > > advantage, I don't think it makes any sense. > > > > Thanks > > Karthik > > > > On Thu, Dec 8, 2011 at 7:26 PM, Leo Neumeyer <[email protected]> > > wrote: > > > > > Sure, if someone really wants it and is willing to design/implement, > > > I don't see why not. It's a plugin so it wouldn't require major > > > modifications. > > > > > > -leo > > > > > > On Thu, Dec 8, 2011 at 2:39 PM, kishore g <[email protected]> wrote: > > > > Sorry, too many lists. > > > > > > > > Comm layer is supporting p2p. I remember some time back there was an > > ask > > > > for support for Active MQ in s4. The comm layer right now can support > > > that > > > > but we need to write one implementation to write/read to/from a pub > sub > > > to > > > > make sure we can support it. We can do it for Active MQ. This is post > > 0.5 > > > > > > > > On Thu, Dec 8, 2011 at 2:25 PM, Leo Neumeyer <[email protected]> > > > wrote: > > > > > > > >> Hi Kishore, > > > >> > > > >> Seems to me that our comm layer is already implementing a type of > > > >> pub/sub system but with more granularity based on keys. I'm not very > > > >> familiar with pub/sub systems though. Can you tell us more about the > > > >> motivation? Is it to integrate with existing systems? In any case, > you > > > >> are not suggesting this for v0.5, right? > > > >> > > > >> (BTW, we should move this to s4-dev) > > > >> > > > >> -leo > > > >> > > > >> On Thu, Dec 8, 2011 at 1:02 PM, kishore g <[email protected]> > > wrote: > > > >> > Hi, > > > >> > > > > >> > In s4, the comm layer currently provides support for udp/tcp. We > > > should > > > >> add > > > >> > support for a pub/sub layer. There are already some pub/sub > systems > > > out > > > >> > there. Any thoughts ? > > > >> > > > > >> > thanks, > > > >> > Kishore G > > > >> > > > >> > > > >> > > > >> -- > > > >> > > > >> Leo Neumeyer (@leoneu) > > > >> > > > > > > > > > > > > -- > > > > > > Leo Neumeyer (@leoneu) > > > > > >
