I like the idea of using pub/sub. It could be used in cases that
losing events is not acceptable. One system that we have some good
knowledge about and could consider is Hedwig, since it is scalable and
provides strong durability guarantees: http://zookeeper.apache.org/bookkeeper
-Flavio
On Dec 9, 2011, at 3:02 AM, kishore g wrote:
External pub/sub is useful because they can provide better buffering
what
s4 system can provide. It is particularly helpful when the pipelines
have
different processing/computing times. Think of a case where you have a
stage which is simply transforming data and another stage is
actually doing
some complex algorithm after accumulating some data. Very soon the
second
stage buffer will become full and start dropping events.
It may not as efficient as doing a tcp/udp but has its own
advantages. Some
pub/sub also provides better fault tolerance.
I am not advocating one over the other because it depends on the use
case
and what trade off user is willing to make. We should provide one such
implementation to make sure we dont have something in s4 that makes it
difficult to support this requirement.
thanks,
Kishore G
On Thu, Dec 8, 2011 at 4:46 PM, Karthik Kambatla <[email protected]
>wrote:
Hi Kishore
Do you see any particular advantages of using pub/sub for inter-node
communication over UDP/TCP? The only advantage I see is an external
pub/sub
being efficient than our implementations of UDP/TCP. If there is no
advantage, I don't think it makes any sense.
Thanks
Karthik
On Thu, Dec 8, 2011 at 7:26 PM, Leo Neumeyer <[email protected]>
wrote:
Sure, if someone really wants it and is willing to design/
implement,
I don't see why not. It's a plugin so it wouldn't require major
modifications.
-leo
On Thu, Dec 8, 2011 at 2:39 PM, kishore g <[email protected]>
wrote:
Sorry, too many lists.
Comm layer is supporting p2p. I remember some time back there was
an
ask
for support for Active MQ in s4. The comm layer right now can
support
that
but we need to write one implementation to write/read to/from a
pub sub
to
make sure we can support it. We can do it for Active MQ. This is
post
0.5
On Thu, Dec 8, 2011 at 2:25 PM, Leo Neumeyer
<[email protected]>
wrote:
Hi Kishore,
Seems to me that our comm layer is already implementing a type of
pub/sub system but with more granularity based on keys. I'm not
very
familiar with pub/sub systems though. Can you tell us more about
the
motivation? Is it to integrate with existing systems? In any
case, you
are not suggesting this for v0.5, right?
(BTW, we should move this to s4-dev)
-leo
On Thu, Dec 8, 2011 at 1:02 PM, kishore g <[email protected]>
wrote:
Hi,
In s4, the comm layer currently provides support for udp/tcp. We
should
add
support for a pub/sub layer. There are already some pub/sub
systems
out
there. Any thoughts ?
thanks,
Kishore G
--
Leo Neumeyer (@leoneu)
--
Leo Neumeyer (@leoneu)
flavio
junqueira
research scientist
[email protected]
direct +34 93-183-8828
avinguda diagonal 177, 8th floor, barcelona, 08018, es
phone (408) 349 3300 fax (408) 349 3301
