i think its just the sheer volume.
the work done per message is pretty minimal:
it basically checks a header field to see if it should be dropped
and then (if not) simply resend.
its not a problem exactly now; they run at an average of 30% cpu.
but i am only running on 10% of the full scale input.
i guess i can eliminate the forwarders; its one less process to worrry about.
adressing is not a concern; the receivers are anonymous now, and they can
remain so, anchoring to the 80 processes rather than just to teh one device.
i guess we have nothing to fear but fear itself.
thanks, pieter.
andrew
On May 20, 2011, at 2:37 AM, Pieter Hintjens wrote:
> On Fri, May 20, 2011 at 11:18 AM, Andrew Hume <[email protected]> wrote:
>
>> i think i need to rework some of my architecture because its not scaling
>> like i need it to.
>
> Could you explain why the forwarders are getting overloaded? What's
> your message volume, message size, and work per message in the
> forwarder?
>
> Normally, you could expect devices to become bottlenecks, but that's
> balanced against them doing much less work per message than endpoints.
> And a device will essentially be CPU bound. So if a device starts to
> do too much work per message, it'll degrade. Or, if the overall
> volumes combine to more than the device can handle, it'll degrade.
>
> Options are therefore to reduce the number of messages, to reduce the
> work done per message, in devices, and eventually to remove devices
> altogether.
>
> The pattern which perhaps works best for such a network is a single
> device for coordination and registration, but all traffic flowing peer
> to peer. 80 sockets per node is not an issue, you can open thousands
> if needed. The issue is usually to address these, and you can solve
> that with a broker / name server device.
>
> It's not been covered in the Guide yet so whatever you come up with,
> please do share, it'll be useful to many people.
>
> -Pieter
> _______________________________________________
> zeromq-dev mailing list
> [email protected]
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
------------------
Andrew Hume (best -> Telework) +1 623-551-2845
[email protected] (Work) +1 973-236-2014
AT&T Labs - Research; member of USENIX and LOPSA
_______________________________________________
zeromq-dev mailing list
[email protected]
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
