On Thu, May 1, 2014 at 9:36 AM, Michael Goulish <[email protected]> wrote:
> > > > > ----- Original Message ----- > > On 05/01/2014 02:09 AM, Michael Goulish wrote: > > > > > > I just had a successful test in which a single dispatch router > > > handled 100,000 unique addresses. > > > > > > The router was running on its own box. On a separate box, I > > > started 100 receiving clients ( proton messenger based ) each > > > receiving 1000 unique addresses. A single sending client > > > ( qpid-messaging based ) sent the 100,000 messages to the > > > router -- 1 to each address. > > > > > > All the receivers got their expected 1000 messages each, > > > and retired happy. > > > > > > I tried to scale up to 1,000,000 addresses. The router seemed > > > OK with that, but my client-box melted down to slag so now I'm > > > a little shy to scale up further... > > > > Was this with 100 clients each receiving on 10,000 unique addresses? Or > > 1000 clients each receiving on 1000 unique addresses? > > > > 1000 * 1000 > > > On the client-box, was it a cpu issue? a memory issue? something else? > > > > > Unfortunately, I don't know. It became so unresponsive that I couldn't > get a prompt, couldn't run top. It felt like swapping. > > I suspect that it was the messenger 1-link-per-address thing. > > Since I reported earlier that 1 messenger-based sender grew to > 3.4 GB after sending to 30,000 unique addrs, it seems reasonable > that 1000 messenger-based receivers, attempting to receive from a total > of 1,000,000 addrs, would have attempted to grow to a total of more > than 100 GB. Which would account very nicely for the behavior I saw. > ( The box had 45 GB mem. ) > > This kind of extreme scale-up with addresses on a single box is not > what messenger was intended for -- I am retooling to use qpid-messaging > based clients on both sides of the router. > I wouldn't say messenger isn't intended for this sort of scale up. The fundamental issue here if you need to scale is multiplexing a single link with messages to different addresses. This is something you can trivially instruct messenger to do with the appropriate route instruction. This should let you scale up indefinitely on the send side at least. The receive side is a bit different, and using qpid-messaging will not necessarily help you scale up. Fundamentally in order to receive messages from N different addresses you need to create N subscriptions. That's going to be just as expensive regardless of which API you use to do it. The one option you might have with qpid-messaging is that you can close your receivers explicitly, whereas messenger doesn't have a way to cancel subscriptions. This isn't really a matter of intent though, it's just a missing feature. I don't know if cancelling subscriptions will help you much or not, but I'm guessing it would be of similar difficulty (and at least more generally useful) to supply a patch for canceling subscriptions rather than retooling your receivers. --Rafael
