We should still be able to do a lot better. I suspect just minimizing the overhead of an empty pn_data_t will help a lot.
--Rafael On Fri, May 2, 2014 at 9:42 AM, Michael Goulish <[email protected]> wrote: > > > That definitely helped! > > I did the same test as yesterday ( with 100 ... 500 addresses per receiver > ). > > Yesterday I saw 115 KB per extra address, today I see 80 KB. > > > That change *might* allow me to attempt a 1,000,000 address test, if I use > my best boxes.... :-) > > > Thanks for the memories! > > > > ----- Original Message ----- > I think a lot of the extra overhead is coming from some overly generous > default allocation sizes that ended up getting used in a lot of places. > I've adjusted these down. If you can retry your testing with trunk I'm > hoping you should see some improvement. > > --Rafael > > > On Fri, May 2, 2014 at 6:29 AM, Rafael Schloming <[email protected]> wrote: > > > On Thu, May 1, 2014 at 8:14 PM, Michael Goulish <[email protected] > >wrote: > > > >> > >> > >> > >> > >> ----- Original Message ----- > >> > On 05/01/2014 08:55 PM, Rafael Schloming wrote: > >> > > On Thu, May 1, 2014 at 3:42 PM, Michael Goulish < > [email protected]> > >> > > wrote: > >> > >> I tried firing up my messenger-based receivers, each subscribing to > >> 100 > >> > >> addresses, then 200, 300, 400, 500. The results are consistent > >> across > >> > >> that range, and show that each extra address costs 115 KB. ( > Looking > >> > >> only at resident-set size. ) > >> > >> > >> > >> So when I tried to do a total of 1,000,000 addrs on one box, I did > >> > >> indeed overwhelm my memory. That would come to 115 GB, which > >> > >> would have been more than double my physical mem. > >> > >> > >> > >> Please note I did not actually send any messages. A router was > >> running > >> > >> for these receivers to attach to, but no senders were running. > >> > >> > >> > >> Does 115 KB per subscribed addr seem fairly reasonable? > >> > >> > >> > > > >> > > No, that seems quite excessive. Can you trace where the memory is > >> actually > >> > > coming from? > >> > > >> > Just for comparison, a qpid::messaging process with 1000 subscriptions > >> > over AMQP 1.0 uses 48MB on my laptop. A similar process using AMQP > 0-10 > >> > uses 8MB. > >> > > >> > > >> > >> Yikes. A 40 KB per subscription added cost. > >> That's because 1.0 is ten times as good as 0.10 . > >> > >> Just off the top of your head -- is there anything inherent > >> in 1.0 that would make you expect that kind of difference? > >> > > > > Definitely not, the overhead should be comparable. > > > > --Rafael > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
