That ought to be interesting. I shall do the profiling and perhaps take a look at the code too. Please do send me the script.
Thanks gs On Mon, Mar 16, 2009 at 6:09 AM, Carl Trieloff <[email protected]>wrote: > > I don't believe any profiling has been done on the headers exchange. If you > are on Linux, run oprofile on it > and either create a patch to correct the hot spot or mail the profile to > the list. > > if needed I can also give mail you a stap (system tap) script for lock > analysis for the headers exchange. > regards, > Carl. > > > > GS.Chandra N wrote: > > Gordon, Tedd > > After the patches to the headersexchange.cpp was applied, i can now clearly > see the incoming rates with the help of the tool Tedd sent. > > The rates are at about 90-100 odd messages per sec (a drop from 5200 odd > without subscriptions). > > Sort of matches my earlier observations using qpid-tool. What next? > > Do you think there might be any way to add the headers exchange performance > improvements to your list of priority fixes? I'm not on the dev forums so i > 'm not sure if the issue is communicated out there. > > I will try to test the multiple subscriptions XML exchange next to if that > improves matters in any respect. > > Thanks for the great support in fully un-reavelling the issue. > > Rgds > gs > > ps : I can also still reproduce the memory pile up by blocking the broker > using qpid-tool and setting the mgmgt-pub interval to 1 which of-course i > can avoid by not letting qpid-tool stick around and increasing the mgmt-pub > interval to 10 or more. > > On Thu, Mar 12, 2009 at 3:45 PM, GS.Chandra N <[email protected]>wrote: > >> Hi, >> >> I found that if i kept the qpid-tool open long enough all my clients and >> publishers would throw exceptions and come out. >> >> Earlier when i was performing my tests i had 3 putty windows opened to all >> the 3 servers and it seems as if, i was keeping qpid-tool opened just long >> enough to cause build up but closing it to run top and so on a so forth such >> that the memory piled up but did not cause timeouts at the client either. >> >> Observations >> >> 1.When there are subscriptions and a qpid-tool around messages seem to >> pile up. Why is this so? * >> * >> 2. When there are no subscribers and no qpid-tool the publishing processes >> cause the CPu to go to 90% at the publishing box. But when i started up >> subscribers, the publishing box CPU fell to 0% and all the python processes >> piled up memory. >> >> Concern : I'm not able to find out here if the publishers are still able >> to send out messages at the speed that it should when subscriptions are >> around. Or perhaps the broker is not able to pull enough messages - iam not >> sure which of this is happeneing. Probably latter? How do i tell? >> >> If i use qpid-tool to connect to the broker at this point, every time i >> hit show exchange i get the same stats with the same time stamp. it looks >> like the broker is too busy trying to match subscriptions even to refresh >> the stats. >> >> 3.Concern : Once the subscriptions are made, the CPU at the broker box* >> *(high >> end dual core XEON with ht) goes to 90%. Even at this level, i'm not sure >> the broker is able to match and discard all the messages fast enough (due to >> 2nd observation). Or how do i tell? >> >> >> Thanks >> gs >> >> ps : Is there a sure shot way to find out the message rates ? (I currently >> use qpid-tool show exchange command to find the no of msgRecieves and divide >> by time-dfference from 2 times the command is run) >> >> On Thu, Mar 12, 2009 at 3:11 PM, GS.Chandra N <[email protected]>wrote: >> >>> I'm able to reproduce the issue, when i keep qpid-tool open all the time >>> with the mgmt switches. >>> >>> However i'm not able to do so if qpid-tool is not open (did not try with >>> qpid-tool and no mgmt switches). >>> >>> Thanks >>> gs >>> >>> On Thu, Mar 12, 2009 at 1:31 AM, Gordon Sim <[email protected]> wrote: >>> >>>> GS.Chandra N wrote: >>>> >>>>> ps : Please find attached the scripts that create the issue >>>>> >>>> >>>> I'm afraid I wasn't able to recreate the issue with your tests. Memory >>>> stayed reasonable and constant as the test was running (5 servers, 5 >>>> clients). >>>> >>>> I'm baffled at this point as to what it is you are seeing. >>>> >>>> broker - runs on a dedicated box >>>>> qpidd -d --port=5672 --mgmt-enable=yes --mgmt-pub-interval=1 >>>>> --auth=no --log-source=yes --log-to-file=/tmp/somename >>>>> >>>> >>>> Do you have qpid-tool or anything else running during the test? Does >>>> turning management off have any impact on the issue? >>>> >>>> >>>> --------------------------------------------------------------------- >>>> Apache Qpid - AMQP Messaging Implementation >>>> Project: http://qpid.apache.org >>>> Use/Interact: mailto:[email protected] >>>> >>>> >>> >> > >
