Hello Wes Hardaker,
I have answers to some your questions again:
WH:>> 2) make sure the code is supporting caching. With 4000 entries in the
WH:>> /proc/net/dev file you *don't* want to re-read it frequently.
As earlier said with strace out put as well while attaching snmpd to gdb to
do frequent back tracing i found that cache_load was being called
frequently.
One sample backtrace goes like this:
--------------------------------------------------------------
#0 0x0000005555b74d60 in write () from /lib64/libc.so.6
#1 0x0000005555b1d9dc in _IO_new_file_write () from /lib64/libc.so.6
#2 0x0000005555b1d570 in new_do_write () from /lib64/libc.so.6
#3 0x0000005555b1d970 in _IO_new_do_write () from /lib64/libc.so.6
#4 0x0000005555b1d724 in _IO_new_file_xsputn () from /lib64/libc.so.6
#5 0x0000005555af83f0 in vfprintf () from /lib64/libc.so.6
#6 0x0000005555afe1f4 in fprintf () from /lib64/libc.so.6
#7 0x0000005555835948 in log_handler_file () from
/opt/nokiasiemens/SS_NetSnmp//lib/libnetsnmp.so.15
#8 0x0000005555835bb4 in snmp_log_string () from
/opt/nokiasiemens/SS_NetSnmp//lib/libnetsnmp.so.15
#9 0x0000005555835ccc in snmp_vlog () from
/opt/nokiasiemens/SS_NetSnmp//lib/libnetsnmp.so.15
#10 0x0000005555830b84 in debugmsg () from
/opt/nokiasiemens/SS_NetSnmp//lib/libnetsnmp.so.15
#11 0x0000005555867bc0 in netsnmp_compare_netsnmp_index () from
/opt/nokiasiemens/SS_NetSnmp//lib/libnetsnmp.so.15
#12 0x00000055558682fc in array_qsort () from
/opt/nokiasiemens/SS_NetSnmp//lib/libnetsnmp.so.15
#13 0x00000055558684c0 in array_qsort () from
/opt/nokiasiemens/SS_NetSnmp//lib/libnetsnmp.so.15
#14 0x00000055558684c0 in array_qsort () from
/opt/nokiasiemens/SS_NetSnmp//lib/libnetsnmp.so.15
#15 0x000000555586861c in Sort_Array () from
/opt/nokiasiemens/SS_NetSnmp//lib/libnetsnmp.so.15
#16 0x0000005555869094 in netsnmp_binary_array_get () from
/opt/nokiasiemens/SS_NetSnmp//lib/libnetsnmp.so.15
#17 0x000000555586925c in netsnmp_binary_array_insert () from
/opt/nokiasiemens/SS_NetSnmp//lib/libnetsnmp.so.15
#18 0x00000055558691d0 in _ba_insert () from
/opt/nokiasiemens/SS_NetSnmp//lib/libnetsnmp.so.15
#19 0x00000055557374bc in CONTAINER_INSERT_HELPER () from
/opt/nokiasiemens/SS_NetSnmp//lib/libnetsnmpmibs.so.15
#20 0x00000055557373fc in CONTAINER_INSERT () from
/opt/nokiasiemens/SS_NetSnmp//lib/libnetsnmpmibs.so.15
#21 0x0000005555737238 in _netsnmp_ioctl_ipaddress_container_load_v4 () from
/opt/nokiasiemens/SS_NetSnmp//lib/libnetsnmpmibs.so.15
#22 0x0000005555732e60 in netsnmp_arch_ipaddress_container_load () from
/opt/nokiasiemens/SS_NetSnmp//lib/libnetsnmpmibs.so.15
#23 0x0000005555731bc8 in netsnmp_access_ipaddress_container_load () from
/opt/nokiasiemens/SS_NetSnmp//lib/libnetsnmpmibs.so.15
#24 0x000000555570bb88 in ipAddressTable_container_load () from
/opt/nokiasiemens/SS_NetSnmp//lib/libnetsnmpmibs.so.15
#25 0x000000555570a49c in _cache_load () from
/opt/nokiasiemens/SS_NetSnmp//lib/libnetsnmpmibs.so.15
#26 0x00000055555eb948 in _cache_load () from
/opt/nokiasiemens/SS_NetSnmp//lib/libnetsnmphelpers.so.15
#27 0x00000055555ea574 in _timer_reload () from
/opt/nokiasiemens/SS_NetSnmp//lib/libnetsnmphelpers.so.15
#28 0x0000005555843ae8 in run_alarms () from
/opt/nokiasiemens/SS_NetSnmp//lib/libnetsnmp.so.15
#29 0x0000000120005498 in receive ()
#30 0x00000001200049f0 in main ()
--------------------------------------------------------------
WH:>> 3) Is it possible that parsing is failing? IE, is it possible there
is
WH:>> a bug in the code that is preventing the function from ever
quitting?
WH:>> Does the CPU stay at 100% forever or does it eventually drop back
down?
YES, CPU stays 100% for ever, i think your guess might be right. But as an
observation when i had less than 2000 VLAN's this was not the case it is
used to fluctuate between 10-40-100 but predominently 100%.
WH:>> If it takes too long to fill the cache because of that many entries,
you
WH:>> may want to increase the caching time for that particular code.
But from the grpofile output i attached it does not look like that, ofcourse
i woudl like to introduce cache->timeout in ipAddressTable_initialise().
Let me know.
Thanks very much.
Best Regards,
Bheemesh
On Fri, Nov 20, 2009 at 12:52 PM, bheemesh v <bheem...@gmail.com> wrote:
> Hello Wes Hardaker,
>
> Thanks very much for your valuable inputs here.
> We are starting the snmpd with -f option, so i guess i am going right
> there.
>
> I am attaching a sample gpofile output file to your refrence generated from
> my HW environnment.
> May be this can give better data for investigations.
>
> I will get back on the remaining questions.
> But as an observation for cache->timeout, it is being used for
> ifTable_initialization, but not for ipAddressTable_initialization.
> So cache-> flags are only being used commonly.
>
> Let me know.
>
> Thanks very much.
>
> BR,
> Bheemesh
>
>
>
>
>
> On Thu, Nov 19, 2009 at 7:10 AM, Wes Hardaker <
> harda...@users.sourceforge.net> wrote:
>
>>
>> bv> Has anybody done so far profiling with gprof or any equivalent tool
>> bv> for src code profiling for net-snmp-5.4.2.1 or any prev versions?
>> bv> So far we are finding it tough to generate the gmon.out file for
>> bv> profiling data generation.
>>
>> I have, but it's been a while. In the past I successfully used gprof to
>> profile various aspects of the agent.
>>
>> bv> We have seen reasons being the start directory, multi threaded
>> programs
>> bv> unsuitable for profiling etc.
>>
>> We don't use multithreading. Do make sure you run the agent with -f
>> though.
>>
>> bv> Also so far for this mail chain, no one has given any leads, is it
>> because
>> bv> my question is ambiguos or do i have to give bit more background of
>> info
>> bv> regarding our system setup?
>>
>> Actually, I think you've done a really nice job explaining your
>> problem (much better than our average user). Unfortunately, your
>> problem is also fairly complex which makes it hard to answer quickly
>> (and a large number of developers got very busy with other aspects of
>> their life and jobs all at the same time).
>>
>> > netsnmp_arch_interface_container_load() function from
>> > "if-mib/data_access/interface_linux.c" has a file pointer to
>> "proc/net/dev"
>> > from it tries to get updates of each interface in a while loop, which
>> takes
>> > too much of cpu time when we have many VLAN interfaces defind in our
>> server
>> > this loads CPU 100%.
>>
>> There are a few things I can think of:
>>
>> 1) the code in question isn't well optimized for 4000 vlans and takes a
>> long time to load. I'm actually not sure, but I'd think it would
>> actually be ok.
>> 2) make sure the code is supporting caching. With 4000 entries in the
>> /proc/net/dev file you *don't* want to re-read it frequently.
>> 3) Is it possible that parsing is failing? IE, is it possible there is
>> a bug in the code that is preventing the function from ever quitting?
>> Does the CPU stay at 100% forever or does it eventually drop back down?
>> 4) You should try running a *single* snmpgetnext on the iftable with the
>> following options: "-r 0 -t 600", which will make snmpgetnext wait
>> for a long time and only try once. See if you ever get a response.
>> If you do, rerun the command *immediately* (and hopefully the next
>> request will return more quickly if it's cached).
>>
>> If it takes too long to fill the cache because of that many entries, you
>> may want to increase the caching time for that particular code.
>> --
>> Wes Hardaker
>> Cobham Analytic Solutions
>>
>
>
------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
Net-snmp-users mailing list
Net-snmp-users@lists.sourceforge.net
Please see the following page to unsubscribe or change other options:
https://lists.sourceforge.net/lists/listinfo/net-snmp-users
