Yes, I can provide the core. How should I share it? I'll see about setting
the (un)summarized_metrics options. Is there a good way to see which are
used or not?
As for rrdcached this is how we have it running:
/usr/bin/rrdcached -l unix:/var/run/rrdcached/sock -j
/var/lib/rrdcached/journal/ -F -P PENDING,FLUSH,STATS,HELP -b
/var/lib/ganglia/rrds -B -p /var/run/rrdcached/pid -g -z 150 -w 1800 -f
3600 -t 12
export RRDCACHED_ADDRESS=unix:/var/run/rrdcached/sock ; /usr/sbin/gmetad -d1
The server that gmetad/rrdcached/the 48 gmond processes (one for each data
source) runs on has dual L5640 processors. Will switch to rrdcached on tcp
also to see if that helps.
-gary
On Thu, Sep 11, 2014 at 6:52 PM, Vladimir Vuksan <vli...@veus.hr> wrote:
> If you can provide the core file of the segfault that would be helpful.
>
> Problem may be with metric summarization where it's locking. One thing to
> do is provide a list of metrics to summarize e.g. I got these options
> turned on in gmetad.conf
>
> summarized_metrics bytes_in bytes_out pkts_in pkts_out cpu_aidle cpu_idle
> cpu_intr cpu_nice cpu_num cpu_sintr cpu_speed cpu_steal cpu_system cpu_user
> cpu_wio
> summarized_metrics mem_buffers mem_cached mem_dirty mem_free mem_mapped
> mem_shared mem_total load_one
>
> as well as metrics that we are never gonna summarize e.g.
>
> unsummarized_metrics varnish rx tx ethtool mysql bird diskstat ipmi tcpext
> tcp et vm ipmi disk icmp procstat icmpmsg ap apssl ip management1
>
> Also can you tell me how your rrdcached is configured ? We have switched
> to TCP e.g. in gmetad.conf we have
>
> rrdcached_address 127.0.0.1:9998
>
> and rrdcached is started via
>
> /usr/bin/rrdcached -t 60 -w 360 -z 360 -F -s ganglia -m 664 -l
> 127.0.0.1:9998 -s ganglia -m 777 -P FLUSH,STATS,HELP -l
> unix:/tmp/rrdcached.limited.sock -b /var/lib/ganglia/rrds -B -p
> /var/lib/ganglia/rrdcached.pid -p /var/lib/ganglia/rrdcached.pid
>
> Let me know if this helps.
>
> Vladimir
> P.S. Our system does over 700k metrics with those settings
>
>
> On 09/10/2014 08:52 PM, Gary Barrueto wrote:
>
> We're seeing gmetad (version 3.7.0) threads segfaulting or gmetad and all
> its threads spinning 100% cpu. The system is running ubuntu 12.04/precise.
> Our ganglia has 48 data source, 1300 hosts and has around 220k metrics. We
> are running this with rrdcached. Would like to fix it from segfault and
> spin on cpu. I'm including a backtrace of when gmetad and threads were
> spinning on cpu.
>
> Is there anything I can provide to help figure this out?
>
> gmetad.conf looks like this:
>
> RRAs "RRA:AVERAGE:0.5:1:5856" "RRA:AVERAGE:0.5:24:1680"
> "RRA:AVERAGE:0.5:168:244" "RRA:AVERAGE:0.5:672:274"
> "RRA:AVERAGE:0.5:5760:3740"
> server_threads 10
> case_sensitive_hostnames 1
>
> and includes 48 data_source tags and each has 2 data sources.
>
> ganglia was ./configured using:
>
> ./configure --host=x86_64-linux-gnu --build=x86_64-linux-gnu
> --prefix=/usr --mandir=${prefix}/share/man --libdir=${prefix}/lib
> --sysconfdir=/etc/ganglia --infodir=${prefix}/share/info --enable-shared
> --with-gmetad --with-memcached --enable-debug
>
> Backtrace:
> GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2.1) 7.4-2012.04
> Copyright (C) 2012 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <
> http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-linux-gnu".
> For bug reporting instructions, please see:
> <http://bugs.launchpad.net/gdb-linaro/>...
> Reading symbols from /usr/sbin/gmetad...done.
> [New LWP 26916]
> [New LWP 26917]
> [New LWP 26918]
> [New LWP 26919]
> [New LWP 26920]
> [New LWP 26921]
> [New LWP 26922]
> [New LWP 26923]
> [New LWP 26924]
> [New LWP 26925]
> [New LWP 26926]
> [New LWP 26927]
> [New LWP 26928]
> [New LWP 26929]
> [New LWP 26930]
> [New LWP 26931]
> [New LWP 26932]
> [New LWP 26933]
> [New LWP 26934]
> [New LWP 26935]
> [New LWP 26936]
> [New LWP 26937]
> [New LWP 26938]
> [New LWP 26939]
> [New LWP 26940]
> [New LWP 26941]
> [New LWP 26942]
> [New LWP 26943]
> [New LWP 26944]
> [New LWP 26946]
> [New LWP 26947]
> [New LWP 26948]
> [New LWP 26949]
> [New LWP 26950]
> [New LWP 26951]
> [New LWP 26952]
> [New LWP 26953]
> [New LWP 26954]
> [New LWP 26955]
> [New LWP 26956]
> [New LWP 26957]
> [New LWP 26958]
> [New LWP 26959]
> [New LWP 26960]
> [New LWP 26961]
> [New LWP 26962]
> [New LWP 26963]
> [New LWP 26964]
> [New LWP 26965]
> [New LWP 26966]
> [New LWP 26967]
> [New LWP 26968]
> [New LWP 26969]
> [New LWP 26970]
> [New LWP 26974]
> [New LWP 26975]
> [New LWP 26976]
> [New LWP 26977]
> [New LWP 26978]
> [New LWP 26979]
> [New LWP 26980]
> [New LWP 26981]
> [New LWP 26982]
> [New LWP 26983]
> [New LWP 26908]
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> Core was generated by `gmetad'.
> #0 __lll_lock_wait () at
> ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:132
> 132 ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S: No such
> file or directory.
> (gdb) thread apply all bt
>
> Thread 65 (LWP 26908):
> #0 0x00007f316bed2436 in ck_pr_stall () at
> /usr/include/gcc/x86_64/ck_pr.h:70
> #1 0x00007f316bed44a8 in ck_rwlock_read_lock (rw=0x1e23df0) at
> /usr/include/ck_rwlock.h:163
> #2 0x00007f316bed556c in hash_foreach (hash=0x1e23420, func=0x4057d1
> <do_root_summary>, arg=0x0) at hash.c:417
> #3 0x00000000004064f5 in main (argc=2, argv=0x7fff547ded18) at
> gmetad.c:586
>
> Thread 64 (LWP 26983):
> #0 0x00007f316bed2533 in ck_pr_fas_uint (target=0x7f30b402c230, v=1) at
> /usr/include/gcc/x86_64/ck_pr.h:115
> #1 0x00007f316bed4376 in ck_rwlock_write_lock (rw=0x7f30b402c230) at
> /usr/include/ck_rwlock.h:115
> #2 0x00007f316bed528f in hash_delete (key=0x7f30b48792e0,
> hash=0x7f30b40f4240) at hash.c:335
> #3 0x00007f316bed4a75 in hash_destroy (hash=0x7f30b40f4240) at hash.c:127
> #4 0x0000000000424d47 in cleanup_source (key=0x7f30b49dd210,
> val=0x7f30b49dd140, arg=0x7f30c77fdca0) at cleanup.c:170
> #5 0x00007f316bed54c0 in hash_walkfrom (hash=0x1e23420, from=0,
> func=0x424c05 <cleanup_source>, arg=0x7f30c77fdca0) at hash.c:400
> #6 0x0000000000424ef1 in cleanup_thread (arg=0x0) at cleanup.c:206
> #7 0x00007f316b1f9e9a in start_thread (arg=0x7f30c77fe700) at
> pthread_create.c:308
> #8 0x00007f316af2731d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #9 0x0000000000000000 in ?? ()
>
> Thread 63 (LWP 26982):
> #0 0x00007f316af20683 in select () at
> ../sysdeps/unix/syscall-template.S:82
> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> #2 0x0000000000409f33 in data_thread (arg=0x1e28950) at data_thread.c:308
> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f30c7fff700) at
> pthread_create.c:308
> #4 0x00007f316af2731d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #5 0x0000000000000000 in ?? ()
>
> Thread 62 (LWP 26981):
> #0 0x00007f316af20683 in select () at
> ../sysdeps/unix/syscall-template.S:82
> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> #2 0x0000000000409f33 in data_thread (arg=0x1e270b0) at data_thread.c:308
> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f30ccff9700) at
> pthread_create.c:308
> #4 0x00007f316af2731d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #5 0x0000000000000000 in ?? ()
>
> Thread 61 (LWP 26980):
> #0 0x00007f316af20683 in select () at
> ../sysdeps/unix/syscall-template.S:82
> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> #2 0x0000000000409f33 in data_thread (arg=0x1e291b0) at
> data_thread.c:308
> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f30cd7fa700) at
> pthread_create.c:308
> #4 0x00007f316af2731d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #5 0x0000000000000000 in ?? ()
>
> Thread 60 (LWP 26979):
> #0 0x00007f316af20683 in select () at
> ../sysdeps/unix/syscall-template.S:82
> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> #2 0x0000000000409f33 in data_thread (arg=0x1e29f30) at data_thread.c:308
> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f30cdffb700) at
> pthread_create.c:308
> #4 0x00007f316af2731d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #5 0x0000000000000000 in ?? ()
>
> Thread 59 (LWP 26978):
> #0 0x00007f316af20683 in select () at
> ../sysdeps/unix/syscall-template.S:82
> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> #2 0x0000000000409f33 in data_thread (arg=0x1e29350) at data_thread.c:308
> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f30ce7fc700) at
> pthread_create.c:308
> #4 0x00007f316af2731d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #5 0x0000000000000000 in ?? ()
>
> Thread 58 (LWP 26977):
> #0 0x00007f316af20683 in select () at
> ../sysdeps/unix/syscall-template.S:82
> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> #2 0x0000000000409f33 in data_thread (arg=0x1e27f70) at data_thread.c:308
> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f30ceffd700) at
> pthread_create.c:308
> #4 0x00007f316af2731d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #5 0x0000000000000000 in ?? ()
>
> Thread 57 (LWP 26976):
> #0 0x00007f316af20683 in select () at
> ../sysdeps/unix/syscall-template.S:82
> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> #2 0x0000000000409f33 in data_thread (arg=0x1e27dd0) at data_thread.c:308
> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f30cf7fe700) at
> pthread_create.c:308
> #4 0x00007f316af2731d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #5 0x0000000000000000 in ?? ()
>
> Thread 56 (LWP 26975):
> #0 0x00007f316af20683 in select () at
> ../sysdeps/unix/syscall-template.S:82
> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> #2 0x0000000000409f33 in data_thread (arg=0x1e2acb0) at data_thread.c:308
> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f30cffff700) at
> pthread_create.c:308
> #4 0x00007f316af2731d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #5 0x0000000000000000 in ?? ()
>
> Thread 55 (LWP 26974):
> #0 0x00007f316af20683 in select () at
> ../sysdeps/unix/syscall-template.S:82
> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> #2 0x0000000000409f33 in data_thread (arg=0x1e2a5f0) at data_thread.c:308
> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f3124ff9700) at
> pthread_create.c:308
> #4 0x00007f316af2731d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #5 0x0000000000000000 in ?? ()
>
> Thread 54 (LWP 26970):
> #0 0x00007f316af20683 in select () at
> ../sysdeps/unix/syscall-template.S:82
> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> #2 0x0000000000409f33 in data_thread (arg=0x1e29a10) at data_thread.c:308
> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f31257fa700) at
> pthread_create.c:308
> #4 0x00007f316af2731d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #5 0x0000000000000000 in ?? ()
>
> Thread 53 (LWP 26969):
> #0 0x00007f316af20683 in select () at
> ../sysdeps/unix/syscall-template.S:82
> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> #2 0x0000000000409f33 in data_thread (arg=0x1e287b0) at data_thread.c:308
> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f3125ffb700) at
> pthread_create.c:308
> #4 0x00007f316af2731d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #5 0x0000000000000000 in ?? ()
>
> Thread 52 (LWP 26968):
> #0 ck_pr_stall () at /usr/include/gcc/x86_64/ck_pr.h:70
> #1 0x00007f316bed4386 in ck_rwlock_write_lock (rw=0x1e23df0) at
> /usr/include/ck_rwlock.h:121
> #2 0x00007f316bed4d9b in hash_insert (key=0x7f31267fa440,
> val=0x7f31267fa450, hash=0x1e23420) at hash.c:210
> #3 0x00000000004135e2 in endElement_CLUSTER (data=0x7f31267fa750,
> el=0x7f30a800e7f0 "CLUSTER") at process_xml.c:1292
> #4 0x00000000004136ba in end (data=0x7f31267fa750, el=0x7f30a800e7f0
> "CLUSTER") at process_xml.c:1332
> #5 0x00007f316b4196f4 in ?? () from /lib/x86_64-linux-gnu/libexpat.so.1
> #6 0x00007f316b41a951 in ?? () from /lib/x86_64-linux-gnu/libexpat.so.1
> #7 0x00007f316b4177c7 in ?? () from /lib/x86_64-linux-gnu/libexpat.so.1
> #8 0x00007f316b41917b in ?? () from /lib/x86_64-linux-gnu/libexpat.so.1
> #9 0x00007f316b41c75d in XML_ParseBuffer () from
> /lib/x86_64-linux-gnu/libexpat.so.1
> #10 0x00000000004137f2 in process_xml (d=0x1e27590,
> buf=0x7f3144778010 "<?xml version=\"1.0\" encoding=\"ISO-8859-1\"
> standalone=\"yes\"?>\n<!DOCTYPE GANGLIA_XML [\n <!ELEMENT GANGLIA_XML
> (GRID|CLUSTER|HOST)*>\n <!ATTLIST GANGLIA_XML VERSION CDATA
> #REQUIRED>\n <!ATTLIST"...) at process_xml.c:1376
> #11 0x0000000000409e67 in data_thread (arg=0x1e27590) at data_thread.c:287
> #12 0x00007f316b1f9e9a in start_thread (arg=0x7f31267fc700) at
> pthread_create.c:308
> #13 0x00007f316af2731d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #14 0x0000000000000000 in ?? ()
>
> Thread 51 (LWP 26967):
> #0 ck_pr_stall () at /usr/include/gcc/x86_64/ck_pr.h:70
> #1 0x00007f316bed44a8 in ck_rwlock_read_lock (rw=0x1e23df0) at
> /usr/include/ck_rwlock.h:163
> #2 0x00007f316bed515f in hash_lookup (key=0x7f3126ffb440, hash=0x1e23420)
> at hash.c:303
> #3 0x0000000000410df7 in startElement_CLUSTER (data=0x7f3126ffb750,
> el=0x7f30b51dd7c0 "CLUSTER", attr=0x7f30b52430a0) at process_xml.c:309
> #4 0x0000000000412f8d in start (data=0x7f3126ffb750, el=0x7f30b51dd7c0
> "CLUSTER", attr=0x7f30b52430a0) at process_xml.c:1106
> #5 0x00007f316b419b5c in ?? () from /lib/x86_64-linux-gnu/libexpat.so.1
> #6 0x00007f316b41a951 in ?? () from /lib/x86_64-linux-gnu/libexpat.so.1
> #7 0x00007f316b4177c7 in ?? () from /lib/x86_64-linux-gnu/libexpat.so.1
> #8 0x00007f316b41917b in ?? () from /lib/x86_64-linux-gnu/libexpat.so.1
> #9 0x00007f316b41c75d in XML_ParseBuffer () from
> /lib/x86_64-linux-gnu/libexpat.so.1
> #10 0x00000000004137f2 in process_xml (d=0x1e26f10,
> buf=0x7f309509c010 "<?xml version=\"1.0\" encoding=\"ISO-8859-1\"
> standalone=\"yes\"?>\n<!DOCTYPE GANGLIA_XML [\n <!ELEMENT GANGLIA_XML
> (GRID|CLUSTER|HOST)*>\n <!ATTLIST GANGLIA_XML VERSION CDATA
> #REQUIRED>\n <!ATTLIST"...) at process_xml.c:1376
> #11 0x0000000000409e67 in data_thread (arg=0x1e26f10) at data_thread.c:287
> #12 0x00007f316b1f9e9a in start_thread (arg=0x7f3126ffd700) at
> pthread_create.c:308
> #13 0x00007f316af2731d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
>
> #14 0x0000000000000000 in ?? ()
>
> Thread 50 (LWP 26966):
> #0 0x00007f316af20683 in select () at
> ../sysdeps/unix/syscall-template.S:82
> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> ---Type <return> to continue, or q <return> to quit---
> #2 0x0000000000409f33 in data_thread (arg=0x1e282d0) at data_thread.c:308
> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f31277fe700) at
> pthread_create.c:308
> #4 0x00007f316af2731d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #5 0x0000000000000000 in ?? ()
>
> Thread 49 (LWP 26965):
> #0 0x00007f316af20683 in select () at
> ../sysdeps/unix/syscall-template.S:82
> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> #2 0x0000000000409f33 in data_thread (arg=0x1e28130) at data_thread.c:308
> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f3127fff700) at
> pthread_create.c:308
> #4 0x00007f316af2731d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #5 0x0000000000000000 in ?? ()
>
> Thread 48 (LWP 26964):
> #0 0x00007f316af20683 in select () at
> ../sysdeps/unix/syscall-template.S:82
> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> #2 0x0000000000409f33 in data_thread (arg=0x1e2b6d0) at data_thread.c:308
> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f3134ff9700) at
> pthread_create.c:308
> #4 0x00007f316af2731d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #5 0x0000000000000000 in ?? ()
>
> Thread 47 (LWP 26963):
> #0 0x00007f316af20683 in select () at
> ../sysdeps/unix/syscall-template.S:82
> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> #2 0x0000000000409f33 in data_thread (arg=0x1e28e50) at data_thread.c:308
> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f31357fa700) at
> pthread_create.c:308
> #4 0x00007f316af2731d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #5 0x0000000000000000 in ?? ()
>
> Thread 46 (LWP 26962):
> #0 0x00007f316af20683 in select () at
> ../sysdeps/unix/syscall-template.S:82
> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> #2 0x0000000000409f33 in data_thread (arg=0x1e2b030) at data_thread.c:308
> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f3135ffb700) at
> pthread_create.c:308
> #4 0x00007f316af2731d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #5 0x0000000000000000 in ?? ()
>
> Thread 45 (LWP 26961):
> #0 0x00007f316af20683 in select () at
> ../sysdeps/unix/syscall-template.S:82
> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> #2 0x0000000000409f33 in data_thread (arg=0x1e28470) at data_thread.c:308
> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f31367fc700) at
> pthread_create.c:308
> #4 0x00007f316af2731d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #5 0x0000000000000000 in ?? ()
>
> Thread 44 (LWP 26960):
> #0 0x00007f316af20683 in select () at
> ../sysdeps/unix/syscall-template.S:82
> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> #2 0x0000000000409f33 in data_thread (arg=0x1e2b530) at data_thread.c:308
> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f3136ffd700) at
> pthread_create.c:308
> #4 0x00007f316af2731d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #5 0x0000000000000000 in ?? ()
>
> Thread 43 (LWP 26959):
> #0 0x00007f316af20683 in select () at
> ../sysdeps/unix/syscall-template.S:82
> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> #2 0x0000000000409f33 in data_thread (arg=0x1e27c30) at data_thread.c:308
> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f31377fe700) at
> pthread_create.c:308
> #4 0x00007f316af2731d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #5 0x0000000000000000 in ?? ()
>
> Thread 42 (LWP 26958):
> #0 0x00007f316af20683 in select () at
> ../sysdeps/unix/syscall-template.S:82
> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> #2 0x0000000000409f33 in data_thread (arg=0x1e29bd0) at data_thread.c:308
> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f3137fff700) at
> pthread_create.c:308
> #4 0x00007f316af2731d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #5 0x0000000000000000 in ?? ()
>
> Thread 41 (LWP 26957):
> #0 0x00007f316af20683 in select () at
> ../sysdeps/unix/syscall-template.S:82
> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> #2 0x0000000000409f33 in data_thread (arg=0x1e2b1f0) at data_thread.c:308
> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f3144ff9700) at
> pthread_create.c:308
> ---Type <return> to continue, or q <return> to quit---
> #4 0x00007f316af2731d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #5 0x0000000000000000 in ?? ()
>
> Thread 40 (LWP 26956):
> #0 0x00007f316af20683 in select () at
> ../sysdeps/unix/syscall-template.S:82
> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> #2 0x0000000000409f33 in data_thread (arg=0x1e2ba10) at data_thread.c:308
> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f31457fa700) at
> pthread_create.c:308
> #4 0x00007f316af2731d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #5 0x0000000000000000 in ?? ()
>
> Thread 39 (LWP 26955):
> #0 0x00007f316af20683 in select () at
> ../sysdeps/unix/syscall-template.S:82
> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> #2 0x0000000000409f33 in data_thread (arg=0x1e27a90) at data_thread.c:308
> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f3145ffb700) at
> pthread_create.c:308
> #4 0x00007f316af2731d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #5 0x0000000000000000 in ?? ()
>
> Thread 38 (LWP 26954):
> #0 0x00007f316af20683 in select () at
> ../sysdeps/unix/syscall-template.S:82
> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> #2 0x0000000000409f33 in data_thread (arg=0x1e2bbb0) at data_thread.c:308
> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f31467fc700) at
> pthread_create.c:308
> #4 0x00007f316af2731d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #5 0x0000000000000000 in ?? ()
>
> Thread 37 (LWP 26953):
> #0 0x00007f316af20683 in select () at
> ../sysdeps/unix/syscall-template.S:82
> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> #2 0x0000000000409f33 in data_thread (arg=0x1e2a2b0) at data_thread.c:308
> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f3146ffd700) at
> pthread_create.c:308
> #4 0x00007f316af2731d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #5 0x0000000000000000 in ?? ()
>
> Thread 36 (LWP 26952):
> #0 0x00007f316af20683 in select () at
> ../sysdeps/unix/syscall-template.S:82
> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> #2 0x0000000000409f33 in data_thread (arg=0x1e296b0) at
> data_thread.c:308
> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f31477fe700) at
> pthread_create.c:308
> #4 0x00007f316af2731d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #5 0x0000000000000000 in ?? ()
>
> Thread 35 (LWP 26951):
> #0 0x00007f316af20683 in select () at
> ../sysdeps/unix/syscall-template.S:82
> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> #2 0x0000000000409f33 in data_thread (arg=0x1e24e10) at data_thread.c:308
> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f3147fff700) at
> pthread_create.c:308
> #4 0x00007f316af2731d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #5 0x0000000000000000 in ?? ()
>
> Thread 34 (LWP 26950):
> #0 0x00007f316af20683 in select () at
> ../sysdeps/unix/syscall-template.S:82
> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> #2 0x0000000000409f33 in data_thread (arg=0x1e28cb0) at data_thread.c:308
> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f314cff9700) at
> pthread_create.c:308
> #4 0x00007f316af2731d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #5 0x0000000000000000 in ?? ()
>
> Thread 33 (LWP 26949):
> #0 0x00007f316af19f1d in read () at ../sysdeps/unix/syscall-template.S:82
> #1 0x00007f316aeadca8 in _IO_new_file_underflow (fp=0x7f314802f2a0) at
> fileops.c:619
> #2 0x00007f316aeaecee in _IO_default_uflow (fp=0x7f314802f2a0) at
> genops.c:440
> #3 0x00007f316aea2e3a in _IO_getline_info (fp=0x7f314802f2a0,
> buf=0x7f314d7f2fc0 "0 errors, enqueued 1 value(s).", n=4095, delim=10,
> extract_delim=1, eof=0x0) at iogetline.c:74
> #4 0x00007f316aea1d1b in _IO_fgets (buf=0x7f314d7f2fc0 "0 errors,
> enqueued 1 value(s).", n=<optimized out>, fp=0x7f314802f2a0) at iofgets.c:58
> #5 0x00007f316bcad64b in fgets (__stream=0x7f314802f2a0, __n=4096,
> __s=0x7f314d7f2fc0 "0 errors, enqueued 1 value(s).")
> at /usr/include/x86_64-linux-gnu/bits/stdio2.h:255
> #6 response_read (ret_response=0x7f314d7f4018) at rrd_client.c:251
> #7 0x00007f316bcad860 in request (ret_response=0x7f314d7f4068,
> buffer=<optimized out>, buffer_size=<optimized out>) at rrd_client.c:325
> #8 request (buffer=<optimized out>, buffer_size=<optimized out>,
> ret_response=0x7f314d7f4068) at rrd_client.c:305
> #9 0x00007f316bcae1d8 in rrdc_update (filename=<optimized out>,
> values_num=1, values=0x7f314d7f71b0) at rrd_client.c:627
> #10 0x00007f316bcb42b4 in rrd_update (argc=<optimized out>,
> argv=0x7f314d7f71a0) at rrd_update.c:452
> #11 0x000000000041627c in RRD_update (rrd=0x7f314d7f7390
> "/var/lib/ganglia/rrds/video/
> video13.jfk01.justin.tv/tcp_retrans_percentage.rrd",
> sum=0x7f30fa09ae93 "0.07773", num=0x0, process_time=1410382058) at
> rrd_helpers.c:212
> #12 0x00000000004166f2 in push_data_to_rrd (rrd=0x7f314d7f7390
> "/var/lib/ganglia/rrds/video/
> video13.jfk01.justin.tv/tcp_retrans_percentage.rrd",
> sum=0x7f30fa09ae93 "0.07773", num=0x0, step=15,
> process_time=1410382058, slope=GANGLIA_SLOPE_BOTH) at rrd_helpers.c:344
> #13 0x0000000000416b6c in write_data_to_rrd (source=0x7f30f8831660
> "video", host=0x7f30f9ff63f0 "video13.jfk01.justin.tv",
> metric=0x7f30fa09ae7c "tcp_retrans_percentage", sum=0x7f30fa09ae93
> "0.07773", num=0x0, step=15, process_time=1410382058,
> slope=GANGLIA_SLOPE_BOTH) at rrd_helpers.c:388
> #14 0x0000000000411e86 in startElement_METRIC (data=0x7f314d7f8750,
> el=0x7f30f8001b80 "METRIC", attr=0x7f30fa09b290) at process_xml.c:733
> #15 0x0000000000412fea in start (data=0x7f314d7f8750, el=0x7f30f8001b80
> "METRIC", attr=0x7f30fa09b290) at process_xml.c:1118
> #16 0x00007f316b419b5c in ?? () from /lib/x86_64-linux-gnu/libexpat.so.1
> #17 0x00007f316b41a951 in ?? () from /lib/x86_64-linux-gnu/libexpat.so.1
> #18 0x00007f316b4177c7 in ?? () from /lib/x86_64-linux-gnu/libexpat.so.1
> #19 0x00007f316b41917b in ?? () from /lib/x86_64-linux-gnu/libexpat.so.1
> #20 0x00007f316b41c75d in XML_ParseBuffer () from
> /lib/x86_64-linux-gnu/libexpat.so.1
> #21 0x00000000004137f2 in process_xml (d=0x1e28610,
> buf=0x7f307d77b010 "<?xml version=\"1.0\" encoding=\"ISO-8859-1\"
> standalone=\"yes\"?>\n<!DOCTYPE GANGLIA_XML [\n <!ELEMENT GANGLIA_XML
> (GRID|CLUSTER|HOST)*>\n <!ATTLIST GANGLIA_XML VERSION CDATA
> #REQUIRED>\n <!ATTLIST"...) at process_xml.c:1376
> #22 0x0000000000409e67 in data_thread (arg=0x1e28610) at data_thread.c:287
> #23 0x00007f316b1f9e9a in start_thread (arg=0x7f314d7fa700) at
> pthread_create.c:308
> #24 0x00007f316af2731d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #25 0x0000000000000000 in ?? ()
>
> Thread 32 (LWP 26948):
> #0 0x00007f316af20683 in select () at
> ../sysdeps/unix/syscall-template.S:82
> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> #2 0x0000000000409f33 in data_thread (arg=0x1e2a0f0) at data_thread.c:308
> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f314dffb700) at
> pthread_create.c:308
> #4 0x00007f316af2731d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #5 0x0000000000000000 in ?? ()
>
> Thread 31 (LWP 26947):
> #0 0x00007f316af20683 in select () at
> ../sysdeps/unix/syscall-template.S:82
> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> #2 0x0000000000409f33 in data_thread (arg=0x1e27750) at data_thread.c:308
> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f314e7fc700) at
> pthread_create.c:308
> #4 0x00007f316af2731d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #5 0x0000000000000000 in ?? ()
>
> Thread 30 (LWP 26946):
> #0 0x00007f316af20683 in select () at
> ../sysdeps/unix/syscall-template.S:82
> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> #2 0x0000000000409f33 in data_thread (arg=0x1e2a450) at data_thread.c:308
> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f314effd700) at
> pthread_create.c:308
> #4 0x00007f316af2731d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #5 0x0000000000000000 in ?? ()
>
> Thread 29 (LWP 26944):
> #0 0x00007f316af20683 in select () at
> ../sysdeps/unix/syscall-template.S:82
> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> #2 0x0000000000409f33 in data_thread (arg=0x1e2a790) at data_thread.c:308
> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f314f7fe700) at
> pthread_create.c:308
> #4 0x00007f316af2731d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #5 0x0000000000000000 in ?? ()
>
> Thread 28 (LWP 26943):
> #0 0x00007f316af20683 in select () at
> ../sysdeps/unix/syscall-template.S:82
> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> #2 0x0000000000409f33 in data_thread (arg=0x1e29870) at data_thread.c:308
> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f314ffff700) at
> pthread_create.c:308
> #4 0x00007f316af2731d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #5 0x0000000000000000 in ?? ()
>
> Thread 27 (LWP 26942):
> #0 0x00007f316af20683 in select () at
> ../sysdeps/unix/syscall-template.S:82
> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> #2 0x0000000000409f33 in data_thread (arg=0x1e2b390) at data_thread.c:308
> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f3154ff9700) at
> pthread_create.c:308
> #4 0x00007f316af2731d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #5 0x0000000000000000 in ?? ()
>
> Thread 26 (LWP 26941):
> #0 0x00007f316af20683 in select () at
> ../sysdeps/unix/syscall-template.S:82
> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> #2 0x0000000000409f33 in data_thread (arg=0x1e2aaf0) at data_thread.c:308
> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f31557fa700) at
> pthread_create.c:308
> #4 0x00007f316af2731d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #5 0x0000000000000000 in ?? ()
>
> Thread 25 (LWP 26940):
> #0 0x00007f316af20683 in select () at
> ../sysdeps/unix/syscall-template.S:82
> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> #2 0x0000000000409f33 in data_thread (arg=0x1e2a950) at data_thread.c:308
> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f3155ffb700) at
> pthread_create.c:308
> #4 0x00007f316af2731d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #5 0x0000000000000000 in ?? ()
>
> Thread 24 (LWP 26939):
> #0 0x00007f316af20683 in select () at
> ../sysdeps/unix/syscall-template.S:82
> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> #2 0x0000000000409f33 in data_thread (arg=0x1e2b870) at data_thread.c:308
> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f31567fc700) at
> pthread_create.c:308
> #4 0x00007f316af2731d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #5 0x0000000000000000 in ?? ()
>
> Thread 23 (LWP 26938):
> #0 0x00007f316af20683 in select () at
> ../sysdeps/unix/syscall-template.S:82
> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> #2 0x0000000000409f33 in data_thread (arg=0x1e294f0) at data_thread.c:308
> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f3156ffd700) at
> pthread_create.c:308
> #4 0x00007f316af2731d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #5 0x0000000000000000 in ?? ()
>
> Thread 22 (LWP 26937):
> #0 0x00007f316af20683 in select () at
> ../sysdeps/unix/syscall-template.S:82
> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> #2 0x0000000000409f33 in data_thread (arg=0x1e27250) at data_thread.c:308
> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f31577fe700) at
> pthread_create.c:308
> #4 0x00007f316af2731d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #5 0x0000000000000000 in ?? ()
>
> Thread 21 (LWP 26936):
> #0 0x00007f316af20683 in select () at
> ../sysdeps/unix/syscall-template.S:82
> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> #2 0x0000000000409f33 in data_thread (arg=0x1e28b10) at
> data_thread.c:308
> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f3157fff700) at
> pthread_create.c:308
> #4 0x00007f316af2731d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #5 0x0000000000000000 in ?? ()
>
> Thread 20 (LWP 26935):
> #0 0x00007f316af20683 in select () at
> ../sysdeps/unix/syscall-template.S:82
> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> #2 0x0000000000409f33 in data_thread (arg=0x1e273f0) at data_thread.c:308
> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f315ca6a700) at
> pthread_create.c:308
> #4 0x00007f316af2731d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #5 0x0000000000000000 in ?? ()
>
> Thread 19 (LWP 26934):
> #0 0x00007f316af20683 in select () at
> ../sysdeps/unix/syscall-template.S:82
> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> #2 0x0000000000409f33 in data_thread (arg=0x1e29d70) at data_thread.c:308
> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f315d26b700) at
> pthread_create.c:308
> #4 0x00007f316af2731d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #5 0x0000000000000000 in ?? ()
>
> Thread 18 (LWP 26933):
> #0 0x00007f316af20683 in select () at
> ../sysdeps/unix/syscall-template.S:82
> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> #2 0x0000000000409f33 in data_thread (arg=0x1e2ae70) at data_thread.c:308
> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f315da6c700) at
> pthread_create.c:308
> #4 0x00007f316af2731d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #5 0x0000000000000000 in ?? ()
>
> Thread 17 (LWP 26932):
> #0 0x00007f316af20683 in select () at
> ../sysdeps/unix/syscall-template.S:82
> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> #2 0x0000000000409f33 in data_thread (arg=0x1e29010) at data_thread.c:308
> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f315e26d700) at
> pthread_create.c:308
> #4 0x00007f316af2731d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #5 0x0000000000000000 in ?? ()
>
> Thread 16 (LWP 26931):
> #0 0x00007f316af20683 in select () at
> ../sysdeps/unix/syscall-template.S:82
> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> #2 0x0000000000409f33 in data_thread (arg=0x1e278f0) at data_thread.c:308
> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f315ea6e700) at
> pthread_create.c:308
> #4 0x00007f316af2731d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #5 0x0000000000000000 in ?? ()
>
> Thread 15 (LWP 26930):
> #0 ck_pr_stall () at /usr/include/gcc/x86_64/ck_pr.h:70
> #1 0x00007f316bed44a8 in ck_rwlock_read_lock (rw=0x1e23df0) at
> /usr/include/ck_rwlock.h:163
> #2 0x00007f316bed556c in hash_foreach (hash=0x1e23420, func=0x40d313
> <tree_report>, arg=0x7f315f26ec80) at hash.c:417
> #3 0x000000000040d463 in process_path (client=0x7f315f26ec80, path=0x0,
> myroot=0x7f315f26e440, key=0x0) at server.c:435
> #4 0x000000000040d51e in process_path (client=0x7f315f26ec80,
> path=0x7f315f26e470 "/", myroot=0x7f315f26e440, key=0x0) at server.c:452
> #5 0x000000000040e009 in server_thread (arg=0x1) at server.c:796
> #6 0x00007f316b1f9e9a in start_thread (arg=0x7f315f26f700) at
> pthread_create.c:308
> #7 0x00007f316af2731d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #8 0x0000000000000000 in ?? ()
>
> Thread 14 (LWP 26929):
> #0 0x00007f316bed262e in ck_pr_load_uint (target=0x1e23df0) at
> /usr/include/gcc/x86_64/ck_pr.h:145
> #1 0x00007f316bed44b4 in ck_rwlock_read_lock (rw=0x1e23df0) at
> /usr/include/ck_rwlock.h:162
> #2 0x00007f316bed556c in hash_foreach (hash=0x1e23420, func=0x40d313
> <tree_report>, arg=0x7f315fa6fc80) at hash.c:417
> #3 0x000000000040d463 in process_path (client=0x7f315fa6fc80, path=0x0,
> myroot=0x7f315fa6f440, key=0x0) at server.c:435
> #4 0x000000000040d51e in process_path (client=0x7f315fa6fc80,
> path=0x7f315fa6f470 "/", myroot=0x7f315fa6f440, key=0x0) at server.c:452
> #5 0x000000000040e009 in server_thread (arg=0x1) at server.c:796
> #6 0x00007f316b1f9e9a in start_thread (arg=0x7f315fa70700) at
> pthread_create.c:308
> #7 0x00007f316af2731d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #8 0x0000000000000000 in ?? ()
>
> Thread 13 (LWP 26928):
> #0 0x00007f316bed2436 in ck_pr_stall () at
> /usr/include/gcc/x86_64/ck_pr.h:70
> #1 0x00007f316bed44a8 in ck_rwlock_read_lock (rw=0x1e23df0) at
> /usr/include/ck_rwlock.h:163
> #2 0x00007f316bed556c in hash_foreach (hash=0x1e23420, func=0x40d313
> <tree_report>, arg=0x7f3160270c80) at hash.c:417
> #3 0x000000000040d463 in process_path (client=0x7f3160270c80, path=0x0,
> myroot=0x7f3160270440, key=0x0) at server.c:435
> #4 0x000000000040d51e in process_path (client=0x7f3160270c80,
> path=0x7f3160270470 "/", myroot=0x7f3160270440, key=0x0) at server.c:452
> #5 0x000000000040e009 in server_thread (arg=0x1) at server.c:796
> #6 0x00007f316b1f9e9a in start_thread (arg=0x7f3160271700) at
> pthread_create.c:308
> #7 0x00007f316af2731d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #8 0x0000000000000000 in ?? ()
>
> Thread 12 (LWP 26927):
> #0 0x00007f316bed262e in ck_pr_load_uint (target=0x1e23df0) at
> /usr/include/gcc/x86_64/ck_pr.h:145
> #1 0x00007f316bed44b4 in ck_rwlock_read_lock (rw=0x1e23df0) at
> /usr/include/ck_rwlock.h:162
> #2 0x00007f316bed556c in hash_foreach (hash=0x1e23420, func=0x40d313
> <tree_report>, arg=0x7f3160a71c80) at hash.c:417
> #3 0x000000000040d463 in process_path (client=0x7f3160a71c80, path=0x0,
> myroot=0x7f3160a71440, key=0x0) at server.c:435
> #4 0x000000000040d51e in process_path (client=0x7f3160a71c80,
> path=0x7f3160a71470 "/", myroot=0x7f3160a71440, key=0x0) at server.c:452
> #5 0x000000000040e009 in server_thread (arg=0x1) at server.c:796
> #6 0x00007f316b1f9e9a in start_thread (arg=0x7f3160a72700) at
> pthread_create.c:308
> #7 0x00007f316af2731d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #8 0x0000000000000000 in ?? ()
>
> Thread 11 (LWP 26926):
> #0 ck_rwlock_read_lock (rw=0x1e23df0) at /usr/include/ck_rwlock.h:163
> #1 0x00007f316bed556c in hash_foreach (hash=0x1e23420, func=0x40d313
> <tree_report>, arg=0x7f3161272c80) at hash.c:417
> #2 0x000000000040d463 in process_path (client=0x7f3161272c80, path=0x0,
> myroot=0x7f3161272440, key=0x0) at server.c:435
> #3 0x000000000040d51e in process_path (client=0x7f3161272c80,
> path=0x7f3161272470 "/", myroot=0x7f3161272440, key=0x0) at server.c:452
> #4 0x000000000040e009 in server_thread (arg=0x1) at server.c:796
> #5 0x00007f316b1f9e9a in start_thread (arg=0x7f3161273700) at
> pthread_create.c:308
> #6 0x00007f316af2731d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #7 0x0000000000000000 in ?? ()
>
> Thread 10 (LWP 26925):
> #0 0x00007f316bed44b4 in ck_rwlock_read_lock (rw=0x1e23df0) at
> /usr/include/ck_rwlock.h:162
> #1 0x00007f316bed556c in hash_foreach (hash=0x1e23420, func=0x40d313
> <tree_report>, arg=0x7f3161a73c80) at hash.c:417
> #2 0x000000000040d463 in process_path (client=0x7f3161a73c80, path=0x0,
> myroot=0x7f3161a73440, key=0x0) at server.c:435
> #3 0x000000000040d51e in process_path (client=0x7f3161a73c80,
> path=0x7f3161a73470 "/", myroot=0x7f3161a73440, key=0x0) at server.c:452
> #4 0x000000000040e009 in server_thread (arg=0x1) at server.c:796
> #5 0x00007f316b1f9e9a in start_thread (arg=0x7f3161a74700) at
> pthread_create.c:308
> #6 0x00007f316af2731d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #7 0x0000000000000000 in ?? ()
>
> Thread 9 (LWP 26924):
> #0 0x00007f316bed44ac in ck_rwlock_read_lock (rw=0x1e23df0) at
> /usr/include/ck_rwlock.h:162
> #1 0x00007f316bed556c in hash_foreach (hash=0x1e23420, func=0x40d313
> <tree_report>, arg=0x7f3162274c80) at hash.c:417
> #2 0x000000000040d463 in process_path (client=0x7f3162274c80, path=0x0,
> myroot=0x7f3162274440, key=0x0) at server.c:435
> #3 0x000000000040d51e in process_path (client=0x7f3162274c80,
> path=0x7f3162274470 "/", myroot=0x7f3162274440, key=0x0) at server.c:452
> #4 0x000000000040e009 in server_thread (arg=0x1) at server.c:796
> #5 0x00007f316b1f9e9a in start_thread (arg=0x7f3162275700) at
> pthread_create.c:308
> #6 0x00007f316af2731d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #7 0x0000000000000000 in ?? ()
>
> Thread 8 (LWP 26923):
> #0 0x00007f316bed262e in ck_pr_load_uint (target=0x1e23df0) at
> /usr/include/gcc/x86_64/ck_pr.h:145
> #1 0x00007f316bed44b4 in ck_rwlock_read_lock (rw=0x1e23df0) at
> /usr/include/ck_rwlock.h:162
> #2 0x00007f316bed556c in hash_foreach (hash=0x1e23420, func=0x40d313
> <tree_report>, arg=0x7f3162a75c80) at hash.c:417
> #3 0x000000000040d463 in process_path (client=0x7f3162a75c80, path=0x0,
> myroot=0x7f3162a75440, key=0x0) at server.c:435
> #4 0x000000000040d51e in process_path (client=0x7f3162a75c80,
> path=0x7f3162a75470 "/", myroot=0x7f3162a75440, key=0x0) at server.c:452
> #5 0x000000000040e009 in server_thread (arg=0x1) at server.c:796
> #6 0x00007f316b1f9e9a in start_thread (arg=0x7f3162a76700) at
> pthread_create.c:308
> #7 0x00007f316af2731d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #8 0x0000000000000000 in ?? ()
>
> Thread 7 (LWP 26922):
> #0 0x00007f316bed262e in ck_pr_load_uint (target=0x1e23df0) at
> /usr/include/gcc/x86_64/ck_pr.h:145
> #1 0x00007f316bed44b4 in ck_rwlock_read_lock (rw=0x1e23df0) at
> /usr/include/ck_rwlock.h:162
> #2 0x00007f316bed556c in hash_foreach (hash=0x1e23420, func=0x40d313
> <tree_report>, arg=0x7f3163276c80) at hash.c:417
> #3 0x000000000040d463 in process_path (client=0x7f3163276c80, path=0x0,
> myroot=0x7f3163276440, key=0x0) at server.c:435
> #4 0x000000000040d51e in process_path (client=0x7f3163276c80,
> path=0x7f3163276470 "/", myroot=0x7f3163276440, key=0x0) at server.c:452
> #5 0x000000000040e009 in server_thread (arg=0x1) at server.c:796
> #6 0x00007f316b1f9e9a in start_thread (arg=0x7f3163277700) at
> pthread_create.c:308
> #7 0x00007f316af2731d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #8 0x0000000000000000 in ?? ()
>
> Thread 6 (LWP 26921):
> #0 0x00007f316bed2631 in ck_pr_load_uint (target=0x1e23df0) at
> /usr/include/gcc/x86_64/ck_pr.h:145
> #1 0x00007f316bed44b4 in ck_rwlock_read_lock (rw=0x1e23df0) at
> /usr/include/ck_rwlock.h:162
> #2 0x00007f316bed556c in hash_foreach (hash=0x1e23420, func=0x40d313
> <tree_report>, arg=0x7f3163a77c80) at hash.c:417
> #3 0x000000000040d463 in process_path (client=0x7f3163a77c80, path=0x0,
> myroot=0x7f3163a77440, key=0x0) at server.c:435
> #4 0x000000000040d51e in process_path (client=0x7f3163a77c80,
> path=0x7f3163a77470 "/", myroot=0x7f3163a77440, key=0x0) at server.c:452
> #5 0x000000000040e009 in server_thread (arg=0x1) at server.c:796
> #6 0x00007f316b1f9e9a in start_thread (arg=0x7f3163a78700) at
> pthread_create.c:308
> #7 0x00007f316af2731d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #8 0x0000000000000000 in ?? ()
>
> Thread 5 (LWP 26920):
> #0 __lll_lock_wait () at
> ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:132
> #1 0x00007f316b1fc065 in _L_lock_858 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #2 0x00007f316b1fbeba in __pthread_mutex_lock (mutex=0x644200) at
> pthread_mutex_lock.c:61
> #3 0x000000000040dd3f in server_thread (arg=0x0) at server.c:723
> #4 0x00007f316b1f9e9a in start_thread (arg=0x7f3164279700) at
> pthread_create.c:308
> #5 0x00007f316af2731d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #6 0x0000000000000000 in ?? ()
>
> Thread 4 (LWP 26919):
> #0 0x00007f316b20104d in accept () at
> ../sysdeps/unix/syscall-template.S:82
> #1 0x000000000040dd64 in server_thread (arg=0x0) at server.c:724
> #2 0x00007f316b1f9e9a in start_thread (arg=0x7f3164a7a700) at
> pthread_create.c:308
> #3 0x00007f316af2731d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #4 0x0000000000000000 in ?? ()
>
> Thread 3 (LWP 26918):
> #0 __lll_lock_wait () at
> ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:132
> #1 0x00007f316b1fc065 in _L_lock_858 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #2 0x00007f316b1fbeba in __pthread_mutex_lock (mutex=0x644200) at
> pthread_mutex_lock.c:61
> #3 0x000000000040dd3f in server_thread (arg=0x0) at server.c:723
> #4 0x00007f316b1f9e9a in start_thread (arg=0x7f316527b700) at
> pthread_create.c:308
> #5 0x00007f316af2731d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #6 0x0000000000000000 in ?? ()
>
> Thread 2 (LWP 26917):
> #0 __lll_lock_wait () at
> ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:132
> #1 0x00007f316b1fc065 in _L_lock_858 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #2 0x00007f316b1fbeba in __pthread_mutex_lock (mutex=0x644200) at
> pthread_mutex_lock.c:61
> #3 0x000000000040dd3f in server_thread (arg=0x0) at server.c:723
> #4 0x00007f316b1f9e9a in start_thread (arg=0x7f3165a7c700) at
> pthread_create.c:308
> #5 0x00007f316af2731d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #6 0x0000000000000000 in ?? ()
>
> Thread 1 (LWP 26916):
> #0 __lll_lock_wait () at
> ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:132
> #1 0x00007f316b1fc065 in _L_lock_858 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #2 0x00007f316b1fbeba in __pthread_mutex_lock (mutex=0x644200) at
> pthread_mutex_lock.c:61
> #3 0x000000000040dd3f in server_thread (arg=0x0) at server.c:723
> #4 0x00007f316b1f9e9a in start_thread (arg=0x7f316627d700) at
> pthread_create.c:308
> #5 0x00007f316af2731d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #6 0x0000000000000000 in ?? ()
> (gdb)
>
> (gdb) quit
>
>
>
> ------------------------------------------------------------------------------
> Want excitement?
> Manually upgrade your production database.
> When you want reliability, choose Perforce
> Perforce version control. Predictably
> reliable.http://pubads.g.doubleclick.net/gampad/clk?id=157508191&iu=/4140/ostg.clktrk
>
>
>
> _______________________________________________
> Ganglia-developers mailing
> listGanglia-developers@lists.sourceforge.nethttps://lists.sourceforge.net/lists/listinfo/ganglia-developers
>
>
>
>
> ------------------------------------------------------------------------------
> Want excitement?
> Manually upgrade your production database.
> When you want reliability, choose Perforce
> Perforce version control. Predictably reliable.
>
> http://pubads.g.doubleclick.net/gampad/clk?id=157508191&iu=/4140/ostg.clktrk
> _______________________________________________
> Ganglia-developers mailing list
> Ganglia-developers@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/ganglia-developers
>
>
------------------------------------------------------------------------------
Want excitement?
Manually upgrade your production database.
When you want reliability, choose Perforce
Perforce version control. Predictably reliable.
http://pubads.g.doubleclick.net/gampad/clk?id=157508191&iu=/4140/ostg.clktrk
_______________________________________________
Ganglia-developers mailing list
Ganglia-developers@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ganglia-developers
