On 3/17/2013 5:56 PM, Jamie Nguyen wrote:
On 17/03/13 22:50, Alan Robertson wrote:
Copious Spare Time or not - your packages are much appreciated!
Thanks!
If you had a stack trace that would be wonderful. That's a weird
message - because that means that that address has never been allocated
to any object before - or that I failed to tell the object system about
it. Usually it's a "use-after-free" problem. But ConfigValues aren't
first-class objects - so maybe that is somehow causing the problem.
Weird... If the nanoprobe is printing that message, then it crashed
too. The nanoprobe is _far_ more likely to produce a usable stack trace
than the python code in the CMA. Giving either one a -f option will
keep it in the foreground and keep it from forking - so you can run it
under gdb - which tends to give better results than looking at a core file.
From looking at the stack traceback, it looks odder and odder... It is
trying to free something that was never an object at all...
#2 0x00007fc19e00837a in IA__g_logv (log_domain=0x0, log_level=<value optimized
out>, format=
0x7fc19e5005a8 "Attempt to cast %s pointer at address %p to %s (formerly a
%s)", args1=0x7fffbeb88450) at gmessages.c:549
#3 0x00007fc19e008413 in IA__g_log (log_domain=<value optimized out>, log_level=<value
optimized out>, format=<value optimized out>) at gmessages.c:569
#4 0x00007fc19e4ecbb2 in proj_class_castas (object=0x1f38020, castclass=0x7fc19e50348d
"ConfigValue")
at
/usr/src/debug/Experimental-Assimilation-v0.1.0-RC1/clientlib/proj_classes.c:292
#5 0x00007fc19e4f76cd in _configcontext_value_finalize (vself=<value optimized
out>)
at
/usr/src/debug/Experimental-Assimilation-v0.1.0-RC1/clientlib/configcontext.c:428
#6 0x00007fc19e4ec846 in g_slist_free_full (list=0x1f38020 = {...},
free_func=0x7fc19e4f76c0 <_configcontext_value_finalize>)
at
/usr/src/debug/Experimental-Assimilation-v0.1.0-RC1/clientlib/proj_classes.c:439
#7 0x00007fc19e4f7728 in _configcontext_value_finalize (vself=<value optimized
out>)
at
/usr/src/debug/Experimental-Assimilation-v0.1.0-RC1/clientlib/configcontext.c:447
#8 0x00007fc19dff005e in g_hash_table_remove_all_nodes (hash_table=0x1f34850 =
{...}, notify=1) at ghash.c:374
#9 0x00007fc19dff04cd in IA__g_hash_table_remove_all (hash_table=0x1f34850 =
{...}) at ghash.c:1052
#10 0x00007fc19dff0565 in IA__g_hash_table_destroy (hash_table=0x1f34850 =
{...}) at ghash.c:759
#11 0x00007fc19e4f7d5e in _configcontext_finalize (aself=<value optimized out>)
at /usr/src/debug/Experimental-Assimilation-v0.1.0-RC1/clientlib/configcontext.c:135
#12 0x00007fc19e4f76fa in _configcontext_value_finalize (vself=<value optimized
out>)
at
/usr/src/debug/Experimental-Assimilation-v0.1.0-RC1/clientlib/configcontext.c:442
#13 0x00007fc19dff005e in g_hash_table_remove_all_nodes (hash_table=0x1f348f0 =
{...}, notify=1) at ghash.c:374
#14 0x00007fc19dff04cd in IA__g_hash_table_remove_all (hash_table=0x1f348f0 =
{...}) at ghash.c:1052
#15 0x00007fc19dff0565 in IA__g_hash_table_destroy (hash_table=0x1f348f0 =
{...}) at ghash.c:759
#16 0x00007fc19e4f7d5e in _configcontext_finalize (aself=<value optimized out>)
at /usr/src/debug/Experimental-Assimilation-v0.1.0-RC1/clientlib/configcontext.c:135
#17 0x00007fc19e4f76fa in _configcontext_value_finalize (vself=<value optimized
out>)
at
/usr/src/debug/Experimental-Assimilation-v0.1.0-RC1/clientlib/configcontext.c:442
#18 0x00007fc19dff005e in g_hash_table_remove_all_nodes (hash_table=0x1f348a0 =
{...}, notify=1) at ghash.c:374
#19 0x00007fc19dff04cd in IA__g_hash_table_remove_all (hash_table=0x1f348a0 =
{...}) at ghash.c:1052
#20 0x00007fc19dff0565 in IA__g_hash_table_destroy (hash_table=0x1f348a0 =
{...}) at ghash.c:759
#21 0x00007fc19e4f7d5e in _configcontext_finalize (aself=<value optimized out>)
at /usr/src/debug/Experimental-Assimilation-v0.1.0-RC1/clientlib/configcontext.c:135
It was likely walking through a JSON object that was pretty deeply nested
structure - 3 layers of hash tables, and what I would guess was an array. This
could be a case of code I hadn't tested.
This is likely the result of processing your discovery data.
nanoprobe appears to be just mirroring the error message from cma, as it
doesn't crash itself. nanoprobe seems to just carry on. Here are some logs:
https://jamielinux.com/pub/stderr-nanoprobe.txt
Well... There's something odd about your logging configuration.
Message from syslogd@test at Mar 17 23:49:47 ...
"root@test:~" 23:49 17-Mar-13
cma DEBUG: DroneInfo.logjson: JSON text for
test.default.jnguyen.uk0.bigv.io/netconfig already processed - ignoring.
Both the CMA and the nanoprobe send everything of importance goes to
syslog. There's some dupication to stdout/stderr - but everything of
interest should also go to syslog. That kind of message above comes
from something like wall. I think you can configure syslog to wall out
various messages (but why?)
You didn't send the messages fromthe CMA. Seeing it run with 3 -d
options should prove insightful. (Fewer might be enough, but 3 would
certainly do it). For reproducing it, it would be cool if you could
send me the outputs of these commands on whatever machine was running
the nanoprobe code:
/usr/share/assimilation/discovery_agents/tcplisteners
/usr/share/assimilation/discovery_agents/tcpclients
/usr/share/assimilation/discovery_agents/cpu
/usr/share/assimilation/discovery_agents/os
/usr/share/assimilation/discovery_agents/arpcache
DroneInfo.logjson: JSON text for test.default.jnguyen.uk0.bigv.io/arpcache
already processed - ignoring.
DroneInfo.logjson: JSON text for test.default.jnguyen.uk0.bigv.io/OS already
processed - ignoring.
Processing tcpclients JSON data from test.default.jnguyen.uk0.bigv.io into
graph.
Processed tcpclients JSON data from test.default.jnguyen.uk0.bigv.io into graph.
I would guess it was either the tcpclients or the tcplisteners JSON
that triggered the crash (but could also be the cpu command). It isn't
likely to be the arpcache or OS command.
(Another odd thing is that both cma and nanoprobe output debug messages
by default when on EPEL 6 even without passing "-d" option.)
I'll look into those particular messages. I could easily have missed
the if(DEBUG) or equivalent...
Thanks!
-- Alan Robertson
[email protected]
_______________________________________________
Assimilation mailing list - Discovery-Driven Monitoring
[email protected]
http://lists.community.tummy.com/cgi-bin/mailman/listinfo/assimilation
http://assimmon.org/
