Hi all,
I'm running ganglia on a Debian node and a Solaris 8 x86 node to test
it, and I'm running into problems on the Solaris node. The Debian node
is currently the one running gmetad.
When I pring gconfd up on the solaris node, it works fine for a few
minutes, but very shortly it is reported as being down by the web
interface, even though it is very clearly not down and the process is
still running.
When I run the daemon with any kind of debugging, this problem never
occurs, but I get the following output in -d 1:
cpustuff: Permission denied
cpustuff: Permission denied
cpustuff: Permission denied
cpustuff: Permission denied
cpustuff: Permission denied
cpustuff: Permission denied
cpustuff: Not enough space
cpustuff: Not enough space
I just keep getting 'Not enough space' forever after that, but the
daemon continues to function just fine.
If I run with with -d 2 or higher, I get the following type of output
(this is multiple instances, pulled somewhat randomly):
set_metric_value() exec'd cpu_user_func (10)
cpustuff: Permission denied
offset = -20931796, cpu_now[1] = 346648
cpustuff: Permission denied
offset = -523107352, cpu_now[1] = 586301
Raw: bread / bwrite / lread / lwrite / phread / phwrite
110481,110481,0 / 372254,372241,13 / 5049919,5049897,22 /
938684,938649,35 / 34,34,0 / 0,0,0
Aftermath: 0.000000 0.812500 1.375000 2.187500 0.000000 0.000000
XDR data successfully sent
mcast_value() mcasting os_release value
encoded 12 XDR bytes
XDR data successfully sent
set_metric_value() exec'd cpu_user_func (10)
* * * * Setting alpha to 0.016667 and beta to 0.983333 because timediff = 0
pre_process_node() received a new node
cpustuff: Permission denied
offset = -20931796, cpu_now[1] = 346731
cpustuff: Permission denied
offset = -523107352, cpu_now[1] = 586391
Raw: bread / bwrite / lread / lwrite / phread / phwrite
110481,0,110481 / 372291,0,372291 / 5049968,0,5049968 / 938746,0,938746
/ 34,0,34 / 0,0,0
Aftermath: 0.000100 0.000338 0.004586 0.000852 0.000000 0.000000 delta =
1101251722
** ** ** ** ** Are percentages electric? Try -23%, -3% , 11% , 4% , 0% 0%
mcast_value() mcasting cpu_user value
set_metric_value() exec'd cpu_user_func (10)
cpustuff: Permission denied
offset = -20931796, cpu_now[1] = 346752
cpustuff: Permission denied
offset = -523107352, cpu_now[1] = 586423
Raw: bread / bwrite / lread / lwrite / phread / phwrite
110481,110481,0 / 372317,372316,1 / 5050063,5050044,19 / 938796,938791,5
/ 34,34,0 / 0,0,0
Aftermath: 0.000000 0.062500 1.187500 0.312500 0.000000 0.000000 delta = 16
** ** ** ** ** Are percentages electric? Try -13%, 6% , 10% , 4% , 0% 0%
set_metric_value() exec'd bwrite_sec_func (31)
cpustuff: Not enough space
offset = -20931796, cpu_now[1] = 347152
cpustuff: Not enough space
offset = -523107352, cpu_now[1] = 587363
Raw: bread / bwrite / lread / lwrite / phread / phwrite
110500,110488,12 / 372398,372389,9 / 5050309,5050246,63 /
938981,938962,19 / 34,34,0 / 0,0,0
Aftermath: 0.750000 0.562500 3.937500 1.187500 0.000000 0.000000 delta = 16
** ** ** ** ** Are percentages electric? Try -13%, 6% , 10% , 4% , 0% 0%
set_metric_value() exec'd cpu_idle_func (13)
set_metric_value() exec'd lwrite_sec_func (33)
set_metric_value() exec'd cpu_user_func (10)
cpustuff: Not enough space
offset = -20931796, cpu_now[1] = 347389
cpustuff: Not enough space
offset = -523107352, cpu_now[1] = 588141
Raw: bread / bwrite / lread / lwrite / phread / phwrite
110509,110509,0 / 372715,372700,15 / 5051982,5051912,70 /
939971,939912,59 / 34,34,0 / 0,0,0
Aftermath: 0.000000 0.937500 4.375000 3.687500 0.000000 0.000000 delta = 16
** ** ** ** ** Are percentages electric? Try -13%, 6% , 10% , 4% , 0% 0%
Notice the 'Permission denied' and 'Not enough space' errors.
I can't seem to find either of those errors in the ganglia source tree,
so they are apparently system errors. If I run the process under truss
(by attaching a truss to the process after it's started successfully),
the host is marked down within about 2 minutes, and I get the following
in the output:
recvfrom(1, "\0\0\01D D 8 8B0", 1472, 0, 0xDF605958, 0xDF604DD4) = 8
lwp_sema_post(0xDF504E6C) = 0
lwp_sema_wait(0xDF504E6C) = 0
gettimeofday(0xDF605378) = 0
pause() (sleeping...)
signotifywait() (sleeping...)
lwp_sema_wait(0xDF504E6C) (sleeping...)
accept(2, 0xDF302C4C, 0xDF302C44, 1) (sleeping...)
pread64(3, 0xDF100A08, 712, 0xFEC09B2C) (sleeping...)
lwp_cond_wait(0xDF8E95E8, 0xDF8E95D0, 0xDEC07D78) (sleeping...)
recvfrom(1, 0xDF605968, 1472, 0, 0xDF605958, 0xDF604DD4) (sleeping...)
lwp_cond_wait(0xDF8EFE10, 0xDF8EFE20, 0xDF8E9640) (sleeping...)
recvfrom(1, "\0\0\0\f >86 002", 1472, 0, 0xDF605958, 0xDF604DD4) = 8
lwp_sema_post(0xDF504E6C) = 0
lwp_sema_wait(0xDF504E6C) = 0
gettimeofday(0xDF605378) = 0
pause() (sleeping...)
signotifywait() (sleeping...)
lwp_sema_wait(0xDF504E6C) (sleeping...)
accept(2, 0xDF302C4C, 0xDF302C44, 1) (sleeping...)
pread64(3, 0xDF100A08, 712, 0xFEC09B2C) (sleeping...)
lwp_cond_wait(0xDF8E95E8, 0xDF8E95D0, 0xDEC07D78) (sleeping...)
recvfrom(1, 0xDF605968, 1472, 0, 0xDF605958, 0xDF604DD4) (sleeping...)
lwp_cond_wait(0xDF8EFE10, 0xDF8EFE20, 0xDF8E9640) (sleeping...)
recvfrom(1, "\0\0\01A AA3B5 !", 1472, 0, 0xDF605958, 0xDF604DD4) = 8
lwp_sema_post(0xDF504E6C) = 0
lwp_sema_wait(0xDF504E6C) = 0
gettimeofday(0xDF604D10) = 0
pause() (sleeping...)
signotifywait() (sleeping...)
lwp_sema_wait(0xDF504E6C) (sleeping...)
accept(2, 0xDF302C4C, 0xDF302C44, 1) (sleeping...)
pread64(3, 0xDF100A08, 712, 0xFEC09B2C) (sleeping...)
lwp_cond_wait(0xDF8E95E8, 0xDF8E95D0, 0xDEC07D78) (sleeping...)
recvfrom(1, 0xDF605968, 1472, 0, 0xDF605958, 0xDF604DD4) (sleeping...)
lwp_cond_wait(0xDF8EFE10, 0xDF8EFE20, 0xDF8E9640) (sleeping...)
It doesn't seem particularly useful, but hey, there is a '!' character
in there... :)
This isn't exactly life-threatening, since I am only testing at this
point and it still works fine in minimal debugging mode, but this seems
pretty weird.
The process is running as the 'nobody' user, and everything was compiled
with '--prefix=/usr/local --with-metad' using gcc 3.3.2.
Anyone have any ideas?
