Re: [Ganglia-developers] gmond segfault with libpython
You can install the newest version of web interface without any interfere
with gmond/gmetad. Those are totally separated packages / sources now :)
On Mon, Feb 10, 2014 at 04:33:00PM -0500, Jeff Layton wrote:
I'm leaning this way :) I think things have gotten too screwed up
(to use a technical term) and there are problems.
The thing I'm concerned about is that the epel repo only has
version 3.1.7 (seems pretty darn old to me). I want something newer
and I want the new web interface.
[root@home4 ganglia]# yum list all | grep -i ganglia
ganglia.i686 3.1.7-6.el6epel
ganglia.x86_64 3.1.7-6.el6epel
ganglia-devel.i686 3.1.7-6.el6epel
ganglia-devel.x86_64 3.1.7-6.el6epel
ganglia-gmetad.x86_64 3.1.7-6.el6epel
ganglia-gmond.x86_64 3.1.7-6.el6epel
ganglia-gmond-python.x86_64 3.1.7-6.el6epel
ganglia-web.x86_64 3.1.7-6.el6epel
libnodeupdown-backend-ganglia.x86_64 1.14-1.el6 epel
I'm going to try Vladimir's rpm's first but they look really old to
me (May 7 2013) which is before Centos 6.5 was out.
I may be hitting the mailing list again this evening (I'm writing
an article about ganglia that is due in 2 days so I need to finish
quickly).
Thanks!
Jeff
At this point I suggest you - wipe out that Ganglia installation and
just use Epel repo - it has everything you need (gmond, gmetad,
ganglia-gmond-python). That blog you was basing on is terrible. It's
very bad to make install without creating packages - no one should do
this.
Moreover - this installation is not based on any good filesystem
hierarchy standard. Configuration files in /usr/local? Editing
ld.so.conf instead of creating file in ld.so.conf.d? Those are really
bad practices that lead guys to situations like yours.
Epel repo is very good, stable and secure. You can easily use it instead
of creating your own packages. And if you really have to - use rpmbuild
or https://github.com/jordansissel/fpm
And try installing Centos minimal at first - without any additional
packages. It really makes things simple :)
This segfault looks like some Python version problem; maybe you have
more than one Python installed or maybe you have some issues with Python
libraries. It's really hard to find sometimes - I would suggest you
cleaning this installation and starting over using packages.
On Mon, Feb 10, 2014 at 10:54:35AM -0500, Jeff Layton wrote:
The only thing in /usr/local/etc/conf.d/ is modpython.conf.
Given your guidance I think I've figured things out (I think).
It does appear that the python modules get loaded twice
(actually 3 times in my case). The time is in gmond.conf
where I have it in the modules section:
modules {
module {
name = core_metrics
}
module {
name = python_module
path = /usr/local/lib64/ganglia/modpython.so
params = /usr/local/lib64/ganglia/python_modules/
}
...
}
At the end of /etc/ganglia/gmond.conf I have two include
lines:
include (/usr/local/etc/conf.d/*.conf)
include('/etc/ganglia/conf.d/*.pyconf')
The first line includes the file /usr/local/etc/conf.d/modpython.conf.
This file has the following lines:
[root@home4 ganglia]# more /usr/local/etc/conf.d/modpython.conf
/*
params - path to the directory where mod_python
should look for python metric modules
the pyconf files in the include directory below
will be scanned for configurations for those modules
*/
modules {
module {
name = python_module
path = modpython.so
params = /usr/local/lib64/ganglia/python_modules
}
}
include (/etc/ganglia/conf.d/*.pyconf)
So it looks like the python modules get loaded 3 times (once for the
first include, a second time for the include line in the file
/usr/local/etc/conf.d/modpython.conf, and then a third time for the
second include line in gmond.conf.
Therefore, I erased the module lines in gmond.conf so that
I don't load them. I also erased the include line at the end
of gmond.conf pointing to /etc/ganglia/conf.d/*.pyconf. The
only include line in gmond.conf is the following:
include (/usr/local/etc/conf.d/*.conf)
You can find my current gmond.conf file here: http://pastebin.com/FJ2WAC4D
In the file /usr/local/etc/conf.d/modpython.conf, I commented out
the last line which is an include line pointing to
/etc/ganglia/conf.d/*.pyconf. The file now simply reads:
/*
params - path to the directory where mod_python
should look for python metric modules
the pyconf files in the include directory below
will be scanned for configurations for those modules
*/
modules {
module {
name = python_module
path = modpython.so
params = /usr/local/lib64/ganglia/python_modules
}
}
I think all of this means that
Re: [Ganglia-developers] gmond segfault with libpython
If you'd look for someone I could help with that - just send me a msg when you're sure ;) On Sun, Feb 09, 2014 at 11:07:22PM -0800, Bernard Li wrote: Do we still have a maintainer for the Ganglia packages for EPEL? If not, should we see if somebody would like to fill that position? Thanks, Bernard On Sun, Feb 9, 2014 at 6:15 PM, Vladimir Vuksan vli...@veus.hr wrote: Those RPMS work just fine for me [root@localhost ~]# uname -a Linux localhost.localdomain 2.6.32-431.el6.x86_64 #1 SMP Fri Nov 22 03:15:09 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux [root@localhost ~]# cat /etc/issue CentOS release 6.5 (Final) Kernel \r on an \m [root@localhost ~]# rpm -ivh http://vuksan.com/centos/RPMS-6/x86_64/ganglia-gmond-modules-python-3.6.0-1.x86_64.rpm http://vuksan.com/centos/RPMS-6/x86_64/libganglia-3.6.0-1.x86_64.rpm http://vuksan.com/centos/RPMS-6/x86_64/ganglia-gmond-3.6.0-1.x86_64.rpm http://vuksan.com/centos/RPMS-6/x86_64/libconfuse-2.6-2.el6.rf.x86_64.rpm Retrieving http://vuksan.com/centos/RPMS-6/x86_64/ganglia-gmond-modules-python-3.6.0-1.x86_64.rpm Retrieving http://vuksan.com/centos/RPMS-6/x86_64/libganglia-3.6.0-1.x86_64.rpm Retrieving http://vuksan.com/centos/RPMS-6/x86_64/ganglia-gmond-3.6.0-1.x86_64.rpm Retrieving http://vuksan.com/centos/RPMS-6/x86_64/libconfuse-2.6-2.el6.rf.x86_64.rpm warning: /var/tmp/rpm-tmp.JtASFF: Header V3 DSA/SHA1 Signature, key ID 6b8d79e6: NOKEY Preparing...### [100%] 1:libconfuse ### [ 25%] 2:libganglia ### [ 50%] 3:ganglia-gmond ### [ 75%] 4:ganglia-gmond-modules-p### [100%] [root@localhost ~]# gmond -d 2 loaded module: core_metrics loaded module: cpu_module loaded module: disk_module loaded module: load_module loaded module: mem_module loaded module: net_module loaded module: proc_module loaded module: sys_module loaded module: python_module udp_recv_channel mcast_join=239.2.11.71 mcast_if=NULL port=8649 bind=239.2.11.71 buffer=0 On 02/09/2014 09:39 AM, Jeff Layton wrote: Vladimir, I initially tried your binaries on my 6.5 system and I could not get them to install and run (I think they were built with a 6.3 system). At some point I'll try building the rpm's and installing those. Hopefully there is no different in the build process - that would be very interesting if the rpm's worked and building from source didn't :) I'll let you know - but first I'm going to try Maciej's strace idea. Thanks! Jeff P.S. There are some pretty significant differences between 6.4 and 6.5. One big one that I know of is the ntp format changed. I have not seen issues with Centos 6 however I usually build my RPM packages. You could do that if you type rpmbuild -tb ganglia-3.6.0.tar.gz Alternatively if you are interested to try prebuilt packages you can find them here. http://vuksan.com/centos/RPMS-6/x86_64/ Vladimir On 02/08/2014 11:11 AM, Jeff Layton wrote: Good morning, I'm running a CentOS 6.5 system with ganglia 3.6.0 and ganglia-web 3.5.12. I'm following the general guidelines in this article: http://sachinsharm.wordpress.com/tag/installing-ganglia/ Everything goes swimmingly and ganglia itself works fine. So I decide to go to the next step and try using Python with gmond. I followed the general guidelines in this article: http://sachinsharm.wordpress.com/2013/08/19/setup-and-configure-ganglia-python-modules-on-centosrhel-6-3/ But when I start up gmond I get a segfault as reported in /var/log/messages. Feb 5 19:58:47 home4 kernel: gmond[17992]: segfault at 8 ip 0036a7ce6ceb sp 7fffaad46bf0 error 4 in libpython2.6.so.1.0[36a7c0+15d000] Feb 5 19:58:47 home4 abrt[18003]: Saved core dump of pid 17992 (/usr/local/sbin/gmond) to /var/spool/abrt/ccpp-2014-02-05-19:58:47-17992 (4284416 bytes) Feb 5 19:58:47 home4 abrtd: Directory 'ccpp-2014-02-05-19:58:47-17992' creation detected Feb 5 19:58:47 home4 abrtd: Executable '/usr/local/sbin/gmond' doesn't belong to any package and ProcessUnpackaged is set to 'no' Feb 5 19:58:47 home4 abrtd: 'post-create' on '/var/spool/abrt/ccpp-2014-02-05-19:58:47-17992' exited with 1 Feb 5 19:58:47 home4 abrtd: Deleting problem directory '/var/spool/abrt/ccpp-2014-02-05-19:58:47-17992' I'm been trying to debug this but I have to admit that I'm coming up blank. Running gmond with debug doesn't give too much information: [root@home4 laytonjb]# gmond -d 5 -c /etc/ganglia/gmond.conf loaded module: core_metrics loaded module: cpu_module loaded module: disk_module loaded module: load_module loaded module: mem_module loaded module:
Re: [Ganglia-developers] gmond segfault with libpython
Ok so from that I can see that you're including:
include (/usr/local/etc/conf.d/*.conf)
include('/etc/ganglia/conf.d/*.pyconf')
Could you recheck what conf files you have in /usr/local/etc/conf.d/ ?
Next thing - why are you building those packages without setting any
proper (FHS like) directories
(http://pl.wikipedia.org/wiki/Filesystem_Hierarchy_Standard)?
I'm almost sure that there is some configuration issue there
On Sun, Feb 09, 2014 at 06:17:04PM -0500, Jeff Layton wrote:
Sure thing - I appreciate the help.
Build options:
./configure --with-gmetad
gmond.conf: http://pastebin.com/ExiMgqv0
strace output:
I ran the strace using the following command:
strace -s 1024 -ff -o strace.log gmond -d 5 -c /etc/ganglia/gmond.conf
The output of the thread that has the segfault in it was uploaded
to pastebin: http://pastebin.com/xScMVU6P
I had to erase the top 200 lines of the strace (too big and I'm not
a pro user - yet :) ).
But... just to be sure, I'm attaching the compressed tarball. Apologies
to all but I just wanted to be sure.
Once again - thanks a million!
Jeff
Could you post here your build options (that ones you entered while
./configure) and also could you paste gmond.conf into pastebin?
Also plz strace one more time, but now with strace -s 1024 -e trace=file
and paste the output to pastebin
On Sun, Feb 09, 2014 at 04:55:14PM -0500, Jeff Layton wrote:
I hope this isn't too much output (I've heard about pastebin.com
but never really used it).
[root@home4 ganglia-3.6.0]# ldd /usr/local/sbin/gmond
linux-vdso.so.1 = (0x7fff667f6000)
libapr-1.so.0 = /usr/lib64/libapr-1.so.0 (0x7f6a24049000)
libresolv.so.2 = /lib64/libresolv.so.2 (0x00337dc0)
libganglia-3.6.0.so.0 =
/usr/local/lib64/libganglia-3.6.0.so.0 (0x7f6a23e0d000)
libdl.so.2 = /lib64/libdl.so.2 (0x00337c40)
libnsl.so.1 = /lib64/libnsl.so.1 (0x003390c0)
libz.so.1 = /lib64/libz.so.1 (0x00337d00)
libpcre.so.0 = /lib64/libpcre.so.0 (0x003f7360)
libexpat.so.1 = /lib64/libexpat.so.1 (0x00337f80)
libconfuse.so.0 = /usr/lib64/libconfuse.so.0 (0x7f6a23bff000)
libpthread.so.0 = /lib64/libpthread.so.0 (0x00337c80)
libc.so.6 = /lib64/libc.so.6 (0x00337c00)
libuuid.so.1 = /lib64/libuuid.so.1 (0x00338380)
libcrypt.so.1 = /lib64/libcrypt.so.1 (0x00338c60)
/lib64/ld-linux-x86-64.so.2 (0x00337bc0)
libfreebl3.so = /usr/lib64/libfreebl3.so (0x00338ca0)
Below is the tree output:
[root@home4 ganglia-3.6.0]# tree /etc/ganglia
/etc/ganglia
??? conf.d
? ??? procstat.pyconf
??? gmetad.conf
??? gmond.conf
1 directory, 3 files
I looked at the strace file for process 3537 and I did see two places
where gmond does an access() on the python_modules directory.
Does gmond automatically look for the python modes so I don't need
to put them the modules section of gmond.conf?
Thanks a million!
Jeff
Oh I didn't think about going that lowlevel :) Could you run ldd on
gmond also?
Could you also run 'tree' command on /etc/ganglia ? It's interesting
that you have two times msg: loaded module: python_module while
starting gmond. Rechecking this with strace log shows that it looks like
double loading of those modules? http://pastebin.com/BjdCGgbj
On Sun, Feb 09, 2014 at 03:28:10PM -0500, Jeff Layton wrote:
On 02/09/2014 02:48 PM, Jeff Layton wrote:
On 02/09/2014 02:28 PM, Maciej Lasyk wrote:
You could also try to catch on which particular check this segfault
happens..?
Not sure how to check this. When I run gmond interactively, it
segfaults just after it says,
[root@home4 yum.repos.d]# /usr/local/sbin/gmond -d 5 -c
/etc/ganglia/gmond.conf
loaded module: core_metrics
loaded module: python_module
loaded module: cpu_module
loaded module: disk_module
loaded module: load_module
loaded module: mem_module
loaded module: net_module
loaded module: proc_module
loaded module: sys_module
loaded module: python_module
Segmentation fault (core dumped)
I'm not sure where to begin checking. I'm a very old-fashioned
debugger - I tend to use a great deal of print statements to
track down where things are happening. I can start doing this
in gmond.
I tried putting fprintf's all over the gmond.c (yep - I'm that
poor of a debugger). I'm not sure but if looks like it segfaults
in the function setup_metric_callbacks on the statement,
if (modp-init modp-init(global_context)) {
or on the function,
apr_pool_cleanup_register(global_context, modp,
modular_metric_cleanup,
apr_pool_cleanup_null);
I'm not too sure.
I apologize if I'm wasting your time with my poor debugging
skills.
Thanks!
Re: [Ganglia-developers] gmond segfault with libpython
The only thing in /usr/local/etc/conf.d/ is modpython.conf.
Given your guidance I think I've figured things out (I think).
It does appear that the python modules get loaded twice
(actually 3 times in my case). The time is in gmond.conf
where I have it in the modules section:
modules {
module {
name = core_metrics
}
module {
name = python_module
path = /usr/local/lib64/ganglia/modpython.so
params = /usr/local/lib64/ganglia/python_modules/
}
...
}
At the end of /etc/ganglia/gmond.conf I have two include
lines:
include (/usr/local/etc/conf.d/*.conf)
include('/etc/ganglia/conf.d/*.pyconf')
The first line includes the file /usr/local/etc/conf.d/modpython.conf.
This file has the following lines:
[root@home4 ganglia]# more /usr/local/etc/conf.d/modpython.conf
/*
params - path to the directory where mod_python
should look for python metric modules
the pyconf files in the include directory below
will be scanned for configurations for those modules
*/
modules {
module {
name = python_module
path = modpython.so
params = /usr/local/lib64/ganglia/python_modules
}
}
include (/etc/ganglia/conf.d/*.pyconf)
So it looks like the python modules get loaded 3 times (once for the
first include, a second time for the include line in the file
/usr/local/etc/conf.d/modpython.conf, and then a third time for the
second include line in gmond.conf.
Therefore, I erased the module lines in gmond.conf so that
I don't load them. I also erased the include line at the end
of gmond.conf pointing to /etc/ganglia/conf.d/*.pyconf. The
only include line in gmond.conf is the following:
include (/usr/local/etc/conf.d/*.conf)
You can find my current gmond.conf file here: http://pastebin.com/FJ2WAC4D
In the file /usr/local/etc/conf.d/modpython.conf, I commented out
the last line which is an include line pointing to
/etc/ganglia/conf.d/*.pyconf. The file now simply reads:
/*
params - path to the directory where mod_python
should look for python metric modules
the pyconf files in the include directory below
will be scanned for configurations for those modules
*/
modules {
module {
name = python_module
path = modpython.so
params = /usr/local/lib64/ganglia/python_modules
}
}
I think all of this means that python modules only get loaded
once when it gmond.conf does the include that points to
/usr/local/etc/conf.d/*.conf
Note - this file looks like:
/*
params - path to the directory where mod_python
should look for python metric modules
the pyconf files in the include directory below
will be scanned for configurations for those modules
*/
modules {
module {
name = python_module
path = /usr/local/lib64/ganglia/modpython.so
params = /usr/local/lib64/ganglia/python_modules
}
}
I think this should fix the problem so I tried running gmond
interactively:
/usr/local/sbin/gmond -d 5 -c /etc/ganglia/gmond.conf
I still get a segfault.
As an aside, this is just an experiment so I can learn about writing
python modules in Ganglia. Therefore I'm not too concerned
about the location of configuration files since it's temporary.
But, I followed all of the defaults in ganglia about installing
the code to /usr/local. I did create the directory /etc/ganglia
since I wanted all ganglia related files to be in one location
rather spread across all of /etc *it may not be FHS compliant
but it's a practice I have developed over the years.
In general I followed this blog:
http://sachinsharm.wordpress.com/tag/installing-ganglia/
for building and installing ganglia. Everything worked just
fine until I followed this blog
http://sachinsharm.wordpress.com/2013/08/19/setup-and-configure-ganglia-python-modules-on-centosrhel-6-3/
for configuring Python modules. But I backed out all of the
changes in that blog so that I was starting in a clean configuration.
Thanks for the help! You have been very patient and I really
appreciate it.
Jeff
Maciej Lasyk wrote:
Ok so from that I can see that you're including:
include (/usr/local/etc/conf.d/*.conf)
include('/etc/ganglia/conf.d/*.pyconf')
Could you recheck what conf files you have in /usr/local/etc/conf.d/ ?
Next thing - why are you building those packages without setting any
proper (FHS like) directories
(http://pl.wikipedia.org/wiki/Filesystem_Hierarchy_Standard)?
I'm almost sure that there is some configuration issue there
On Sun, Feb 09, 2014 at 06:17:04PM -0500, Jeff Layton wrote:
Sure thing - I appreciate the help.
Build options:
./configure --with-gmetad
gmond.conf: http://pastebin.com/ExiMgqv0
strace output:
I ran the strace using the following command:
strace -s 1024 -ff -o strace.log gmond -d 5 -c /etc/ganglia/gmond.conf
The output of the thread that has the segfault in it was uploaded
to pastebin: http://pastebin.com/xScMVU6P
I had to erase the top 200 lines of the strace (too big and I'm
Re: [Ganglia-developers] gmond segfault with libpython
On 02/09/2014 09:15 PM, Vladimir Vuksan wrote: Those RPMS work just fine for me [root@localhost ~]# uname -a Linux localhost.localdomain 2.6.32-431.el6.x86_64 #1 SMP Fri Nov 22 03:15:09 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux [root@localhost ~]# cat /etc/issue CentOS release 6.5 (Final) Kernel \r on an \m [root@localhost ~]# rpm -ivh http://vuksan.com/centos/RPMS-6/x86_64/ganglia-gmond-modules-python-3.6.0-1.x86_64.rpm http://vuksan.com/centos/RPMS-6/x86_64/libganglia-3.6.0-1.x86_64.rpm http://vuksan.com/centos/RPMS-6/x86_64/ganglia-gmond-3.6.0-1.x86_64.rpm http://vuksan.com/centos/RPMS-6/x86_64/libconfuse-2.6-2.el6.rf.x86_64.rpm Retrieving http://vuksan.com/centos/RPMS-6/x86_64/ganglia-gmond-modules-python-3.6.0-1.x86_64.rpm Retrieving http://vuksan.com/centos/RPMS-6/x86_64/libganglia-3.6.0-1.x86_64.rpm Retrieving http://vuksan.com/centos/RPMS-6/x86_64/ganglia-gmond-3.6.0-1.x86_64.rpm Retrieving http://vuksan.com/centos/RPMS-6/x86_64/libconfuse-2.6-2.el6.rf.x86_64.rpm warning: /var/tmp/rpm-tmp.JtASFF: Header V3 DSA/SHA1 Signature, key ID 6b8d79e6: NOKEY Preparing... ### [100%] 1:libconfuse ### [ 25%] 2:libganglia ### [ 50%] 3:ganglia-gmond ### [ 75%] 4:ganglia-gmond-modules-p### [100%] [root@localhost ~]# gmond -d 2 loaded module: core_metrics loaded module: cpu_module loaded module: disk_module loaded module: load_module loaded module: mem_module loaded module: net_module loaded module: proc_module loaded module: sys_module loaded module: python_module udp_recv_channel mcast_join=239.2.11.71 mcast_if=NULL port=8649 bind=239.2.11.71 buffer=0 Interesting. When I started my project about a month ago, I tried your rpms' (they came up first on a google search) but I couldn't get them to install it appears they were built for CentOS 6.3 perhaps? The link that came up is: http://vuksan.com/centos/RPMS/x86_64/ that doesn't match your URL (I didn't know about RPMS-6. The 3.6.0 rpm's are dated 2013 (07-May-2013) which is why I had problems. Jeff -- Managing the Performance of Cloud-Based Applications Take advantage of what the Cloud has to offer - Avoid Common Pitfalls. Read the Whitepaper. http://pubads.g.doubleclick.net/gampad/clk?id=121051231iu=/4140/ostg.clktrk___ Ganglia-developers mailing list Ganglia-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ganglia-developers
Re: [Ganglia-developers] gmond segfault with libpython
At this point I suggest you - wipe out that Ganglia installation and
just use Epel repo - it has everything you need (gmond, gmetad,
ganglia-gmond-python). That blog you was basing on is terrible. It's
very bad to make install without creating packages - no one should do
this.
Moreover - this installation is not based on any good filesystem
hierarchy standard. Configuration files in /usr/local? Editing
ld.so.conf instead of creating file in ld.so.conf.d? Those are really
bad practices that lead guys to situations like yours.
Epel repo is very good, stable and secure. You can easily use it instead
of creating your own packages. And if you really have to - use rpmbuild
or https://github.com/jordansissel/fpm
And try installing Centos minimal at first - without any additional
packages. It really makes things simple :)
This segfault looks like some Python version problem; maybe you have
more than one Python installed or maybe you have some issues with Python
libraries. It's really hard to find sometimes - I would suggest you
cleaning this installation and starting over using packages.
On Mon, Feb 10, 2014 at 10:54:35AM -0500, Jeff Layton wrote:
The only thing in /usr/local/etc/conf.d/ is modpython.conf.
Given your guidance I think I've figured things out (I think).
It does appear that the python modules get loaded twice
(actually 3 times in my case). The time is in gmond.conf
where I have it in the modules section:
modules {
module {
name = core_metrics
}
module {
name = python_module
path = /usr/local/lib64/ganglia/modpython.so
params = /usr/local/lib64/ganglia/python_modules/
}
...
}
At the end of /etc/ganglia/gmond.conf I have two include
lines:
include (/usr/local/etc/conf.d/*.conf)
include('/etc/ganglia/conf.d/*.pyconf')
The first line includes the file /usr/local/etc/conf.d/modpython.conf.
This file has the following lines:
[root@home4 ganglia]# more /usr/local/etc/conf.d/modpython.conf
/*
params - path to the directory where mod_python
should look for python metric modules
the pyconf files in the include directory below
will be scanned for configurations for those modules
*/
modules {
module {
name = python_module
path = modpython.so
params = /usr/local/lib64/ganglia/python_modules
}
}
include (/etc/ganglia/conf.d/*.pyconf)
So it looks like the python modules get loaded 3 times (once for the
first include, a second time for the include line in the file
/usr/local/etc/conf.d/modpython.conf, and then a third time for the
second include line in gmond.conf.
Therefore, I erased the module lines in gmond.conf so that
I don't load them. I also erased the include line at the end
of gmond.conf pointing to /etc/ganglia/conf.d/*.pyconf. The
only include line in gmond.conf is the following:
include (/usr/local/etc/conf.d/*.conf)
You can find my current gmond.conf file here: http://pastebin.com/FJ2WAC4D
In the file /usr/local/etc/conf.d/modpython.conf, I commented out
the last line which is an include line pointing to
/etc/ganglia/conf.d/*.pyconf. The file now simply reads:
/*
params - path to the directory where mod_python
should look for python metric modules
the pyconf files in the include directory below
will be scanned for configurations for those modules
*/
modules {
module {
name = python_module
path = modpython.so
params = /usr/local/lib64/ganglia/python_modules
}
}
I think all of this means that python modules only get loaded
once when it gmond.conf does the include that points to
/usr/local/etc/conf.d/*.conf
Note - this file looks like:
/*
params - path to the directory where mod_python
should look for python metric modules
the pyconf files in the include directory below
will be scanned for configurations for those modules
*/
modules {
module {
name = python_module
path = /usr/local/lib64/ganglia/modpython.so
params = /usr/local/lib64/ganglia/python_modules
}
}
I think this should fix the problem so I tried running gmond
interactively:
/usr/local/sbin/gmond -d 5 -c /etc/ganglia/gmond.conf
I still get a segfault.
As an aside, this is just an experiment so I can learn about writing
python modules in Ganglia. Therefore I'm not too concerned
about the location of configuration files since it's temporary.
But, I followed all of the defaults in ganglia about installing
the code to /usr/local. I did create the directory /etc/ganglia
since I wanted all ganglia related files to be in one location
rather spread across all of /etc *it may not be FHS compliant
but it's a practice I have developed over the years.
In general I followed this blog:
http://sachinsharm.wordpress.com/tag/installing-ganglia/
for building and installing ganglia. Everything worked just
fine until I followed this blog
Re: [Ganglia-developers] gmond segfault with libpython
I'm leaning this way :) I think things have gotten too screwed up
(to use a technical term) and there are problems.
The thing I'm concerned about is that the epel repo only has
version 3.1.7 (seems pretty darn old to me). I want something newer
and I want the new web interface.
[root@home4 ganglia]# yum list all | grep -i ganglia
ganglia.i686 3.1.7-6.el6epel
ganglia.x86_64 3.1.7-6.el6epel
ganglia-devel.i686 3.1.7-6.el6epel
ganglia-devel.x86_64 3.1.7-6.el6epel
ganglia-gmetad.x86_64 3.1.7-6.el6epel
ganglia-gmond.x86_64 3.1.7-6.el6epel
ganglia-gmond-python.x86_64 3.1.7-6.el6epel
ganglia-web.x86_64 3.1.7-6.el6epel
libnodeupdown-backend-ganglia.x86_64 1.14-1.el6 epel
I'm going to try Vladimir's rpm's first but they look really old to
me (May 7 2013) which is before Centos 6.5 was out.
I may be hitting the mailing list again this evening (I'm writing
an article about ganglia that is due in 2 days so I need to finish
quickly).
Thanks!
Jeff
At this point I suggest you - wipe out that Ganglia installation and
just use Epel repo - it has everything you need (gmond, gmetad,
ganglia-gmond-python). That blog you was basing on is terrible. It's
very bad to make install without creating packages - no one should do
this.
Moreover - this installation is not based on any good filesystem
hierarchy standard. Configuration files in /usr/local? Editing
ld.so.conf instead of creating file in ld.so.conf.d? Those are really
bad practices that lead guys to situations like yours.
Epel repo is very good, stable and secure. You can easily use it instead
of creating your own packages. And if you really have to - use rpmbuild
or https://github.com/jordansissel/fpm
And try installing Centos minimal at first - without any additional
packages. It really makes things simple :)
This segfault looks like some Python version problem; maybe you have
more than one Python installed or maybe you have some issues with Python
libraries. It's really hard to find sometimes - I would suggest you
cleaning this installation and starting over using packages.
On Mon, Feb 10, 2014 at 10:54:35AM -0500, Jeff Layton wrote:
The only thing in /usr/local/etc/conf.d/ is modpython.conf.
Given your guidance I think I've figured things out (I think).
It does appear that the python modules get loaded twice
(actually 3 times in my case). The time is in gmond.conf
where I have it in the modules section:
modules {
module {
name = core_metrics
}
module {
name = python_module
path = /usr/local/lib64/ganglia/modpython.so
params = /usr/local/lib64/ganglia/python_modules/
}
...
}
At the end of /etc/ganglia/gmond.conf I have two include
lines:
include (/usr/local/etc/conf.d/*.conf)
include('/etc/ganglia/conf.d/*.pyconf')
The first line includes the file /usr/local/etc/conf.d/modpython.conf.
This file has the following lines:
[root@home4 ganglia]# more /usr/local/etc/conf.d/modpython.conf
/*
params - path to the directory where mod_python
should look for python metric modules
the pyconf files in the include directory below
will be scanned for configurations for those modules
*/
modules {
module {
name = python_module
path = modpython.so
params = /usr/local/lib64/ganglia/python_modules
}
}
include (/etc/ganglia/conf.d/*.pyconf)
So it looks like the python modules get loaded 3 times (once for the
first include, a second time for the include line in the file
/usr/local/etc/conf.d/modpython.conf, and then a third time for the
second include line in gmond.conf.
Therefore, I erased the module lines in gmond.conf so that
I don't load them. I also erased the include line at the end
of gmond.conf pointing to /etc/ganglia/conf.d/*.pyconf. The
only include line in gmond.conf is the following:
include (/usr/local/etc/conf.d/*.conf)
You can find my current gmond.conf file here: http://pastebin.com/FJ2WAC4D
In the file /usr/local/etc/conf.d/modpython.conf, I commented out
the last line which is an include line pointing to
/etc/ganglia/conf.d/*.pyconf. The file now simply reads:
/*
params - path to the directory where mod_python
should look for python metric modules
the pyconf files in the include directory below
will be scanned for configurations for those modules
*/
modules {
module {
name = python_module
path = modpython.so
params = /usr/local/lib64/ganglia/python_modules
}
}
I think all of this means that python modules only get loaded
once when it gmond.conf does the include that points to
/usr/local/etc/conf.d/*.conf
Note - this file looks like:
/*
params - path to the directory where mod_python
should look for python metric modules
the pyconf
Re: [Ganglia-developers] gmond segfault with libpython
Jeff,
RPMS-6 are the Centos 6 RPMS. RPMS/ are Centos 5 RPMS. Sorry about the
confusion.
On 02/10/2014 04:33 PM, Jeff Layton wrote:
I'm leaning this way :) I think things have gotten too screwed up
(to use a technical term) and there are problems.
The thing I'm concerned about is that the epel repo only has
version 3.1.7 (seems pretty darn old to me). I want something newer
and I want the new web interface.
[root@home4 ganglia]# yum list all | grep -i ganglia
ganglia.i686 3.1.7-6.el6epel
ganglia.x86_64 3.1.7-6.el6epel
ganglia-devel.i686 3.1.7-6.el6epel
ganglia-devel.x86_64 3.1.7-6.el6epel
ganglia-gmetad.x86_64 3.1.7-6.el6epel
ganglia-gmond.x86_64 3.1.7-6.el6epel
ganglia-gmond-python.x86_64 3.1.7-6.el6epel
ganglia-web.x86_64 3.1.7-6.el6epel
libnodeupdown-backend-ganglia.x86_64 1.14-1.el6 epel
I'm going to try Vladimir's rpm's first but they look really old to
me (May 7 2013) which is before Centos 6.5 was out.
I may be hitting the mailing list again this evening (I'm writing
an article about ganglia that is due in 2 days so I need to finish
quickly).
Thanks!
Jeff
At this point I suggest you - wipe out that Ganglia installation and
just use Epel repo - it has everything you need (gmond, gmetad,
ganglia-gmond-python). That blog you was basing on is terrible. It's
very bad to make install without creating packages - no one should do
this.
Moreover - this installation is not based on any good filesystem
hierarchy standard. Configuration files in /usr/local? Editing
ld.so.conf instead of creating file in ld.so.conf.d? Those are really
bad practices that lead guys to situations like yours.
Epel repo is very good, stable and secure. You can easily use it instead
of creating your own packages. And if you really have to - use rpmbuild
or https://github.com/jordansissel/fpm
And try installing Centos minimal at first - without any additional
packages. It really makes things simple :)
This segfault looks like some Python version problem; maybe you have
more than one Python installed or maybe you have some issues with Python
libraries. It's really hard to find sometimes - I would suggest you
cleaning this installation and starting over using packages.
On Mon, Feb 10, 2014 at 10:54:35AM -0500, Jeff Layton wrote:
The only thing in /usr/local/etc/conf.d/ is modpython.conf.
Given your guidance I think I've figured things out (I think).
It does appear that the python modules get loaded twice
(actually 3 times in my case). The time is in gmond.conf
where I have it in the modules section:
modules {
module {
name = core_metrics
}
module {
name = python_module
path = /usr/local/lib64/ganglia/modpython.so
params = /usr/local/lib64/ganglia/python_modules/
}
...
}
At the end of /etc/ganglia/gmond.conf I have two include
lines:
include (/usr/local/etc/conf.d/*.conf)
include('/etc/ganglia/conf.d/*.pyconf')
The first line includes the file /usr/local/etc/conf.d/modpython.conf.
This file has the following lines:
[root@home4 ganglia]# more /usr/local/etc/conf.d/modpython.conf
/*
params - path to the directory where mod_python
should look for python metric modules
the pyconf files in the include directory below
will be scanned for configurations for those modules
*/
modules {
module {
name = python_module
path = modpython.so
params = /usr/local/lib64/ganglia/python_modules
}
}
include (/etc/ganglia/conf.d/*.pyconf)
So it looks like the python modules get loaded 3 times (once for the
first include, a second time for the include line in the file
/usr/local/etc/conf.d/modpython.conf, and then a third time for the
second include line in gmond.conf.
Therefore, I erased the module lines in gmond.conf so that
I don't load them. I also erased the include line at the end
of gmond.conf pointing to /etc/ganglia/conf.d/*.pyconf. The
only include line in gmond.conf is the following:
include (/usr/local/etc/conf.d/*.conf)
You can find my current gmond.conf file here: http://pastebin.com/FJ2WAC4D
In the file /usr/local/etc/conf.d/modpython.conf, I commented out
the last line which is an include line pointing to
/etc/ganglia/conf.d/*.pyconf. The file now simply reads:
/*
params - path to the directory where mod_python
should look for python metric modules
the pyconf files in the include directory below
will be scanned for configurations for those modules
*/
modules {
module {
name = python_module
path = modpython.so
params = /usr/local/lib64/ganglia/python_modules
}
}
I think all of this means that python modules only get loaded
once when it gmond.conf does the include that points to
Re: [Ganglia-developers] gmond segfault with libpython
Hi Maciej: Please come find us on IRC and let's talk. Thanks, Bernard On Mon, Feb 10, 2014 at 1:24 AM, Maciej Lasyk mac...@lasyk.info wrote: If you'd look for someone I could help with that - just send me a msg when you're sure ;) On Sun, Feb 09, 2014 at 11:07:22PM -0800, Bernard Li wrote: Do we still have a maintainer for the Ganglia packages for EPEL? If not, should we see if somebody would like to fill that position? Thanks, Bernard On Sun, Feb 9, 2014 at 6:15 PM, Vladimir Vuksan vli...@veus.hr wrote: Those RPMS work just fine for me [root@localhost ~]# uname -a Linux localhost.localdomain 2.6.32-431.el6.x86_64 #1 SMP Fri Nov 22 03:15:09 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux [root@localhost ~]# cat /etc/issue CentOS release 6.5 (Final) Kernel \r on an \m [root@localhost ~]# rpm -ivh http://vuksan.com/centos/RPMS-6/x86_64/ganglia-gmond-modules-python-3.6.0-1.x86_64.rpm http://vuksan.com/centos/RPMS-6/x86_64/libganglia-3.6.0-1.x86_64.rpm http://vuksan.com/centos/RPMS-6/x86_64/ganglia-gmond-3.6.0-1.x86_64.rpm http://vuksan.com/centos/RPMS-6/x86_64/libconfuse-2.6-2.el6.rf.x86_64.rpm Retrieving http://vuksan.com/centos/RPMS-6/x86_64/ganglia-gmond-modules-python-3.6.0-1.x86_64.rpm Retrieving http://vuksan.com/centos/RPMS-6/x86_64/libganglia-3.6.0-1.x86_64.rpm Retrieving http://vuksan.com/centos/RPMS-6/x86_64/ganglia-gmond-3.6.0-1.x86_64.rpm Retrieving http://vuksan.com/centos/RPMS-6/x86_64/libconfuse-2.6-2.el6.rf.x86_64.rpm warning: /var/tmp/rpm-tmp.JtASFF: Header V3 DSA/SHA1 Signature, key ID 6b8d79e6: NOKEY Preparing...### [100%] 1:libconfuse ### [ 25%] 2:libganglia ### [ 50%] 3:ganglia-gmond ### [ 75%] 4:ganglia-gmond-modules-p### [100%] [root@localhost ~]# gmond -d 2 loaded module: core_metrics loaded module: cpu_module loaded module: disk_module loaded module: load_module loaded module: mem_module loaded module: net_module loaded module: proc_module loaded module: sys_module loaded module: python_module udp_recv_channel mcast_join=239.2.11.71 mcast_if=NULL port=8649 bind=239.2.11.71 buffer=0 On 02/09/2014 09:39 AM, Jeff Layton wrote: Vladimir, I initially tried your binaries on my 6.5 system and I could not get them to install and run (I think they were built with a 6.3 system). At some point I'll try building the rpm's and installing those. Hopefully there is no different in the build process - that would be very interesting if the rpm's worked and building from source didn't :) I'll let you know - but first I'm going to try Maciej's strace idea. Thanks! Jeff P.S. There are some pretty significant differences between 6.4 and 6.5. One big one that I know of is the ntp format changed. I have not seen issues with Centos 6 however I usually build my RPM packages. You could do that if you type rpmbuild -tb ganglia-3.6.0.tar.gz Alternatively if you are interested to try prebuilt packages you can find them here. http://vuksan.com/centos/RPMS-6/x86_64/ Vladimir On 02/08/2014 11:11 AM, Jeff Layton wrote: Good morning, I'm running a CentOS 6.5 system with ganglia 3.6.0 and ganglia-web 3.5.12. I'm following the general guidelines in this article: http://sachinsharm.wordpress.com/tag/installing-ganglia/ Everything goes swimmingly and ganglia itself works fine. So I decide to go to the next step and try using Python with gmond. I followed the general guidelines in this article: http://sachinsharm.wordpress.com/2013/08/19/setup-and-configure-ganglia-python-modules-on-centosrhel-6-3/ But when I start up gmond I get a segfault as reported in /var/log/messages. Feb 5 19:58:47 home4 kernel: gmond[17992]: segfault at 8 ip 0036a7ce6ceb sp 7fffaad46bf0 error 4 in libpython2.6.so.1.0[36a7c0+15d000] Feb 5 19:58:47 home4 abrt[18003]: Saved core dump of pid 17992 (/usr/local/sbin/gmond) to /var/spool/abrt/ccpp-2014-02-05-19:58:47-17992 (4284416 bytes) Feb 5 19:58:47 home4 abrtd: Directory 'ccpp-2014-02-05-19:58:47-17992' creation detected Feb 5 19:58:47 home4 abrtd: Executable '/usr/local/sbin/gmond' doesn't belong to any package and ProcessUnpackaged is set to 'no' Feb 5 19:58:47 home4 abrtd: 'post-create' on '/var/spool/abrt/ccpp-2014-02-05-19:58:47-17992' exited with 1 Feb 5 19:58:47 home4 abrtd: Deleting problem directory '/var/spool/abrt/ccpp-2014-02-05-19:58:47-17992' I'm been trying to debug this but I have to admit that I'm coming up blank. Running gmond with debug doesn't give too much information: [root@home4 laytonjb]# gmond -d 5 -c /etc/ganglia/gmond.conf loaded
Re: [Ganglia-developers] gmond segfault with libpython
Ok, that's weird. Could you recheck SELinux config and/or logs in /var/log/messages and audit.log? On Sun, Feb 09, 2014 at 09:54:09AM -0500, Jeff Layton wrote: Maciej, I'm attaching a tar-gzip file with the strace logs (I hope it gets through the various filters). There are 11 separate strace files (I didn't combine them into one). The only one that has a segfault in it is 3537: access(/usr/local/lib64/ganglia/python_modules, F_OK) = 0 access(/usr/local/lib64/ganglia/python_modules, R_OK) = 0 --- SIGSEGV (Segmentation fault) @ 0 (0) --- I ran gmond as root. It appears to be something just after the permissions on python module directory are checked (my naive attempt at understand the strace output). Thanks! Jeff strace -s 1024 -ff -o strace.log gmond-c /etc/ganglia/gmond.conf -- -- pozdrawiam, Maciej Lasyk GPG key ID: FFA8AEEC GPG info: http://maciek.lasyk.info/gpg.txt pgpjwSku3ALi3.pgp Description: PGP signature -- Managing the Performance of Cloud-Based Applications Take advantage of what the Cloud has to offer - Avoid Common Pitfalls. Read the Whitepaper. http://pubads.g.doubleclick.net/gampad/clk?id=121051231iu=/4140/ostg.clktrk___ Ganglia-developers mailing list Ganglia-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ganglia-developers
Re: [Ganglia-developers] gmond segfault with libpython
Maciej, Yep, SELinux is off: [root@home4 laytonjb]# cat /etc/sysconfig/selinux # This file controls the state of SELinux on the system. # SELINUX= can take one of these three values: # enforcing - SELinux security policy is enforced. # permissive - SELinux prints warnings instead of enforcing. # disabled - No SELinux policy is loaded. SELINUX=disabled # SELINUXTYPE= can take one of these two values: # targeted - Targeted processes are protected, # mls - Multi Level Security protection. SELINUXTYPE=targeted I don't see anything in any logs. Here is the end of /var/log/messages: Feb 9 13:14:07 home4 kernel: gmond[8866]: segfault at 8 ip 0036a7ce6ceb sp 7fffabb234d0 error 4 in libpython2.6.so.1.0[36a7c0+15d000] Feb 9 13:14:07 home4 abrt[8877]: Saved core dump of pid 8866 (/usr/local/sbin/gmond) to /var/spool/abrt/ccpp-2014-02-09-13:14:07-8866 (5361664 bytes) Feb 9 13:14:07 home4 abrtd: Directory 'ccpp-2014-02-09-13:14:07-8866' creation detected Feb 9 13:14:07 home4 abrtd: Executable '/usr/local/sbin/gmond' doesn't belong to any package and ProcessUnpackaged is set to 'no' Feb 9 13:14:07 home4 abrtd: 'post-create' on '/var/spool/abrt/ccpp-2014-02-09-13:14:07-8866' exited with 1 Feb 9 13:14:07 home4 abrtd: Deleting problem directory '/var/spool/abrt/ccpp-2014-02-09-13:14:07-8866' I did check audit.log and there was only one line that pertained to anything type=ANOM_ABEND msg=audit(1391969647.201:480): auid=500 uid=0 gid=0 ses=1 pid=8866 comm=gmond sig=11 I built the code with almost everything enabled. I thought about removing some packages from my system and rebuilding. For example, I could try removing libcmemcached but I'm not sure that will do any good. Maybe I need to drop back to an older ganglia version? Thanks for your help! Jeff Ok, that's weird. Could you recheck SELinux config and/or logs in /var/log/messages and audit.log? On Sun, Feb 09, 2014 at 09:54:09AM -0500, Jeff Layton wrote: Maciej, I'm attaching a tar-gzip file with the strace logs (I hope it gets through the various filters). There are 11 separate strace files (I didn't combine them into one). The only one that has a segfault in it is 3537: access(/usr/local/lib64/ganglia/python_modules, F_OK) = 0 access(/usr/local/lib64/ganglia/python_modules, R_OK) = 0 --- SIGSEGV (Segmentation fault) @ 0 (0) --- I ran gmond as root. It appears to be something just after the permissions on python module directory are checked (my naive attempt at understand the strace output). Thanks! Jeff strace -s 1024 -ff -o strace.log gmond-c /etc/ganglia/gmond.conf -- Managing the Performance of Cloud-Based Applications Take advantage of what the Cloud has to offer - Avoid Common Pitfalls. Read the Whitepaper. http://pubads.g.doubleclick.net/gampad/clk?id=121051231iu=/4140/ostg.clktrk ___ Ganglia-developers mailing list Ganglia-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ganglia-developers -- Managing the Performance of Cloud-Based Applications Take advantage of what the Cloud has to offer - Avoid Common Pitfalls. Read the Whitepaper. http://pubads.g.doubleclick.net/gampad/clk?id=121051231iu=/4140/ostg.clktrk___ Ganglia-developers mailing list Ganglia-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ganglia-developers
Re: [Ganglia-developers] gmond segfault with libpython
On 02/09/2014 02:28 PM, Maciej Lasyk wrote: On Sun, Feb 09, 2014 at 01:22:08PM -0500, Jeff Layton wrote: Maybe I need to drop back to an older ganglia version? What kind of repositories do you use on this Centos (yum repolist all)? Does 'yum update' finishes without any problems? I apologies for the length. repo idrepo name status C6.0-base CentOS-6.0 - Base disabled C6.0-centosplusCentOS-6.0 - CentOSPlus disabled C6.0-contrib CentOS-6.0 - Contribdisabled C6.0-extrasCentOS-6.0 - Extras disabled C6.0-updates CentOS-6.0 - Updatesdisabled C6.1-base CentOS-6.1 - Base disabled C6.1-centosplusCentOS-6.1 - CentOSPlus disabled C6.1-contrib CentOS-6.1 - Contribdisabled C6.1-extrasCentOS-6.1 - Extras disabled C6.1-updates CentOS-6.1 - Updatesdisabled C6.2-base CentOS-6.2 - Base disabled C6.2-centosplusCentOS-6.2 - CentOSPlus disabled C6.2-contrib CentOS-6.2 - Contribdisabled C6.2-extrasCentOS-6.2 - Extras disabled C6.2-updates CentOS-6.2 - Updatesdisabled C6.3-base CentOS-6.3 - Base disabled C6.3-centosplusCentOS-6.3 - CentOSPlus disabled C6.3-contrib CentOS-6.3 - Contribdisabled C6.3-extrasCentOS-6.3 - Extras disabled C6.3-updates CentOS-6.3 - Updatesdisabled C6.4-base CentOS-6.4 - Base disabled C6.4-centosplusCentOS-6.4 - CentOSPlus disabled C6.4-contrib CentOS-6.4 - Contribdisabled C6.4-extrasCentOS-6.4 - Extras disabled C6.4-updates CentOS-6.4 - Updatesdisabled adobe-linux-i386 Adobe Systems Incorporated enabled: 2 adobe-linux-x86_64 Adobe Systems Incorporated enabled: 2 base CentOS-6 - Base enabled: 6,367 c6-media CentOS-6 - Mediadisabled centosplus CentOS-6 - Plus disabled contribCentOS-6 - Contrib disabled debug CentOS-6 - Debuginfodisabled elrepo ELRepo.org Community Enterprise Linux Repository - el6 enabled:262 elrepo-extras ELRepo.org Community Enterprise Linux Extras Repository - e disabled elrepo-kernel ELRepo.org Community Enterprise Linux Kernel Repository - e disabled elrepo-testing ELRepo.org Community Enterprise Linux Testing Repository - disabled epel Extra Packages for Enterprise Linux 6 - x86_64 enabled: 10,444 epel-debuginfo Extra Packages for Enterprise Linux 6 - x86_64 - Debug disabled epel-sourceExtra Packages for Enterprise Linux 6 - x86_64 - Source disabled epel-testing Extra Packages for Enterprise Linux 6 - Testing - x86_64disabled epel-testing-debuginfo Extra Packages for Enterprise Linux 6 - Testing - x86_64 - disabled epel-testing-sourceExtra Packages for Enterprise Linux 6 - Testing - x86_64 - disabled extras CentOS-6 - Extras enabled: 14 updatesCentOS-6 - Updates enabled:464 virtualbox Oracle Linux / RHEL / CentOS-6 / x86_64 - VirtualBoxenabled: 22 repolist: 17,577 Everything updates correctly (i.e. it finishes). I built ganglia from source - I didn't use any rpm's. You could also try to catch on which particular check this segfault happens..? Not sure how to check this. When I run gmond interactively, it segfaults just after it says, [root@home4 yum.repos.d]# /usr/local/sbin/gmond -d
Re: [Ganglia-developers] gmond segfault with libpython
On 02/09/2014 02:48 PM, Jeff Layton wrote:
On 02/09/2014 02:28 PM, Maciej Lasyk wrote:
You could also try to catch on which particular check this segfault
happens..?
Not sure how to check this. When I run gmond interactively, it
segfaults just after it says,
[root@home4 yum.repos.d]# /usr/local/sbin/gmond -d 5 -c
/etc/ganglia/gmond.conf
loaded module: core_metrics
loaded module: python_module
loaded module: cpu_module
loaded module: disk_module
loaded module: load_module
loaded module: mem_module
loaded module: net_module
loaded module: proc_module
loaded module: sys_module
loaded module: python_module
Segmentation fault (core dumped)
I'm not sure where to begin checking. I'm a very old-fashioned
debugger - I tend to use a great deal of print statements to
track down where things are happening. I can start doing this
in gmond.
I tried putting fprintf's all over the gmond.c (yep - I'm that
poor of a debugger). I'm not sure but if looks like it segfaults
in the function setup_metric_callbacks on the statement,
if (modp-init modp-init(global_context)) {
or on the function,
apr_pool_cleanup_register(global_context, modp,
modular_metric_cleanup,
apr_pool_cleanup_null);
I'm not too sure.
I apologize if I'm wasting your time with my poor debugging
skills.
Thanks!
Jeff
--
Managing the Performance of Cloud-Based Applications
Take advantage of what the Cloud has to offer - Avoid Common Pitfalls.
Read the Whitepaper.
http://pubads.g.doubleclick.net/gampad/clk?id=121051231iu=/4140/ostg.clktrk
___
Ganglia-developers mailing list
Ganglia-developers@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ganglia-developers
Re: [Ganglia-developers] gmond segfault with libpython
Sure - let me run off and do this.
In the meantime, I tried rebuilding ganglia with 3.5.0 and
3.4.0. For both of these cases, I get the same segfault. This
makes me think I'm doing something wrong but I can seem
to find it :)
I'll let you know what I find out as I go back to ganglia-3.6.0.
Thanks!
Jeff
Oh I didn't think about going that lowlevel :) Could you run ldd on
gmond also?
Could you also run 'tree' command on /etc/ganglia ? It's interesting
that you have two times msg: loaded module: python_module while
starting gmond. Rechecking this with strace log shows that it looks like
double loading of those modules? http://pastebin.com/BjdCGgbj
On Sun, Feb 09, 2014 at 03:28:10PM -0500, Jeff Layton wrote:
On 02/09/2014 02:48 PM, Jeff Layton wrote:
On 02/09/2014 02:28 PM, Maciej Lasyk wrote:
You could also try to catch on which particular check this segfault
happens..?
Not sure how to check this. When I run gmond interactively, it
segfaults just after it says,
[root@home4 yum.repos.d]# /usr/local/sbin/gmond -d 5 -c
/etc/ganglia/gmond.conf
loaded module: core_metrics
loaded module: python_module
loaded module: cpu_module
loaded module: disk_module
loaded module: load_module
loaded module: mem_module
loaded module: net_module
loaded module: proc_module
loaded module: sys_module
loaded module: python_module
Segmentation fault (core dumped)
I'm not sure where to begin checking. I'm a very old-fashioned
debugger - I tend to use a great deal of print statements to
track down where things are happening. I can start doing this
in gmond.
I tried putting fprintf's all over the gmond.c (yep - I'm that
poor of a debugger). I'm not sure but if looks like it segfaults
in the function setup_metric_callbacks on the statement,
if (modp-init modp-init(global_context)) {
or on the function,
apr_pool_cleanup_register(global_context, modp,
modular_metric_cleanup,
apr_pool_cleanup_null);
I'm not too sure.
I apologize if I'm wasting your time with my poor debugging
skills.
Thanks!
Jeff
--
Managing the Performance of Cloud-Based Applications
Take advantage of what the Cloud has to offer - Avoid Common Pitfalls.
Read the Whitepaper.
http://pubads.g.doubleclick.net/gampad/clk?id=121051231iu=/4140/ostg.clktrk
___
Ganglia-developers mailing list
Ganglia-developers@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ganglia-developers
--
Managing the Performance of Cloud-Based Applications
Take advantage of what the Cloud has to offer - Avoid Common Pitfalls.
Read the Whitepaper.
http://pubads.g.doubleclick.net/gampad/clk?id=121051231iu=/4140/ostg.clktrk___
Ganglia-developers mailing list
Ganglia-developers@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ganglia-developers
Re: [Ganglia-developers] gmond segfault with libpython
I hope this isn't too much output (I've heard about pastebin.com
but never really used it).
[root@home4 ganglia-3.6.0]# ldd /usr/local/sbin/gmond
linux-vdso.so.1 = (0x7fff667f6000)
libapr-1.so.0 = /usr/lib64/libapr-1.so.0 (0x7f6a24049000)
libresolv.so.2 = /lib64/libresolv.so.2 (0x00337dc0)
libganglia-3.6.0.so.0 = /usr/local/lib64/libganglia-3.6.0.so.0
(0x7f6a23e0d000)
libdl.so.2 = /lib64/libdl.so.2 (0x00337c40)
libnsl.so.1 = /lib64/libnsl.so.1 (0x003390c0)
libz.so.1 = /lib64/libz.so.1 (0x00337d00)
libpcre.so.0 = /lib64/libpcre.so.0 (0x003f7360)
libexpat.so.1 = /lib64/libexpat.so.1 (0x00337f80)
libconfuse.so.0 = /usr/lib64/libconfuse.so.0 (0x7f6a23bff000)
libpthread.so.0 = /lib64/libpthread.so.0 (0x00337c80)
libc.so.6 = /lib64/libc.so.6 (0x00337c00)
libuuid.so.1 = /lib64/libuuid.so.1 (0x00338380)
libcrypt.so.1 = /lib64/libcrypt.so.1 (0x00338c60)
/lib64/ld-linux-x86-64.so.2 (0x00337bc0)
libfreebl3.so = /usr/lib64/libfreebl3.so (0x00338ca0)
Below is the tree output:
[root@home4 ganglia-3.6.0]# tree /etc/ganglia
/etc/ganglia
??? conf.d
? ??? procstat.pyconf
??? gmetad.conf
??? gmond.conf
1 directory, 3 files
I looked at the strace file for process 3537 and I did see two places
where gmond does an access() on the python_modules directory.
Does gmond automatically look for the python modes so I don't need
to put them the modules section of gmond.conf?
Thanks a million!
Jeff
Oh I didn't think about going that lowlevel :) Could you run ldd on
gmond also?
Could you also run 'tree' command on /etc/ganglia ? It's interesting
that you have two times msg: loaded module: python_module while
starting gmond. Rechecking this with strace log shows that it looks like
double loading of those modules? http://pastebin.com/BjdCGgbj
On Sun, Feb 09, 2014 at 03:28:10PM -0500, Jeff Layton wrote:
On 02/09/2014 02:48 PM, Jeff Layton wrote:
On 02/09/2014 02:28 PM, Maciej Lasyk wrote:
You could also try to catch on which particular check this segfault
happens..?
Not sure how to check this. When I run gmond interactively, it
segfaults just after it says,
[root@home4 yum.repos.d]# /usr/local/sbin/gmond -d 5 -c
/etc/ganglia/gmond.conf
loaded module: core_metrics
loaded module: python_module
loaded module: cpu_module
loaded module: disk_module
loaded module: load_module
loaded module: mem_module
loaded module: net_module
loaded module: proc_module
loaded module: sys_module
loaded module: python_module
Segmentation fault (core dumped)
I'm not sure where to begin checking. I'm a very old-fashioned
debugger - I tend to use a great deal of print statements to
track down where things are happening. I can start doing this
in gmond.
I tried putting fprintf's all over the gmond.c (yep - I'm that
poor of a debugger). I'm not sure but if looks like it segfaults
in the function setup_metric_callbacks on the statement,
if (modp-init modp-init(global_context)) {
or on the function,
apr_pool_cleanup_register(global_context, modp,
modular_metric_cleanup,
apr_pool_cleanup_null);
I'm not too sure.
I apologize if I'm wasting your time with my poor debugging
skills.
Thanks!
Jeff
--
Managing the Performance of Cloud-Based Applications
Take advantage of what the Cloud has to offer - Avoid Common Pitfalls.
Read the Whitepaper.
http://pubads.g.doubleclick.net/gampad/clk?id=121051231iu=/4140/ostg.clktrk
___
Ganglia-developers mailing list
Ganglia-developers@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ganglia-developers
--
Managing the Performance of Cloud-Based Applications
Take advantage of what the Cloud has to offer - Avoid Common Pitfalls.
Read the Whitepaper.
http://pubads.g.doubleclick.net/gampad/clk?id=121051231iu=/4140/ostg.clktrk___
Ganglia-developers mailing list
Ganglia-developers@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ganglia-developers
Re: [Ganglia-developers] gmond segfault with libpython
Could you post here your build options (that ones you entered while
./configure) and also could you paste gmond.conf into pastebin?
Also plz strace one more time, but now with strace -s 1024 -e trace=file
and paste the output to pastebin
On Sun, Feb 09, 2014 at 04:55:14PM -0500, Jeff Layton wrote:
I hope this isn't too much output (I've heard about pastebin.com
but never really used it).
[root@home4 ganglia-3.6.0]# ldd /usr/local/sbin/gmond
linux-vdso.so.1 = (0x7fff667f6000)
libapr-1.so.0 = /usr/lib64/libapr-1.so.0 (0x7f6a24049000)
libresolv.so.2 = /lib64/libresolv.so.2 (0x00337dc0)
libganglia-3.6.0.so.0 =
/usr/local/lib64/libganglia-3.6.0.so.0 (0x7f6a23e0d000)
libdl.so.2 = /lib64/libdl.so.2 (0x00337c40)
libnsl.so.1 = /lib64/libnsl.so.1 (0x003390c0)
libz.so.1 = /lib64/libz.so.1 (0x00337d00)
libpcre.so.0 = /lib64/libpcre.so.0 (0x003f7360)
libexpat.so.1 = /lib64/libexpat.so.1 (0x00337f80)
libconfuse.so.0 = /usr/lib64/libconfuse.so.0 (0x7f6a23bff000)
libpthread.so.0 = /lib64/libpthread.so.0 (0x00337c80)
libc.so.6 = /lib64/libc.so.6 (0x00337c00)
libuuid.so.1 = /lib64/libuuid.so.1 (0x00338380)
libcrypt.so.1 = /lib64/libcrypt.so.1 (0x00338c60)
/lib64/ld-linux-x86-64.so.2 (0x00337bc0)
libfreebl3.so = /usr/lib64/libfreebl3.so (0x00338ca0)
Below is the tree output:
[root@home4 ganglia-3.6.0]# tree /etc/ganglia
/etc/ganglia
??? conf.d
? ??? procstat.pyconf
??? gmetad.conf
??? gmond.conf
1 directory, 3 files
I looked at the strace file for process 3537 and I did see two places
where gmond does an access() on the python_modules directory.
Does gmond automatically look for the python modes so I don't need
to put them the modules section of gmond.conf?
Thanks a million!
Jeff
Oh I didn't think about going that lowlevel :) Could you run ldd on
gmond also?
Could you also run 'tree' command on /etc/ganglia ? It's interesting
that you have two times msg: loaded module: python_module while
starting gmond. Rechecking this with strace log shows that it looks like
double loading of those modules? http://pastebin.com/BjdCGgbj
On Sun, Feb 09, 2014 at 03:28:10PM -0500, Jeff Layton wrote:
On 02/09/2014 02:48 PM, Jeff Layton wrote:
On 02/09/2014 02:28 PM, Maciej Lasyk wrote:
You could also try to catch on which particular check this segfault
happens..?
Not sure how to check this. When I run gmond interactively, it
segfaults just after it says,
[root@home4 yum.repos.d]# /usr/local/sbin/gmond -d 5 -c
/etc/ganglia/gmond.conf
loaded module: core_metrics
loaded module: python_module
loaded module: cpu_module
loaded module: disk_module
loaded module: load_module
loaded module: mem_module
loaded module: net_module
loaded module: proc_module
loaded module: sys_module
loaded module: python_module
Segmentation fault (core dumped)
I'm not sure where to begin checking. I'm a very old-fashioned
debugger - I tend to use a great deal of print statements to
track down where things are happening. I can start doing this
in gmond.
I tried putting fprintf's all over the gmond.c (yep - I'm that
poor of a debugger). I'm not sure but if looks like it segfaults
in the function setup_metric_callbacks on the statement,
if (modp-init modp-init(global_context)) {
or on the function,
apr_pool_cleanup_register(global_context, modp,
modular_metric_cleanup,
apr_pool_cleanup_null);
I'm not too sure.
I apologize if I'm wasting your time with my poor debugging
skills.
Thanks!
Jeff
--
Managing the Performance of Cloud-Based Applications
Take advantage of what the Cloud has to offer - Avoid Common Pitfalls.
Read the Whitepaper.
http://pubads.g.doubleclick.net/gampad/clk?id=121051231iu=/4140/ostg.clktrk
___
Ganglia-developers mailing list
Ganglia-developers@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ganglia-developers
--
--
pozdrawiam,
Maciej Lasyk
GPG key ID: FFA8AEEC
GPG info: http://maciek.lasyk.info/gpg.txt
pgphnhmNENVnW.pgp
Description: PGP signature
--
Managing the Performance of Cloud-Based Applications
Take advantage of what the Cloud has to offer - Avoid Common Pitfalls.
Read the Whitepaper.
http://pubads.g.doubleclick.net/gampad/clk?id=121051231iu=/4140/ostg.clktrk___
Ganglia-developers mailing list
Ganglia-developers@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ganglia-developers
Re: [Ganglia-developers] gmond segfault with libpython
Do we still have a maintainer for the Ganglia packages for EPEL? If not, should we see if somebody would like to fill that position? Thanks, Bernard On Sun, Feb 9, 2014 at 6:15 PM, Vladimir Vuksan vli...@veus.hr wrote: Those RPMS work just fine for me [root@localhost ~]# uname -a Linux localhost.localdomain 2.6.32-431.el6.x86_64 #1 SMP Fri Nov 22 03:15:09 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux [root@localhost ~]# cat /etc/issue CentOS release 6.5 (Final) Kernel \r on an \m [root@localhost ~]# rpm -ivh http://vuksan.com/centos/RPMS-6/x86_64/ganglia-gmond-modules-python-3.6.0-1.x86_64.rpm http://vuksan.com/centos/RPMS-6/x86_64/libganglia-3.6.0-1.x86_64.rpm http://vuksan.com/centos/RPMS-6/x86_64/ganglia-gmond-3.6.0-1.x86_64.rpm http://vuksan.com/centos/RPMS-6/x86_64/libconfuse-2.6-2.el6.rf.x86_64.rpm Retrieving http://vuksan.com/centos/RPMS-6/x86_64/ganglia-gmond-modules-python-3.6.0-1.x86_64.rpm Retrieving http://vuksan.com/centos/RPMS-6/x86_64/libganglia-3.6.0-1.x86_64.rpm Retrieving http://vuksan.com/centos/RPMS-6/x86_64/ganglia-gmond-3.6.0-1.x86_64.rpm Retrieving http://vuksan.com/centos/RPMS-6/x86_64/libconfuse-2.6-2.el6.rf.x86_64.rpm warning: /var/tmp/rpm-tmp.JtASFF: Header V3 DSA/SHA1 Signature, key ID 6b8d79e6: NOKEY Preparing...### [100%] 1:libconfuse ### [ 25%] 2:libganglia ### [ 50%] 3:ganglia-gmond ### [ 75%] 4:ganglia-gmond-modules-p### [100%] [root@localhost ~]# gmond -d 2 loaded module: core_metrics loaded module: cpu_module loaded module: disk_module loaded module: load_module loaded module: mem_module loaded module: net_module loaded module: proc_module loaded module: sys_module loaded module: python_module udp_recv_channel mcast_join=239.2.11.71 mcast_if=NULL port=8649 bind=239.2.11.71 buffer=0 On 02/09/2014 09:39 AM, Jeff Layton wrote: Vladimir, I initially tried your binaries on my 6.5 system and I could not get them to install and run (I think they were built with a 6.3 system). At some point I'll try building the rpm's and installing those. Hopefully there is no different in the build process - that would be very interesting if the rpm's worked and building from source didn't :) I'll let you know - but first I'm going to try Maciej's strace idea. Thanks! Jeff P.S. There are some pretty significant differences between 6.4 and 6.5. One big one that I know of is the ntp format changed. I have not seen issues with Centos 6 however I usually build my RPM packages. You could do that if you type rpmbuild -tb ganglia-3.6.0.tar.gz Alternatively if you are interested to try prebuilt packages you can find them here. http://vuksan.com/centos/RPMS-6/x86_64/ Vladimir On 02/08/2014 11:11 AM, Jeff Layton wrote: Good morning, I'm running a CentOS 6.5 system with ganglia 3.6.0 and ganglia-web 3.5.12. I'm following the general guidelines in this article: http://sachinsharm.wordpress.com/tag/installing-ganglia/ Everything goes swimmingly and ganglia itself works fine. So I decide to go to the next step and try using Python with gmond. I followed the general guidelines in this article: http://sachinsharm.wordpress.com/2013/08/19/setup-and-configure-ganglia-python-modules-on-centosrhel-6-3/ But when I start up gmond I get a segfault as reported in /var/log/messages. Feb 5 19:58:47 home4 kernel: gmond[17992]: segfault at 8 ip 0036a7ce6ceb sp 7fffaad46bf0 error 4 in libpython2.6.so.1.0[36a7c0+15d000] Feb 5 19:58:47 home4 abrt[18003]: Saved core dump of pid 17992 (/usr/local/sbin/gmond) to /var/spool/abrt/ccpp-2014-02-05-19:58:47-17992 (4284416 bytes) Feb 5 19:58:47 home4 abrtd: Directory 'ccpp-2014-02-05-19:58:47-17992' creation detected Feb 5 19:58:47 home4 abrtd: Executable '/usr/local/sbin/gmond' doesn't belong to any package and ProcessUnpackaged is set to 'no' Feb 5 19:58:47 home4 abrtd: 'post-create' on '/var/spool/abrt/ccpp-2014-02-05-19:58:47-17992' exited with 1 Feb 5 19:58:47 home4 abrtd: Deleting problem directory '/var/spool/abrt/ccpp-2014-02-05-19:58:47-17992' I'm been trying to debug this but I have to admit that I'm coming up blank. Running gmond with debug doesn't give too much information: [root@home4 laytonjb]# gmond -d 5 -c /etc/ganglia/gmond.conf loaded module: core_metrics loaded module: cpu_module loaded module: disk_module loaded module: load_module loaded module: mem_module loaded module: net_module loaded module: proc_module loaded module: sys_module loaded module: python_module loaded module: python_module Segmentation fault (core dumped) I went back to the Mod_Python section of the book (p. 89) and tried to make things as simple as possible to trace down the
Re: [Ganglia-developers] gmond segfault with libpython
On Sat, Feb 08, 2014 at 11:11:19AM -0500, Jeff Layton wrote: Good morning, I'm running a CentOS 6.5 system with ganglia 3.6.0 and ganglia-web 3.5.12. I'm following the general guidelines in this article: http://sachinsharm.wordpress.com/tag/installing-ganglia/ Everything goes swimmingly and ganglia itself works fine. So I decide to go to the next step and try using Python with gmond. I followed the general guidelines in this article: http://sachinsharm.wordpress.com/2013/08/19/setup-and-configure-ganglia-python-modules-on-centosrhel-6-3/ But when I start up gmond I get a segfault as reported in /var/log/messages. Feb 5 19:58:47 home4 kernel: gmond[17992]: segfault at 8 ip 0036a7ce6ceb sp 7fffaad46bf0 error 4 in libpython2.6.so.1.0[36a7c0+15d000] Feb 5 19:58:47 home4 abrt[18003]: Saved core dump of pid 17992 (/usr/local/sbin/gmond) to /var/spool/abrt/ccpp-2014-02-05-19:58:47-17992 (4284416 bytes) Feb 5 19:58:47 home4 abrtd: Directory 'ccpp-2014-02-05-19:58:47-17992' creation detected Feb 5 19:58:47 home4 abrtd: Executable '/usr/local/sbin/gmond' doesn't belong to any package and ProcessUnpackaged is set to 'no' Feb 5 19:58:47 home4 abrtd: 'post-create' on '/var/spool/abrt/ccpp-2014-02-05-19:58:47-17992' exited with 1 Feb 5 19:58:47 home4 abrtd: Deleting problem directory '/var/spool/abrt/ccpp-2014-02-05-19:58:47-17992' I'm been trying to debug this but I have to admit that I'm coming up blank. Running gmond with debug doesn't give too much information: [root@home4 laytonjb]# gmond -d 5 -c /etc/ganglia/gmond.conf loaded module: core_metrics loaded module: cpu_module loaded module: disk_module loaded module: load_module loaded module: mem_module loaded module: net_module loaded module: proc_module loaded module: sys_module loaded module: python_module loaded module: python_module Segmentation fault (core dumped) Could you try here stracing it? strace -s 1024 -ff -o strace.log gmond-c /etc/ganglia/gmond.conf ? I never had problems with Centos and Ganglia. I created my packages myself with fpm (https://github.com/jordansissel/fpm) and also used built packages from Epel. -- pozdrawiam, Maciej Lasyk GPG key ID: FFA8AEEC GPG info: http://maciek.lasyk.info/gpg.txt pgpK9xkxUEvr9.pgp Description: PGP signature -- Managing the Performance of Cloud-Based Applications Take advantage of what the Cloud has to offer - Avoid Common Pitfalls. Read the Whitepaper. http://pubads.g.doubleclick.net/gampad/clk?id=121051231iu=/4140/ostg.clktrk___ Ganglia-developers mailing list Ganglia-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ganglia-developers
