Re: [SOGo] Sogod process don't start correctly
Hi to all, again. We've still the same problem. I've downgraded SOGo to version 1.3.12x that was working ok for us until some weeks ago and the behavior is the same. RPM Packages now are: sogo-1.3.12c-1.centos6.x86_64 sope49-appserver-4.9-20120214_1664.el6.1.x86_64 sope49-cards-1.3.12c-1.centos6.x86_64 sope49-core-4.9-20120214_1664.el6.1.x86_64 sope49-gdl1-4.9-20120214_1664.el6.1.x86_64 sope49-gdl1-contentstore-1.3.12c-1.centos6.x86_64 sope49-gdl1-mysql-4.9-20120214_1664.el6.1.x86_64 sope49-ldap-4.9-20120214_1664.el6.1.x86_64 sope49-mime-4.9-20120214_1664.el6.1.x86_64 sope49-sbjson-2.3.1-20120214_1664.el6.1.x86_64 sope49-xml-4.9-20120214_1664.el6.1.x86_64 It seems as if sogod process is saving some kind of state from last execution.., or lockfile, or something. I'm searching the server for this kind of lock files, temporary files, etc. but I couldn't find anything. Anyone has an idea of how can I continue debugging this problem? Many thanks in advance -- Lic. Gabriel Gomiz - Jefe de Sistemas / Administrador Gerencia de Sistemas - Cooperativa Obrera Ltda. Tel: (0291) 456-0084 On 12/19/2012 06:57 PM, Gabriel Gomiz wrote: I've ran again with gnustep debuginfo installed and first two entries in backtrace are resolved. Backtrace head is: #0 0x7442a735 in _xstat () from /lib64/libc.so.6 #1 0x751d33f7 in -[NSFileManager fileExistsAtPath:isDirectory:] (self=0x7838f158, _cmd=0x756b39b0, path=0x785f6d58, isDirectory=0x0) at NSFileManager.m:1516 #2 0x751d3352 in -[NSFileManager fileExistsAtPath:] (self=0x7838f158, _cmd=0x760c1580, path=0x785f6d58) at NSFileManager.m:1470 . . Sorry for the noise. signature.asc Description: OpenPGP digital signature
Re: [SOGo] Sogod process don't start correctly
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 hi gabriel, Am 20.12.2012 12:27 schrieb Gabriel Gomiz: Anyone has an idea of how can I continue debugging this problem? maybe try to strace the process? that way, you should at least be able to see the files sogod is accessing. with kind regards, t. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAlDS/0EACgkQjgOUn50Ju0CJOACeNP/7aeIw3VG5Pm3CcdcgrHM4 dpoAnjIS6n8nmWr0eGSjrnwi4pheXxRb =Lkq6 -END PGP SIGNATURE- -- users@sogo.nu https://inverse.ca/sogo/lists
Re: [SOGo] Sogod process don't start correctly
On 12/20/2012 09:06 AM, Thoralf Schulze wrote: hi gabriel, Am 20.12.2012 12:27 schrieb Gabriel Gomiz: Anyone has an idea of how can I continue debugging this problem? maybe try to strace the process? that way, you should at least be able to see the files sogod is accessing. Thoralf: Thanks, I've followed your suggestion. Executing this command: # strace /usr/sbin/sogod -WOWorkersCount 5 -WOPidFile /var/run/sogo/sogo.pid -WOLogFile /var/log/sogo/sogo.log -WONoDetach YES ... Lots of output ... stat(/boot/bundle-info.plist, 0x7fff50b0b940) = -1 ENOENT (No such file or directory) stat(/net/bundle-info.plist, And then hangs in that last system call. BTW, /net directory is empty. What is sogod searching in there? with kind regards, t. signature.asc Description: OpenPGP digital signature
Re: [SOGo] Sogod process don't start correctly
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 hi gabriel, on 20.12.2012 13:14, Gabriel Gomiz wrote: # strace /usr/sbin/sogod -WOWorkersCount 5 -WOPidFile /var/run/sogo/sogo.pid -WOLogFile /var/log/sogo/sogo.log -WONoDetach YES i would set WOWorkersCount to 1, makes debugging easier :-) ... Lots of output ... stat(/boot/bundle-info.plist, 0x7fff50b0b940) = -1 ENOENT (No such file or directory) stat(/net/bundle-info.plist, And then hangs in that last system call. BTW, /net directory is empty. What is sogod searching in there? good question, these directories do look really strange for sure … and sorry, i'm not really familiar with centos / the rpm flavour of sogo. on my debian system, the sogo debs provide these bundle-info.plist files: thoralf@sogo:~$ dpkg -L $(dpkg -l | awk '/sope|sogo/ {print $2}') | grep bundle-info.plist /usr/lib/GNUstep/WOxElemBuilders-4.9/WOExtensions.wox/bundle-info.plist /usr/lib/GNUstep/WOxElemBuilders-4.9/WEExtensions.wox/bundle-info.plist /usr/lib/GNUstep/WOxElemBuilders-4.9/SOGoElements.wox/bundle-info.plist /usr/lib/GNUstep/SOGo/Appointments.SOGo/bundle-info.plist /usr/lib/GNUstep/SOGo/ContactsUI.SOGo/bundle-info.plist /usr/lib/GNUstep/SOGo/MailerUI.SOGo/bundle-info.plist /usr/lib/GNUstep/SOGo/Mailer.SOGo/bundle-info.plist /usr/lib/GNUstep/SOGo/CommonUI.SOGo/bundle-info.plist /usr/lib/GNUstep/SOGo/SchedulerUI.SOGo/bundle-info.plist /usr/lib/GNUstep/SaxDrivers-4.9/versitCardsSaxDriver.sax/Resources/bundle-info.plist /usr/lib/GNUstep/SaxDrivers-4.9/versitCardsSaxDriver.sax/bundle-info.plist /usr/lib/GNUstep/SaxDrivers-4.9/libxmlSAXDriver.sax/Resources/bundle-info.plist /usr/lib/GNUstep/SaxDrivers-4.9/libxmlSAXDriver.sax/bundle-info.plist thoralf@sogo:~$ the debian init script sources /usr/share/GNUstep/Makefiles/GNUstep.sh , there seems to be some path magic happening in there. might be worth looking at as well … hth, t. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAlDTBfYACgkQjgOUn50Ju0A0zQCfSH7MQ20D/gyyntHJRTwteD06 52sAnieUj+o08/F57Tvc+v3lOBy56Hhh =4XoQ -END PGP SIGNATURE- -- users@sogo.nu https://inverse.ca/sogo/lists
[SOGo] Sogod process don't start correctly
Hi to all. We're having a problem with sogod version 2.0.3 on CentOS 6.3 server. After last night sogod process hanged with the following messages: Dec 19 03:18:34 sogod [20103]: [WARN] 0x0x7feab4339908[WOWatchDogChild] pid 14374 has been hanging in the same request for 1 minutes Dec 19 03:18:34 sogod [20103]: [WARN] 0x0x7feab430a838[WOWatchDogChild] pid 14375 has been hanging in the same request for 1 minutes Dec 19 03:18:38 sogod [20103]: [WARN] 0x0x7feab4434be8[WOWatchDogChild] pid 14387 has been hanging in the same request for 1 minutes Dec 19 03:19:34 sogod [20103]: [WARN] 0x0x7feab4339908[WOWatchDogChild] pid 14374 has been hanging in the same request for 2 minutes Dec 19 03:19:34 sogod [20103]: [WARN] 0x0x7feab430a838[WOWatchDogChild] pid 14375 has been hanging in the same request for 2 minutes Dec 19 03:19:38 sogod [20103]: [WARN] 0x0x7feab4434be8[WOWatchDogChild] pid 14387 has been hanging in the same request for 2 minutes Dec 19 03:20:34 sogod [20103]: [WARN] 0x0x7feab4339908[WOWatchDogChild] pid 14374 has been hanging in the same request for 3 minutes Dec 19 03:20:34 sogod [20103]: [WARN] 0x0x7feab430a838[WOWatchDogChild] pid 14375 has been hanging in the same request for 3 minutes Dec 19 03:20:38 sogod [20103]: [WARN] 0x0x7feab4434be8[WOWatchDogChild] pid 14387 has been hanging in the same request for 3 minutes Dec 19 03:21:09 sogod [20103]: 0x0x7feab4434be8[WOWatchDogChild] sending terminate signal to pid 14387 Dec 19 03:21:09 sogod [20103]: 0x0x7feab4339908[WOWatchDogChild] sending terminate signal to pid 14374 Dec 19 03:21:09 sogod [20103]: 0x0x7feab430a838[WOWatchDogChild] sending terminate signal to pid 14375 We had to kill -9 the processes because it didn't respond to any requests. Now we start sogod service but the process hangs after logging the following messages: Dec 19 11:32:44 sogod [24424]: version 2.0.3 (build root@shiva 201212070954) -- starting Dec 19 11:32:44 sogod [24424]: vmem size check enabled: shutting down app when vmem 384 MB Dec 19 11:32:44 sogod [24424]: 0x0x7f9f3efc3e88[SOGoProductLoader] SOGo products loaded from '/usr/lib64/GNUstep/SOGo': Dec 19 11:32:44 sogod [24424]: 0x0x7f9f3efc3e88[SOGoProductLoader] Contacts.SOGo, AdministrationUI.SOGo, Appointments.SOGo, Mailer.SOGo, CommonUI.SOGo, SchedulerUI.SOGo, PreferencesUI.SOGo, MailPartViewers.SOGo, MainUI.SOGo, ContactsUI.SOGo, MailerUI.SOGo SOGo port 2 is not listening. And we see only one process running despite are 5 workers configured ps shows: sogo 52255 0.0 0.0 282104 11992 ?S12:52 0:00 /usr/sbin/sogod -WOWorkersCount 5 -WOPidFile /var/run/sogo/sogo.pid -WOLogFile /var/log/sogo/sogo.log Memcached is running fine. I've also ran a strace to the sogod process and it did not execute any system call. Any help? Mnay thanks! -- Lic. Gabriel Gomiz - Jefe de Sistemas / Administrador Gerencia de Sistemas - Cooperativa Obrera Ltda. Tel: (0291) 456-0084 signature.asc Description: OpenPGP digital signature
Re: [SOGo] Sogod process don't start correctly
Christian, thanks for your answer. We've installed the following RPM packages directly from inverse YUM repos: sogo-2.0.3a-1.centos6.x86_64 sope49-appserver-4.9-20121206_1664.el6.1.x86_64 sope49-cards-2.0.3a-1.centos6.x86_64 sope49-core-4.9-20121206_1664.el6.1.x86_64 sope49-gdl1-4.9-20121206_1664.el6.1.x86_64 sope49-gdl1-contentstore-2.0.3a-1.centos6.x86_64 sope49-gdl1-mysql-4.9-20121206_1664.el6.1.x86_64 sope49-ldap-4.9-20121206_1664.el6.1.x86_64 sope49-mime-4.9-20121206_1664.el6.1.x86_64 sope49-sbjson-2.3.1-20121206_1664.el6.1.x86_64 sope49-xml-4.9-20121206_1664.el6.1.x86_64 The server is not swapping at all. Investigating in logs a little more I see that first indication of problems is here: Dec 18 12:01:18 sogod [20121]: 0xB481B9A8[SOGoAppointmentObject]:914e9c69-5aec-4ae6-9ae1-aa35bbbc81d6.icsD etag 'gcs0001' matches: gcs0001 then Dec 18 12:02:18 sogod [20103]: [WARN] 0x0x7feab4339908[WOWatchDogChild] pid 20121 has been hanging in the same request for 1 minutes. Anyway, my main concern now is finding why the sogod process does not start... :S I've tried restarting memcached, I've activated debug directives in .GNUstepDefaults and nothing new gets logged. It doesn't even connect to database. The only file descriptors that the process has are: 0 - /dev/pts/3 1 - /var/log/sogo/sogo.log 2 - /var/log/sogo/sogo.log -- Lic. Gabriel Gomiz - Jefe de Sistemas / Administrador Gerencia de Sistemas - Cooperativa Obrera Ltda. Tel: (0291) 456-0084 On 12/19/2012 02:22 PM, Christian Mack wrote: On 2012-12-19 17:22, Gabriel Gomiz wrote: Hi to all. We're having a problem with sogod version 2.0.3 on CentOS 6.3 server. After last night sogod process hanged with the following messages: Dec 19 03:18:34 sogod [20103]: [WARN] 0x0x7feab4339908[WOWatchDogChild] pid 14374 has been hanging in the same request for 1 minutes Dec 19 03:18:34 sogod [20103]: [WARN] 0x0x7feab430a838[WOWatchDogChild] pid 14375 has been hanging in the same request for 1 minutes Dec 19 03:18:38 sogod [20103]: [WARN] 0x0x7feab4434be8[WOWatchDogChild] pid 14387 has been hanging in the same request for 1 minutes Dec 19 03:19:34 sogod [20103]: [WARN] 0x0x7feab4339908[WOWatchDogChild] pid 14374 has been hanging in the same request for 2 minutes Dec 19 03:19:34 sogod [20103]: [WARN] 0x0x7feab430a838[WOWatchDogChild] pid 14375 has been hanging in the same request for 2 minutes Dec 19 03:19:38 sogod [20103]: [WARN] 0x0x7feab4434be8[WOWatchDogChild] pid 14387 has been hanging in the same request for 2 minutes Dec 19 03:20:34 sogod [20103]: [WARN] 0x0x7feab4339908[WOWatchDogChild] pid 14374 has been hanging in the same request for 3 minutes Dec 19 03:20:34 sogod [20103]: [WARN] 0x0x7feab430a838[WOWatchDogChild] pid 14375 has been hanging in the same request for 3 minutes Dec 19 03:20:38 sogod [20103]: [WARN] 0x0x7feab4434be8[WOWatchDogChild] pid 14387 has been hanging in the same request for 3 minutes Dec 19 03:21:09 sogod [20103]: 0x0x7feab4434be8[WOWatchDogChild] sending terminate signal to pid 14387 Dec 19 03:21:09 sogod [20103]: 0x0x7feab4339908[WOWatchDogChild] sending terminate signal to pid 14374 Dec 19 03:21:09 sogod [20103]: 0x0x7feab430a838[WOWatchDogChild] sending terminate signal to pid 14375 We had to kill -9 the processes because it didn't respond to any requests. Now we start sogod service but the process hangs after logging the following messages: Dec 19 11:32:44 sogod [24424]: version 2.0.3 (build root@shiva 201212070954) -- starting Dec 19 11:32:44 sogod [24424]: vmem size check enabled: shutting down app when vmem 384 MB Dec 19 11:32:44 sogod [24424]: 0x0x7f9f3efc3e88[SOGoProductLoader] SOGo products loaded from '/usr/lib64/GNUstep/SOGo': Dec 19 11:32:44 sogod [24424]: 0x0x7f9f3efc3e88[SOGoProductLoader] Contacts.SOGo, AdministrationUI.SOGo, Appointments.SOGo, Mailer.SOGo, CommonUI.SOGo, SchedulerUI.SOGo, PreferencesUI.SOGo, MailPartViewers.SOGo, MainUI.SOGo, ContactsUI.SOGo, MailerUI.SOGo SOGo port 2 is not listening. And we see only one process running despite are 5 workers configured ps shows: sogo 52255 0.0 0.0 282104 11992 ?S12:52 0:00 /usr/sbin/sogod -WOWorkersCount 5 -WOPidFile /var/run/sogo/sogo.pid -WOLogFile /var/log/sogo/sogo.log Memcached is running fine. I've also ran a strace to the sogod process and it did not execute any system call. Any help? Do you have the matching SOPE packages installed? Does your server by chance start swapping? Kind regards, Christian Mack signature.asc Description: OpenPGP digital signature
Re: [SOGo] Sogod process don't start correctly
I've ran again with gnustep debuginfo installed and first two entries in backtrace are resolved. Backtrace head is: #0 0x7442a735 in _xstat () from /lib64/libc.so.6 #1 0x751d33f7 in -[NSFileManager fileExistsAtPath:isDirectory:] (self=0x7838f158, _cmd=0x756b39b0, path=0x785f6d58, isDirectory=0x0) at NSFileManager.m:1516 #2 0x751d3352 in -[NSFileManager fileExistsAtPath:] (self=0x7838f158, _cmd=0x760c1580, path=0x785f6d58) at NSFileManager.m:1470 . . Sorry for the noise. -- Lic. Gabriel Gomiz - Jefe de Sistemas / Administrador Gerencia de Sistemas - Cooperativa Obrera Ltda. Tel: (0291) 456-0084 signature.asc Description: OpenPGP digital signature