Re: [gt-user] globus-job-status: UNKNOWN JOB STATE 0
Thanks. Joseph! That helped! Nikolay. Joseph Bester wrote on 25/01/10 20:54: > On Jan 23, 2010, at 4:53 PM, Stuart Martin wrote: >> Joe, >> >> Can you help debug what might be wrong when Nikolay starts the seg: >> >>> [root]$ globus-job-manager-event-generator -scheduler pbs -background >>> -pidfile $GLOBUS_LOCATION/var/globus-job-manager-seg-pbs.pid >>> Error: pbs not configured >> >> -Stu > > Run gpt-postinstall -force to get the setup package to be rerun so that > it will know about the PBS setup. There is currently an unexpressed > order dependency between the PBS SEG and job manager SEG setup packages, > so depending on install order they may get run in the wrong order. > > Joe >
Re: [gt-user] globus-job-status: UNKNOWN JOB STATE 0
On Jan 23, 2010, at 4:53 PM, Stuart Martin wrote: Joe, Can you help debug what might be wrong when Nikolay starts the seg: [root]$ globus-job-manager-event-generator -scheduler pbs -background -pidfile $GLOBUS_LOCATION/var/globus-job-manager-seg-pbs.pid Error: pbs not configured -Stu Run gpt-postinstall -force to get the setup package to be rerun so that it will know about the PBS setup. There is currently an unexpressed order dependency between the PBS SEG and job manager SEG setup packages, so depending on install order they may get run in the wrong order. Joe
Re: [gt-user] globus-job-status: UNKNOWN JOB STATE 0
On Jan 22, 2010, at 8:43 AM, Nikolay Kutovskiy wrote: Hello list, I have installed GRAM5 to use PBS and get the following output of globus-job-status command: $ globus-job-submit :2119/jobmanager-pbs /bin/hostname https://: 51499/16073727533535086921/7782764993921513916/ [user]$ globus-job-status https://: 51499/16073727533535086921/7782764993921513916/ UNKNOWN JOB STATE 0 [user]$ globus-job-status https://: 51499/16073727533535086921/7782764993921513916/ UNKNOWN JOB STATE 0 [user]$ globus-job-status https://: 51499/16073727533535086921/7782764993921513916/ UNKNOWN JOB STATE 0 [user]$ globus-job-status https://: 51499/16073727533535086921/7782764993921513916/ DONE Is that a bug? some GRAM5|PBS misconfiguration? Commands like globus-job-run work fine. Environment: gt5.0.0-all-source-installer.tar.bz2 torque-2.3.7-1cri torque-docs-2.3.7-1cri torque-server-2.3.7-1cri torque-client-2.3.7-1cri torque-scheduler-2.3.7-1cri Thanks in advance, Nikolay That's a bug. In GRAM2, the job manager exited after the job terminated, so globus-job-status would interpret a connection-refused response as DONE. In GRAM5, the job manager may still be running but no longer know about a particular job. You can probably treat this as "job is done" until this issue is resolved in the job manager. Joe
Re: [gt-user] globus-job-status: UNKNOWN JOB STATE 0
>>> Strange. Yea - looks like a bug. Is that repeatable / happens for every >>> PBS job? >> I haven't tested hard yet but at least I saw such status twice (for >> different jobs). >> >>> Are you using the SEG for PBS job monitoring? >> not yet. It wasn't clear for me how to configure SEG. It's written: >> It must be explicitly enabled by adding the -seg-module LRM option to >> the job manager configuration. >> But in what configuration file does it need to specify "-seg-module" >> option? in $GLOBUS_LOCATION/etc/globus-pbs.conf? > > Doc is here: > http://www.globus.org/toolkit/docs/5.0/5.0.0/execution/gram5/admin/#gram5-Interface_Config_Frag-seg_module Sorry, I wasn't reading careful enough. It's clear now how to configure SEG from URL above. So now I have $ ll /usr/local/globus-5.0.0/etc/grid-services/jobmanager lrwxrwxrwx 1 globus globus 56 Jan 22 14:58 /usr/local/globus-5.0.0/etc/grid-services/jobmanager -> /usr/local/globus-5.0.0/etc/grid-services/jobmanager-pbs $ cat /usr/local/globus-5.0.0/etc/grid-services/jobmanager-pbs stderr_log,local_cred - /usr/local/globus-5.0.0/libexec/globus-job-manager globus-job-manager -conf /usr/local/globus-5.0.0/etc/globus-job-manager.conf -type pbs -seg-module pbs $ cat /usr/local/globus-5.0.0/etc/globus-pbs.conf log_path=/var/spool/torque/server_logs $ cat /usr/local/globus-5.0.0/etc/globus-job-manager.conf -home "/usr/local/globus-5.0.0" -globus-gatekeeper-host vps115.jinr.ru -globus-gatekeeper-port 2119 -globus-gatekeeper-subject "/C=RU/O=JINR/OU=LIT/CN=vps115.jinr.ru" -globus-host-cputype i686 -globus-host-manufacturer pc -globus-host-osname Linux -globus-host-osversion 2.6.9-023stab046.2 -globus-toolkit-version 5.0.0 -stdio-log "$(HOME)" -log-levels 'FATAL|ERROR' -state-file-dir /usr/local/globus-5.0.0/tmp/gram_job_state $ globus-job-manager-event-generator -scheduler pbs -background -pidfile $GLOBUS_LOCATION/var/globus-job-manager-seg-pbs.pid Error: pbs not configured I've tried also to reload/restart xinetd since globus-gatekeeper is run by xinetd. More hints? GRAM5 was compiled as $ ./configure --prefix=$GLOBUS_LOCATION $ make But $GLOBUS_LOCATION/etc/globus-pbs.conf could be found that's why I run $ make gram5-pbs install. More hints? Nikolay. > > No - it goes in $GLOBUS_LOCATION/etc/grid-services/jobmanager-pbs > > Mine looks like this: > % cat jobmanager-pbs > stderr_log,local_cred - > /home/smartin/gt/5.0.0/INSTALL/libexec/globus-job-manager globus-job-manager > -conf /home/smartin/gt/5.0.0/INSTALL/etc/globus-job-manager.conf -type pbs > -seg-module pbs > >> How? Just add >> '-seg-module' to new line? e.g. >> log_path=/var/spool/torque/server_logs >> -seg-module >> $ cat /usr/local/globus-5.0.0/etc/globus-pbs.conf >> log_path=/var/spool/torque/server_logs >> -seg-module >> >> [root]$ globus-job-manager-event-generator -scheduler pbs -background >> -pidfile $GLOBUS_LOCATION/var/globus-job-manager-seg-pbs.pid >> Error: pbs not configured > > Maybe that is a result of you adding the extra -seg-module line to the > globus-pbs.conf file ?? > >> So what exact steps needs to be performed to run SEG? >> >> Nikolay. >> >>> If not, try using the SEG and see what happens. >>> >>> http://www.globus.org/toolkit/docs/5.0/5.0.0/execution/gram5/admin/#gram5-Interface_Config_Frag-seg_module >>> >>> http://www.globus.org/toolkit/docs/5.0/5.0.0/execution/gram5/admin/#id2545819 >>> >>> -Stu >>> >>> On Jan 22, 2010, at Jan 22, 7:43 AM, Nikolay Kutovskiy wrote: >>> Hello list, I have installed GRAM5 to use PBS and get the following output of globus-job-status command: $ globus-job-submit :2119/jobmanager-pbs /bin/hostname https://:51499/16073727533535086921/7782764993921513916/ [user]$ globus-job-status https://:51499/16073727533535086921/7782764993921513916/ UNKNOWN JOB STATE 0 [user]$ globus-job-status https://:51499/16073727533535086921/7782764993921513916/ UNKNOWN JOB STATE 0 [user]$ globus-job-status https://:51499/16073727533535086921/7782764993921513916/ UNKNOWN JOB STATE 0 [user]$ globus-job-status https://:51499/16073727533535086921/7782764993921513916/ DONE Is that a bug? some GRAM5|PBS misconfiguration? Commands like globus-job-run work fine. Environment: gt5.0.0-all-source-installer.tar.bz2 torque-2.3.7-1cri torque-docs-2.3.7-1cri torque-server-2.3.7-1cri torque-client-2.3.7-1cri torque-scheduler-2.3.7-1cri Thanks in advance, Nikolay >
Re: [gt-user] globus-job-status: UNKNOWN JOB STATE 0
On Jan 22, 2010, at Jan 22, 12:00 PM, Nikolay Kutovskiy wrote: > Hi Stuart > > Stuart Martin wrote on 22/01/10 19:04: >> Hi Nikolay, >> >> Strange. Yea - looks like a bug. Is that repeatable / happens for every >> PBS job? > I haven't tested hard yet but at least I saw such status twice (for > different jobs). > >> >> Are you using the SEG for PBS job monitoring? > not yet. It wasn't clear for me how to configure SEG. It's written: > It must be explicitly enabled by adding the -seg-module LRM option to > the job manager configuration. > But in what configuration file does it need to specify "-seg-module" > option? in $GLOBUS_LOCATION/etc/globus-pbs.conf? Doc is here: http://www.globus.org/toolkit/docs/5.0/5.0.0/execution/gram5/admin/#gram5-Interface_Config_Frag-seg_module No - it goes in $GLOBUS_LOCATION/etc/grid-services/jobmanager-pbs Mine looks like this: % cat jobmanager-pbs stderr_log,local_cred - /home/smartin/gt/5.0.0/INSTALL/libexec/globus-job-manager globus-job-manager -conf /home/smartin/gt/5.0.0/INSTALL/etc/globus-job-manager.conf -type pbs -seg-module pbs > How? Just add > '-seg-module' to new line? e.g. > log_path=/var/spool/torque/server_logs > -seg-module > $ cat /usr/local/globus-5.0.0/etc/globus-pbs.conf > log_path=/var/spool/torque/server_logs > -seg-module > > [root]$ globus-job-manager-event-generator -scheduler pbs -background > -pidfile $GLOBUS_LOCATION/var/globus-job-manager-seg-pbs.pid > Error: pbs not configured Maybe that is a result of you adding the extra -seg-module line to the globus-pbs.conf file ?? > > So what exact steps needs to be performed to run SEG? > > Nikolay. > >> If not, try using the SEG and see what happens. >> >> http://www.globus.org/toolkit/docs/5.0/5.0.0/execution/gram5/admin/#gram5-Interface_Config_Frag-seg_module >> >> http://www.globus.org/toolkit/docs/5.0/5.0.0/execution/gram5/admin/#id2545819 >> >> -Stu >> >> On Jan 22, 2010, at Jan 22, 7:43 AM, Nikolay Kutovskiy wrote: >> >>> Hello list, >>> >>> I have installed GRAM5 to use PBS and get the following output of >>> globus-job-status command: >>> $ globus-job-submit :2119/jobmanager-pbs /bin/hostname >>> https://:51499/16073727533535086921/7782764993921513916/ >>> >>> [user]$ globus-job-status >>> https://:51499/16073727533535086921/7782764993921513916/ >>> UNKNOWN JOB STATE 0 >>> >>> [user]$ globus-job-status >>> https://:51499/16073727533535086921/7782764993921513916/ >>> UNKNOWN JOB STATE 0 >>> >>> [user]$ globus-job-status >>> https://:51499/16073727533535086921/7782764993921513916/ >>> UNKNOWN JOB STATE 0 >>> [user]$ globus-job-status >>> https://:51499/16073727533535086921/7782764993921513916/ >>> DONE >>> >>> Is that a bug? some GRAM5|PBS misconfiguration? >>> >>> Commands like globus-job-run work fine. >>> Environment: >>> gt5.0.0-all-source-installer.tar.bz2 >>> torque-2.3.7-1cri >>> torque-docs-2.3.7-1cri >>> torque-server-2.3.7-1cri >>> torque-client-2.3.7-1cri >>> torque-scheduler-2.3.7-1cri >>> >>> Thanks in advance, >>> Nikolay >>
Re: [gt-user] globus-job-status: UNKNOWN JOB STATE 0
Hi Stuart Stuart Martin wrote on 22/01/10 19:04: > Hi Nikolay, > > Strange. Yea - looks like a bug. Is that repeatable / happens for every PBS > job? I haven't tested hard yet but at least I saw such status twice (for different jobs). > > Are you using the SEG for PBS job monitoring? not yet. It wasn't clear for me how to configure SEG. It's written: It must be explicitly enabled by adding the -seg-module LRM option to the job manager configuration. But in what configuration file does it need to specify "-seg-module" option? in $GLOBUS_LOCATION/etc/globus-pbs.conf? How? Just add '-seg-module' to new line? e.g. log_path=/var/spool/torque/server_logs -seg-module $ cat /usr/local/globus-5.0.0/etc/globus-pbs.conf log_path=/var/spool/torque/server_logs -seg-module [root]$ globus-job-manager-event-generator -scheduler pbs -background -pidfile $GLOBUS_LOCATION/var/globus-job-manager-seg-pbs.pid Error: pbs not configured So what exact steps needs to be performed to run SEG? Nikolay. > If not, try using the SEG and see what happens. > > http://www.globus.org/toolkit/docs/5.0/5.0.0/execution/gram5/admin/#gram5-Interface_Config_Frag-seg_module > > http://www.globus.org/toolkit/docs/5.0/5.0.0/execution/gram5/admin/#id2545819 > > -Stu > > On Jan 22, 2010, at Jan 22, 7:43 AM, Nikolay Kutovskiy wrote: > >> Hello list, >> >> I have installed GRAM5 to use PBS and get the following output of >> globus-job-status command: >> $ globus-job-submit :2119/jobmanager-pbs /bin/hostname >> https://:51499/16073727533535086921/7782764993921513916/ >> >> [user]$ globus-job-status >> https://:51499/16073727533535086921/7782764993921513916/ >> UNKNOWN JOB STATE 0 >> >> [user]$ globus-job-status >> https://:51499/16073727533535086921/7782764993921513916/ >> UNKNOWN JOB STATE 0 >> >> [user]$ globus-job-status >> https://:51499/16073727533535086921/7782764993921513916/ >> UNKNOWN JOB STATE 0 >> [user]$ globus-job-status >> https://:51499/16073727533535086921/7782764993921513916/ >> DONE >> >> Is that a bug? some GRAM5|PBS misconfiguration? >> >> Commands like globus-job-run work fine. >> Environment: >> gt5.0.0-all-source-installer.tar.bz2 >> torque-2.3.7-1cri >> torque-docs-2.3.7-1cri >> torque-server-2.3.7-1cri >> torque-client-2.3.7-1cri >> torque-scheduler-2.3.7-1cri >> >> Thanks in advance, >> Nikolay >
Re: [gt-user] globus-job-status: UNKNOWN JOB STATE 0
Hi Nikolay, Strange. Yea - looks like a bug. Is that repeatable / happens for every PBS job? Are you using the SEG for PBS job monitoring? If not, try using the SEG and see what happens. http://www.globus.org/toolkit/docs/5.0/5.0.0/execution/gram5/admin/#gram5-Interface_Config_Frag-seg_module http://www.globus.org/toolkit/docs/5.0/5.0.0/execution/gram5/admin/#id2545819 -Stu On Jan 22, 2010, at Jan 22, 7:43 AM, Nikolay Kutovskiy wrote: > Hello list, > > I have installed GRAM5 to use PBS and get the following output of > globus-job-status command: > $ globus-job-submit :2119/jobmanager-pbs /bin/hostname > https://:51499/16073727533535086921/7782764993921513916/ > > [user]$ globus-job-status > https://:51499/16073727533535086921/7782764993921513916/ > UNKNOWN JOB STATE 0 > > [user]$ globus-job-status > https://:51499/16073727533535086921/7782764993921513916/ > UNKNOWN JOB STATE 0 > > [user]$ globus-job-status > https://:51499/16073727533535086921/7782764993921513916/ > UNKNOWN JOB STATE 0 > [user]$ globus-job-status > https://:51499/16073727533535086921/7782764993921513916/ > DONE > > Is that a bug? some GRAM5|PBS misconfiguration? > > Commands like globus-job-run work fine. > Environment: > gt5.0.0-all-source-installer.tar.bz2 > torque-2.3.7-1cri > torque-docs-2.3.7-1cri > torque-server-2.3.7-1cri > torque-client-2.3.7-1cri > torque-scheduler-2.3.7-1cri > > Thanks in advance, > Nikolay
[gt-user] globus-job-status: UNKNOWN JOB STATE 0
Hello list, I have installed GRAM5 to use PBS and get the following output of globus-job-status command: $ globus-job-submit :2119/jobmanager-pbs /bin/hostname https://:51499/16073727533535086921/7782764993921513916/ [user]$ globus-job-status https://:51499/16073727533535086921/7782764993921513916/ UNKNOWN JOB STATE 0 [user]$ globus-job-status https://:51499/16073727533535086921/7782764993921513916/ UNKNOWN JOB STATE 0 [user]$ globus-job-status https://:51499/16073727533535086921/7782764993921513916/ UNKNOWN JOB STATE 0 [user]$ globus-job-status https://:51499/16073727533535086921/7782764993921513916/ DONE Is that a bug? some GRAM5|PBS misconfiguration? Commands like globus-job-run work fine. Environment: gt5.0.0-all-source-installer.tar.bz2 torque-2.3.7-1cri torque-docs-2.3.7-1cri torque-server-2.3.7-1cri torque-client-2.3.7-1cri torque-scheduler-2.3.7-1cri Thanks in advance, Nikolay
