Hi, Am 09.01.2019 um 23:35 schrieb Derrick Lin:
> Hi Reuti, > > I have to say I am still not familiar with the "-i" in qsub after reading the > man page, what does it do? It will be feed as stdin to the jobscript. Hence: $ qsub -i myfile foo.sh is like: $ foo.sh < myfile but in batch. -- Reuti > There is no useful/interesting output in qmaster message or exec node message > log. The only information I could find is from job's trace file: > > [root@zeta-4-12 381.1]# ls > config environment error exit_status pe_hostfile pid trace > [root@zeta-4-12 381.1]# cat trace > 01/10/2019 09:12:07 [997:307578]: shepherd called with uid = 0, euid = 997 > 01/10/2019 09:12:07 [997:307578]: qlogin_daemon = builtin > 01/10/2019 09:12:07 [997:307578]: starting up 8.1.9 > 01/10/2019 09:12:07 [997:307578]: setpgid(307578, 307578) returned 0 > 01/10/2019 09:12:07 [997:307578]: do_core_binding: "binding" parameter not > found in config file > 01/10/2019 09:12:07 [997:307578]: calling fork_pty() > 01/10/2019 09:12:07 [997:307578]: parent: forked "prolog" with pid 307579 > 01/10/2019 09:12:07 [997:307578]: using signal delivery delay of 120 seconds > 01/10/2019 09:12:07 [997:307578]: parent: prolog-pid: 307579 > 01/10/2019 09:12:07 [997:307579]: child: starting son(prolog, > root@/opt/gridengine/default/common/prolog_exec.sh, 0, 10000); > 01/10/2019 09:12:07 [997:307579]: pid=307579 pgrp=307579 sid=307579 old > pgrp=307579 getlogin()=<no login set> > 01/10/2019 09:12:07 [997:307579]: reading passwd information for user 'root' > 01/10/2019 09:12:07 [997:307579]: setting limits > 01/10/2019 09:12:07 [997:307579]: setting environment > 01/10/2019 09:12:07 [997:307579]: Initializing error file > 01/10/2019 09:12:07 [997:307579]: switching to intermediate/target user > 01/10/2019 09:12:07 [997:307579]: setting additional gid=0 > 01/10/2019 09:12:07 [6782:307579]: closing all filedescriptors > 01/10/2019 09:12:07 [6782:307579]: further messages are in "error" and "trace" > 01/10/2019 09:12:07 [997:307578]: Poll received POLLHUP (Hang up). Unregister > the FD. > 01/10/2019 09:12:07 [6782:307579]: using "/bin/bash" as shell of user "root" > 01/10/2019 09:12:07 [0:307579]: now running with uid=0, euid=0 > 01/10/2019 09:12:07 [0:307579]: > execvlp(/opt/gridengine/default/common/prolog_exec.sh, > "/opt/gridengine/default/common/prolog_exec.sh") > ### The process just stuck in the line above > > Here is the trace file for a qsub/batch job, apparently the prolog script got > executed and the process proceeded: > > [root@zeta-4-12 383.1]# ls > addgrpid config environment error exit_status job_pid pe_hostfile pid > trace > [root@zeta-4-12 383.1]# cat trace > 01/10/2019 09:20:22 [997:315329]: shepherd called with uid = 0, euid = 997 > 01/10/2019 09:20:22 [997:315329]: starting up 8.1.9 > 01/10/2019 09:20:22 [997:315329]: setpgid(315329, 315329) returned 0 > 01/10/2019 09:20:22 [997:315329]: do_core_binding: "binding" parameter not > found in config file > 01/10/2019 09:20:22 [997:315329]: parent: forked "prolog" with pid 315330 > 01/10/2019 09:20:22 [997:315329]: using signal delivery delay of 120 seconds > 01/10/2019 09:20:22 [997:315329]: parent: prolog-pid: 315330 > 01/10/2019 09:20:22 [997:315330]: child: starting son(prolog, > root@/opt/gridengine/default/common/prolog_exec.sh, 0, 10000); > 01/10/2019 09:20:22 [997:315330]: pid=315330 pgrp=315330 sid=315330 old > pgrp=315329 getlogin()=<no login set> > 01/10/2019 09:20:22 [997:315330]: reading passwd information for user 'root' > 01/10/2019 09:20:22 [997:315330]: setting limits > 01/10/2019 09:20:22 [997:315330]: setting environment > 01/10/2019 09:20:22 [997:315330]: Initializing error file > 01/10/2019 09:20:22 [997:315330]: switching to intermediate/target user > 01/10/2019 09:20:22 [997:315330]: setting additional gid=0 > 01/10/2019 09:20:22 [6782:315330]: closing all filedescriptors > 01/10/2019 09:20:22 [6782:315330]: further messages are in "error" and "trace" > 01/10/2019 09:20:22 [6782:315330]: using "/bin/bash" as shell of user "root" > 01/10/2019 09:20:22 [6782:315330]: using stdout as stderr > 01/10/2019 09:20:22 [0:315330]: now running with uid=0, euid=0 > 01/10/2019 09:20:22 [0:315330]: > execvlp(/opt/gridengine/default/common/prolog_exec.sh, > "/opt/gridengine/default/common/prolog_exec.sh") > 01/10/2019 09:20:22 [997:315329]: wait3 returned 315330 (status: 0; > WIFSIGNALED: 0, WIFEXITED: 1, WEXITSTATUS: 0) > 01/10/2019 09:20:22 [997:315329]: prolog exited with exit status 0 > 01/10/2019 09:20:22 [997:315329]: reaped "prolog" with pid 315330 > 01/10/2019 09:20:22 [997:315329]: prolog exited not due to signal > 01/10/2019 09:20:22 [997:315329]: prolog exited with status 0 > 01/10/2019 09:20:22 [997:315329]: parent: forked "job" with pid 315345 > 01/10/2019 09:20:22 [997:315329]: parent: job-pid: 315345 > 01/10/2019 09:20:22 [997:315345]: child: starting son(job, sleep, 0, 4096); > 01/10/2019 09:20:22 [997:315345]: pid=315345 pgrp=315345 sid=315345 old > pgrp=315329 getlogin()=<no login set> > 01/10/2019 09:20:22 [997:315345]: reading passwd information for user 'derlin' > 01/10/2019 09:20:22 [997:315345]: setosjobid: uid = 0, euid = 997 > 01/10/2019 09:20:22 [997:315345]: setting limits > 01/10/2019 09:20:22 [997:315345]: RLIMIT_CPU setting: (soft INFINITY hard > INFINITY) resulting: (soft INFINITY hard INFINITY) > 01/10/2019 09:20:22 [997:315345]: RLIMIT_FSIZE setting: (soft INFINITY hard > INFINITY) resulting: (soft INFINITY hard INFINITY) > 01/10/2019 09:20:22 [997:315345]: RLIMIT_DATA setting: (soft INFINITY hard > INFINITY) resulting: (soft INFINITY hard INFINITY) > 01/10/2019 09:20:22 [997:315345]: RLIMIT_STACK setting: (soft INFINITY hard > INFINITY) resulting: (soft INFINITY hard INFINITY) > 01/10/2019 09:20:22 [997:315345]: RLIMIT_CORE setting: (soft INFINITY hard > INFINITY) resulting: (soft INFINITY hard INFINITY) > 01/10/2019 09:20:22 [997:315345]: RLIMIT_VMEM/RLIMIT_AS setting: (soft > 8257536000 hard 8257536000) resulting: (soft 8257536000 hard 8257536000) > 01/10/2019 09:20:22 [997:315345]: RLIMIT_RSS setting: (soft INFINITY hard > INFINITY) resulting: (soft INFINITY hard INFINITY) > 01/10/2019 09:20:22 [997:315345]: setting environment > 01/10/2019 09:20:22 [997:315345]: Initializing error file > 01/10/2019 09:20:22 [997:315345]: switching to intermediate/target user > 01/10/2019 09:20:22 [997:315345]: setting additional gid=20011 > 01/10/2019 09:20:22 [6782:315345]: closing all filedescriptors > 01/10/2019 09:20:22 [6782:315345]: further messages are in "error" and "trace" > 01/10/2019 09:20:22 [6782:315345]: using stdout as stderr > 01/10/2019 09:20:22 [6782:315345]: now running with uid=6782, euid=6782 > 01/10/2019 09:20:22 [6782:315345]: execvlp(/bin/csh, "-csh" "-c" "sleep 10m ") > > I will attach my prolog script in the next post. > > Cheers > Derrick > > > On Wed, Jan 9, 2019 at 7:36 PM Reuti <re...@staff.uni-marburg.de> wrote: > Hi, > > > Am 09.01.2019 um 01:14 schrieb Derrick Lin <klin...@gmail.com>: > > > > Hi guys, > > > > I just brought up a new SGE cluster, but somehow the qrsh session does not > > work: > > > > tester@login-gpu:~$ qrsh > > ^Cerror: error while waiting for builtin IJS connection: "got select > > timeout" > > > > after I hit entered, the session just stuck there forever instead of bring > > me to a compute node. I have to entered Crtl+c to terminate and it gave the > > above error. > > > > I noticed, the SGE did send my qrsh request to a compute node as I could > > tell from qstat: > > > > --------------------------------------------------------------------------------- > > short.q@zeta-4-15.local BIP 0/1/80 0.01 lx-amd64 > > 15 0.55500 QRLOGIN tester r 01/09/2019 10:47:13 1 > > > > We have a prolog script configured globally, the script deals with local > > disk quota and keep all output to a log file for each job. So I went to > > that compute node, and check, found that a log file was created but it was > > empty. > > > > So my thinking so far is, my qrsh stuck because the prolog script is not > > fully executed. > > Is there any statement in the prolog, which could wait for stdin – and in a > batch job there is just no stdin, hence it continues? Could be tested with > "-i" to a batch job. > > -- Reuti > > > > qsub job are working fine. > > > > Any idea will be appreciated > > > > Cheers, > > Derrick > > _______________________________________________ > > users mailing list > > users@gridengine.org > > https://gridengine.org/mailman/listinfo/users > _______________________________________________ users mailing list users@gridengine.org https://gridengine.org/mailman/listinfo/users
