Yes, SIGUSR1 is globaly blocked; inside of a GE starter method is added a line "signal (SIGUSR1, SIG_IGN);" I consider as a weirdness that in the master node execd messages is a recorded exceeded hard walclock time:
"11/29/2011 14:06:33| main|n15|W|job 112781.1 exceeded* hard *wallclock time - initiate terminate method" what is in the contradiction with the record in the messages of the slave: "11/29/2011 14:06:27| main|n1|W|job 112781.1 exceeded *soft *wallclock time - initiate soft notify method" I remark the request s_rt=<time> was configured for qsub command baf035 Dne 29. listopadu 2011 18:26 Reuti <[email protected]> napsal(a): Am 29.11.2011 um 17:37 schrieb baf035: > > > Hello Reuti, > > > > thanks for a clarification, I probably misunderstand man pages and used > both parameters: s_rt and -notify together before. > > > > Our goal is to configure exactly working backfilling. We use for s_rt a > mean value of running time of finished jobs based on their category. > > In a queue configuration is a big value for notify time. > > > > When s_rt is expired, job is signaled but not killed thanks to notify > time. We receive an information in execd messages file: > > "11/29/2011 11:11:28| main|service65|W|job 112756.1 exceeded soft > wallclock time - initiate soft notify method" > > The job is correctly finished. > > This is valid for the single core job or for the parallel job running > only inside 1 node. > > I case of bigger parallel jobs they are killed directly after s_rt > expiration. It seems that GE considers s_rt as h_rt ?! > > execd messages (master node): > > "11/29/2011 14:06:33| main|n15|W|job 112781.1 exceeded hard wallclock > time - initiate terminate method" > > Do you trap the warning signal in all ranks? If you have only one node, > there will be threads used. But on the slave nodes, a new process is > created, which must ignore the signal. > > -- Reuti > > > > qmaster messages > > "11/29/2011 14:06:34|worker|sged3|E|master task of job 112781.1 failed - > killing job > > 11/29/2011 14:06:35|worker|ged3|W|job 112781.1 failed on host n15 > assumedly after job because: job 112781.1 died through signal KILL (9)" > > > > execd strace: > > "17649 open("/<path>/sge_spool//n15/messages", > O_WRONLY|O_CREAT|O_APPEND, 0666) = 5 > > 17649 write(5, "11/29/2011 14:06:33| main|r8i1n"..., 107) = 107 > > 17649 close(5) = 0 > > 17649 kill(2152, SIGTSTP) = 0 > > 17649 futex(0x7f705cd06e04, 0x189 /* FUTEX_??? */, 4655823, {1322571994, > 846698000}, ffffffff <unfinished ...> > > 17650 <... futex resumed> ) = -1 ETIMEDOUT (Connection timed > out) > > 17650 futex(0x7f705cd068f0, FUTEX_WAKE_PRIVATE, 1) = 0 > > 17650 futex(0x7f705cd06954, 0x189 /* FUTEX_??? */, 4332213, {1322571994, > 891528000}, ffffffff <unfinished ...> > > 17653 <... futex resumed> ) = -1 ETIMEDOUT (Connection timed > out) > > 17653 futex(0x7f705cd07100, FUTEX_WAKE_PRIVATE, 1) = 0 > > 17653 futex(0x7f705cd07164, 0x189 /* FUTEX_??? */, 4440135, {1322571995, > 703424000}, ffffffff <unfinished ...> > > 17649 <... futex resumed> ) = ? ERESTART_RESTARTBLOCK (To be > restarted) > > 17649 --- SIGCHLD (Child exited) @ 0 (0) --- > > 17649 rt_sigreturn(0x11) = -1 EINTR (Interrupted system > call)" > > > > Do you have an idea what could be wrong? > > > > Thanks > > Best regards > > > > baf035 > > > > > > > > > > Dne 22. listopadu 2011 16:16 Reuti <[email protected]> > napsal(a): > > Hi, > > > > Am 22.11.2011 um 12:58 schrieb baf035: > > > > > > > > Hello all. > > > > > > 1) When s_rt time is reached a job is signaled by default by SIGUSR1. > One of our applications is dying by the signal. > > > I changed in a global configuration in execd_params a parameter > NOTIFY_KILL=none or NOTIFY_KILL=SIGCONT > > > bur without effect. Applications are still signaled by SIGUSR1. How > could I change the behavior? > > > > these are two different things. > > > > a) when you defined s_rt, it will send SIGUSR1 at the specified time - > there is no way to avoid this. If you don't want it, don't define s_rt. > > > > b) the notification by -notify will be send the specified time before > the final kill/suspend. These notifications can be redefined by the > commands you used. If you don't want it, don't use -notify at submission > time. > > > > NB: The signal will be send to the complete processgroup, i.e. the job > script and the called binary. If you want to have an action happening on > SIGUSR1, you need a proper set "trap" for the shell and/or a proper signal > handling in the binary. The default action for SIGUSR1 is to terminate: > http://kernel.org/doc/man-pages/online/pages/man7/signal.7.html So you > just face the default behavior. > > > > > > > 2) Is the way how to reduce warning messages "job <JID>.1should have > finished since <num> s" in a qmaster messages file > > > when s_rt time is exceeded and a notify time is going on. > > > > Not that I'm aware of. But using s_rt and -notify at the same time > doesn't work in a nice way together is my experience. I suggest to use only > one of them at a time. Do you have a 2 stage warning in mind, which you > want to implement? Using both the signal chain would be: 1) s_rt (sigusr1) > warning, 2) -notify defined warning before kill, 3) kill > > > > -- Reuti > > > > > > > Our productive GE is SoGE 8.0.0a. > > > > > > Thanks for help. Best regards > > > > > > baf035 > > > > > > _______________________________________________ > > > users mailing list > > > [email protected] > > > https://gridengine.org/mailman/listinfo/users > > > > > >
_______________________________________________ users mailing list [email protected] https://gridengine.org/mailman/listinfo/users
