Just got the following error returned to my qrsh command line: error: error while waiting for builtin IJS connection: "got select timeout"
Have only seen this once, it may be unrelated... On Wed, Feb 22, 2012 at 10:12 AM, Michael Coffman <[email protected]> wrote: > I have system that will not accept qrsh jobs. qstat shows the job in > the r state: > > 174955 0.50934 QRLOGIN coffman r 02/22/2012 09:59:21 all.q@cs424 > > But the login never goes through. > > a simple qsub will work fine. > > From the qmaster spool/messages file: > > 02/22/2012 10:00:21|worker|geshadow|W|job 174955.1 failed on host > cs424 assumedly after job because: job 174955.1 died through signal > KILL (9) > > I was seeing the following in the spool/messages file on the exec host: > 02/22/2012 09:39:47| main|cs424|E|shepherd of job 174535.1 exited > with exit status = 25 > > I restarted execd using a softstop and start and now nothing is being logged. > > There are 2 active jobs on the exec host, so I can't reboot it. > There are some comments about the 25 status error in sge_conf. > > max_advance_reservations - If the max_advance_reservations limit is > exceeded by an Advance Reservation request then the submission > command exits with exit status 25 and an appropriate error message > > max_u_jobs - If the max_u_jobs limit is exceeded by a job > submission then the submission command exits with exit status 25 and > an appropriate error message. > > max_jobs - If the max_jobs limit is exceeded by a job submission > then the submission command exits with exit status 25 and an > appropriate error message. > > None of these are the issue. Any thoughts on how to debug this? > > I also tried running strace against the sgeexecd and did not see > anything interesting. A grep of the jid produces the following: > > mkdir("active_jobs/174996.1", 0755) = 0 > mkdir("/tmp/174996.1.all.q", 0755) = 0 > chown("/tmp/174996.1.all.q", 26927, 20) = 0 > open("/opt/grid-6.2u5/ftcrnd/spool/cs424/active_jobs/174996.1/pe_hostfile", > O_WRONLY|O_CREAT|O_TRUNC, 0666) = 4 > open("/opt/grid-6.2u5/ftcrnd/spool/cs424/active_jobs/174996.1/environment", > O_WRONLY|O_CREAT|O_TRUNC, 0666) = 4 > open("active_jobs/174996.1/config", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 4 > chdir("active_jobs/174996.1") = 0 > stat("active_jobs/174996.1/addgrpid", {st_mode=S_IFREG|0644, > st_size=6, ...}) = 0 > open("active_jobs/174996.1/addgrpid", O_RDONLY) = 4 > stat("active_jobs/174996.1", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 > open("active_jobs/174996.1/pid", O_RDONLY) = 6 > stat("active_jobs/174996.1", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 > open("active_jobs/174996.1/config", O_RDONLY) = 4 > open("active_jobs/174996.1/exit_status", O_RDONLY) = 4 > open("active_jobs/174996.1/error", O_RDONLY) = 4 > open("active_jobs/174996.1/pid", O_RDONLY) = 4 > open("active_jobs/174996.1/usage", O_RDONLY) = 4 > stat("active_jobs/174996.1/checkpointed", 0x7ffff75e7280) = -1 ENOENT > (No such file or directory) > stat("active_jobs/174996.1/noresources", 0x7ffff75e7280) = -1 ENOENT > (No such file or directory) > open("active_jobs/174996.1", O_RDONLY|O_NONBLOCK|O_DIRECTORY) = 4 > lstat("active_jobs/174996.1/error", {st_mode=S_IFREG|0644, st_size=0, ...}) = > 0 > unlink("active_jobs/174996.1/error") = 0 > lstat("active_jobs/174996.1/config", {st_mode=S_IFREG|0644, > st_size=2167, ...}) = 0 > unlink("active_jobs/174996.1/config") = 0 > lstat("active_jobs/174996.1/usage", {st_mode=S_IFREG|0644, > st_size=309, ...}) = 0 > unlink("active_jobs/174996.1/usage") = 0 > lstat("active_jobs/174996.1/environment", {st_mode=S_IFREG|0644, > st_size=1364, ...}) = 0 > unlink("active_jobs/174996.1/environment") = 0 > lstat("active_jobs/174996.1/pe_hostfile", {st_mode=S_IFREG|0644, > st_size=66, ...}) = 0 > unlink("active_jobs/174996.1/pe_hostfile") = 0 > lstat("active_jobs/174996.1/addgrpid", {st_mode=S_IFREG|0644, > st_size=6, ...}) = 0 > unlink("active_jobs/174996.1/addgrpid") = 0 > lstat("active_jobs/174996.1/trace", {st_mode=S_IFREG|0644, > st_size=5979, ...}) = 0 > unlink("active_jobs/174996.1/trace") = 0 > lstat("active_jobs/174996.1/job_pid", {st_mode=S_IFREG|0644, > st_size=6, ...}) = 0 > unlink("active_jobs/174996.1/job_pid") = 0 > lstat("active_jobs/174996.1/pid", {st_mode=S_IFREG|0644, st_size=6, ...}) = 0 > unlink("active_jobs/174996.1/pid") = 0 > lstat("active_jobs/174996.1/shepherd_about_to_exit", > {st_mode=S_IFREG|0644, st_size=0, ...}) = 0 > unlink("active_jobs/174996.1/shepherd_about_to_exit") = 0 > lstat("active_jobs/174996.1/exit_status", {st_mode=S_IFREG|0644, > st_size=2, ...}) = 0 > unlink("active_jobs/174996.1/exit_status") = 0 > rmdir("active_jobs/174996.1") = 0 > open("/tmp/174996.1.all.q", O_RDONLY|O_NONBLOCK|O_DIRECTORY) = 4 > rmdir("/tmp/174996.1.all.q") = 0 > > Thanks. > -- > -MichaelC -- -MichaelC _______________________________________________ users mailing list [email protected] https://gridengine.org/mailman/listinfo/users
