I got a problem that only when I run the job on node001(my jobs|qsub -l nodes=node001) it give me back the *.o and *.e files. if i try to run the job on node002 or does not specify in the command, i don't get any *.o or *.e files. does anyone know why? thanks.
log file from node002 looks something wrong there, since it says failur jobs exec failure, but i dont' know exactly what's wrong. and log file for node001 and master head looks fine. thanks a lot [EMAIL PROTECTED] ~]# less /mnt/PBS/node002/PBS_spool/mom_logs/20080806 08/06/2008 04:32:41;0002; pbs_mom;Svr;Log;Log opened 08/06/2008 04:32:41;0080; pbs_mom;n/a;add_static;config[1] add name arch value x86 08/06/2008 04:32:41;0080; pbs_mom;n/a;add_static;config[2] add name opsys value Linux 08/06/2008 04:32:41;0002; pbs_mom;Svr;usecp;master.perceus.centos:/home /home 08/06/2008 04:32:41;0002; pbs_mom;n/a;initialize;independent 08/06/2008 04:32:41;0002; pbs_mom;Svr;pbs_mom;Is up 08/06/2008 04:32:41;0002; pbs_mom;Svr;mom_main;MOM executable path and mtime at launch: /usr/sbin/pbs_mom 1211916151 08/06/2008 04:32:41;0002; pbs_mom;n/a;mom_main;hello sent to server master 08/06/2008 04:35:21;0001; pbs_mom;Job;TMomFinalizeJob3;job not started, Failure job exec failure, before files staged, no retry 08/06/2008 04:35:21;0008; pbs_mom;Req;send_sisters;sending ABORT to sisters 08/06/2008 04:35:21;0008; pbs_mom;Job;137.master.perceus.centos;Job Modified at request of [EMAIL PROTECTED] 08/06/2008 04:44:30;0001; pbs_mom;Job;TMomFinalizeJob3;job not started, Failure job exec failure, before files staged, no retry 08/06/2008 04:44:30;0008; pbs_mom;Req;send_sisters;sending ABORT to sisters 08/06/2008 04:44:30;0008; pbs_mom;Job;140.master.perceus.centos;Job Modified at request of [EMAIL PROTECTED] [EMAIL PROTECTED] ~]# less /mnt/PBS/node001/PBS_spool/mom_logs/20080806 08/06/2008 13:01:52;0002; pbs_mom;Svr;Log;Log opened 08/06/2008 13:01:52;0080; pbs_mom;n/a;add_static;config[1] add name arch value x86 08/06/2008 13:01:52;0080; pbs_mom;n/a;add_static;config[2] add name opsys value Linux 08/06/2008 13:01:52;0002; pbs_mom;Svr;setloglevel;4 08/06/2008 13:01:52;0002; pbs_mom;Svr;usecp;master.perceus.centos:/home /home 08/06/2008 13:01:52;0002; pbs_mom;n/a;initialize;independent 08/06/2008 13:01:52;0002; pbs_mom;Svr;pbs_mom;Is up 08/06/2008 13:01:52;0002; pbs_mom;Svr;mom_main;MOM executable path and mtime at launch: /usr/sbin/pbs_mom 1211916151 08/06/2008 13:01:52;0002; pbs_mom;n/a;init_server_stream;init_server_stream: added connection to master port 15001 08/06/2008 13:01:52;0002; pbs_mom;n/a;mom_main;hello sent to server master 08/06/2008 13:03:22;0002; pbs_mom;n/a;mom_main;connection to server master timeout 08/06/2008 13:03:22;0002; pbs_mom;n/a;init_server_stream;init_server_stream: added connection to master port 15001 08/06/2008 13:03:22;0002; pbs_mom;n/a;mom_main;hello sent to server master 08/06/2008 13:03:56;0008; pbs_mom;Job;do_rpp;got an inter-server request 08/06/2008 13:03:56;0001; pbs_mom;Job;is_request;stream 0 version 1 08/06/2008 13:03:56;0001; pbs_mom;Job;is_request;command 2 received 08/06/2008 13:03:56;0001; pbs_mom;Job;is_request;IS_CLUSTER_ADDRS received 08/06/2008 13:03:56;0001; pbs_mom;Job;is_request;is_request: 10.15.0.1 added to okclients 08/06/2008 13:03:56;0001; pbs_mom;Job;is_request;is_request: 10.15.0.21 added to okclients 08/06/2008 13:03:56;0001; pbs_mom;Job;is_request;is_request: 10.15.0.173 added to okclients 08/06/2008 13:03:57;0002; pbs_mom;n/a;is_update_stat;status update successfully sent to master 08/06/2008 13:04:42;0002; pbs_mom;n/a;is_update_stat;status update successfully sent to master 08/06/2008 13:05:27;0002; pbs_mom;n/a;is_update_stat;status update successfully sent to master 08/06/2008 13:06:12;0002; pbs_mom;n/a;is_update_stat;status update successfully sent to master 08/06/2008 13:06:57;0002; pbs_mom;n/a;is_update_stat;status update successfully sent to master 08/06/2008 13:07:42;0002; pbs_mom;n/a;is_update_stat;status update successfully sent to master 08/06/2008 13:08:27;0002; pbs_mom;n/a;is_update_stat;status update successfully sent to master 08/06/2008 13:08:52;0008; pbs_mom;Job;process_request;request type QueueJob from host master.perceus.centos allowed 08/06/2008 13:08:52;0008; pbs_mom;Job;process_request;request type JobScript from host master.perceus.centos allowed 08/06/2008 13:08:52;0008; pbs_mom;Job;process_request;request type ReadyToCommit from host master.perceus.centos allowed 08/06/2008 13:08:52;0008; pbs_mom;Job;process_request;request type Commit from host master.perceus.centos allowed 08/06/2008 13:08:52;0001; pbs_mom;Job;job_nodes;0: node001/0 08/06/2008 13:08:52;0001; pbs_mom;Job;job_nodes;job: 138.master.perceus.centos numnodes=1 numvnod=1 08/06/2008 13:08:52;0008; pbs_mom;Job;138.master.perceus.centos;evaluating limits for job 08/06/2008 13:08:52;0001; pbs_mom;Job;138.master.perceus.centos;phase 2 of job launch successfully completed 08/06/2008 13:08:52;0001; pbs_mom;Job;TMomFinalizeJob3;read start return code=0 session=15182 08/06/2008 13:08:52;0001; pbs_mom;Job;TMomFinalizeJob3;job 138.master.perceus.centos started, pid = 15182 08/06/2008 13:08:52;0001; pbs_mom;Job;138.master.perceus.centos;job successfully started 08/06/2008 13:08:52;0008; pbs_mom;Job;138.master.perceus.centos;job 138.master.perceus.centos reported successful start on 1 node(s) 08/06/2008 13:08:52;0008; pbs_mom;Job;process_request;request type StatusJob from host master.perceus.centos allowed 08/06/2008 13:08:52;0080; pbs_mom;Job;138.master.perceus.centos;sending "send flagged" attr: session_id 08/06/2008 13:08:52;0008; pbs_mom;Job;process_request;request type ModifyJob from host master.perceus.centos allowed 08/06/2008 13:08:52;0002; pbs_mom;n/a;mom_set_limits;mom_set_limits(138.master.perceus.centos,alter) entered 08/06/2008 13:08:52;0002; pbs_mom;n/a;mom_set_limits;setting limit for attribute 'neednodes' 08/06/2008 13:08:52;0002; pbs_mom;n/a;mom_set_limits;setting limit for attribute 'nodes' 08/06/2008 13:08:52;0002; pbs_mom;n/a;mom_set_limits;setting limit for attribute 'walltime' 08/06/2008 13:08:52;0008; pbs_mom;Job;138.master.perceus.centos;Job Modified at request of [EMAIL PROTECTED] 08/06/2008 13:08:52;0008; pbs_mom;Job;scan_for_terminated;for job 138.master.perceus.centos, task 1, pid=15182, exitcode=126 08/06/2008 13:08:52;0080; pbs_mom;Job;138.master.perceus.centos;scan_for_terminated: job 138.master.perceus.centos task 1 terminated, sid 15182 08/06/2008 13:08:52;0008; pbs_mom;Job;138.master.perceus.centos;job was terminated 08/06/2008 13:08:52;0008; pbs_mom;Req;send_sisters;sending command KILL_JOB for job 138.master.perceus.centos (2) 08/06/2008 13:08:52;0008; pbs_mom;Job;scan_for_terminated;pid 15197 not tracked, exitcode=0 08/06/2008 13:08:52;0008; pbs_mom;Job;process_request;request type CopyFiles from host master.perceus.centos allowed 08/06/2008 13:08:52;0008; pbs_mom;Job;138.master.perceus.centos;attempting to copy file 'master.perceus.centos:/home/yanansun/STDIN.o138' 08/06/2008 13:08:52;0008; pbs_mom;Job;138.master.perceus.centos;forking to user, uid: 556 gid: 100 homedir: '/home/yanansun' 08/06/2008 13:08:52;0008; pbs_mom;Job;process_request;request type DeleteJob from host master.perceus.centos allowed 08/06/2008 13:08:52;0080; pbs_mom;Job;138.master.perceus.centos;deleting job 138.master.perceus.centos in state EXITED 08/06/2008 13:08:52;0008; pbs_mom;Job;scan_for_terminated;pid 15198 not tracked, exitcode=0 08/06/2008 13:09:12;0002; pbs_mom;n/a;is_update_stat;status update successfully sent to master 08/06/2008 13:09:57;0002; pbs_mom;n/a;is_update_stat;status update successfully sent to master 08/06/2008 13:10:42;0002; pbs_mom;n/a;is_update_stat;status update successfully sent to master 08/06/2008 13:11:27;0002; pbs_mom;n/a;is_update_stat;status update successfully sent to master 08/06/2008 13:12:08;0008; pbs_mom;Job;process_request;request type QueueJob from host master.perceus.centos allowed 08/06/2008 13:12:08;0008; pbs_mom;Job;process_request;request type JobScript from host master.perceus.centos allowed 08/06/2008 13:12:08;0008; pbs_mom;Job;process_request;request type ReadyToCommit from host master.perceus.centos allowed 08/06/2008 13:12:08;0008; pbs_mom;Job;process_request;request type Commit from host master.perceus.centos allowed 08/06/2008 13:12:08;0001; pbs_mom;Job;job_nodes;0: node001/0 08/06/2008 13:12:08;0001; pbs_mom;Job;job_nodes;job: 139.master.perceus.centos numnodes=1 numvnod=1 08/06/2008 13:12:08;0008; pbs_mom;Job;139.master.perceus.centos;evaluating limits for job 08/06/2008 13:12:08;0001; pbs_mom;Job;139.master.perceus.centos;phase 2 of job launch successfully completed 08/06/2008 13:12:08;0001; pbs_mom;Job;TMomFinalizeJob3;read start return code=0 session=702 08/06/2008 13:12:08;0001; pbs_mom;Job;TMomFinalizeJob3;job 139.master.perceus.centos started, pid = 702 08/06/2008 13:12:08;0001; pbs_mom;Job;139.master.perceus.centos;job successfully started 08/06/2008 13:12:08;0008; pbs_mom;Job;139.master.perceus.centos;job 139.master.perceus.centos reported successful start on 1 node(s) 08/06/2008 13:12:08;0008; pbs_mom;Job;process_request;request type StatusJob from host master.perceus.centos allowed 08/06/2008 13:12:08;0080; pbs_mom;Job;139.master.perceus.centos;sending "send flagged" attr: session_id 08/06/2008 13:12:08;0008; pbs_mom;Job;process_request;request type ModifyJob from host master.perceus.centos allowed 08/06/2008 13:12:08;0002; pbs_mom;n/a;mom_set_limits;mom_set_limits(139.master.perceus.centos,alter) entered 08/06/2008 13:12:08;0002; pbs_mom;n/a;mom_set_limits;setting limit for attribute 'neednodes' 08/06/2008 13:12:08;0002; pbs_mom;n/a;mom_set_limits;setting limit for attribute 'nodes' 08/06/2008 13:12:08;0002; pbs_mom;n/a;mom_set_limits;setting limit for attribute 'walltime' 08/06/2008 13:12:08;0008; pbs_mom;Job;139.master.perceus.centos;Job Modified at request of [EMAIL PROTECTED] 08/06/2008 13:12:08;0008; pbs_mom;Job;scan_for_terminated;for job 139.master.perceus.centos, task 1, pid=702, exitcode=0 08/06/2008 13:12:08;0080; pbs_mom;Job;139.master.perceus.centos;scan_for_terminated: job 139.master.perceus.centos task 1 terminated, sid 702 08/06/2008 13:12:08;0008; pbs_mom;Job;139.master.perceus.centos;job was terminated 08/06/2008 13:12:08;0008; pbs_mom;Req;send_sisters;sending command KILL_JOB for job 139.master.perceus.centos (2) 08/06/2008 13:12:08;0080; pbs_mom;Job;139.master.perceus.centos;local task termination detected. killing job 08/06/2008 13:12:08;0008; pbs_mom;Job;139.master.perceus.centos;kill_job 08/06/2008 13:12:08;0080; pbs_mom;Job;139.master.perceus.centos;performing job clean-up 08/06/2008 13:12:08;0008; pbs_mom;Job;scan_for_terminated;pid 717 not tracked, exitcode=0 08/06/2008 13:12:08;0008; pbs_mom;Job;process_request;request type CopyFiles from host master.perceus.centos allowed 08/06/2008 13:12:08;0008; pbs_mom;Job;139.master.perceus.centos;attempting to copy file 'master.perceus.centos:/home/yanansun/STDIN.o139' 08/06/2008 13:12:08;0008; pbs_mom;Job;139.master.perceus.centos;forking to user, uid: 556 gid: 100 homedir: '/home/yanansun' 08/06/2008 13:12:08;0008; pbs_mom;Job;process_request;request type DeleteJob from host master.perceus.centos allowed 08/06/2008 13:12:08;0080; pbs_mom;Job;139.master.perceus.centos;deleting job 139.master.perceus.centos in state EXITED 08/06/2008 13:12:08;0008; pbs_mom;Job;scan_for_terminated;pid 719 not tracked, exitcode=0 08/06/2008 13:12:12;0002; pbs_mom;n/a;is_update_stat;status update successfully sent to master 08/06/2008 13:12:57;0002; pbs_mom;n/a;is_update_stat;status update successfully sent to master 08/06/2008 13:13:42;0002; pbs_mom;n/a;is_update_stat;status update successfully sent to master 08/06/2008 13:14:27;0002; pbs_mom;n/a;is_update_stat;status update successfully sent to master 08/06/2008 13:15:12;0002; pbs_mom;n/a;is_update_stat;status update successfully sent to master 08/06/2008 13:15:57;0002; pbs_mom;n/a;is_update_stat;status update successfully sent to master [EMAIL PROTECTED] ~]# less /usr/PBS_spool/mom_logs/20080806 08/06/2008 10:20:47;0002; pbs_mom;Svr;Log;Log opened 08/06/2008 10:20:47;0002; pbs_mom;Svr;pbs_mom;caught signal 15: leaving jobs running, just exiting 08/06/2008 17:04:42;0002; pbs_mom;Svr;Log;Log opened 08/06/2008 17:04:42;0080; pbs_mom;n/a;add_static;config[1] add name arch value x86 08/06/2008 17:04:42;0080; pbs_mom;n/a;add_static;config[2] add name opsys value Linux 08/06/2008 17:04:42;0002; pbs_mom;Svr;usecp;master.perceus.centos:/home /mnt/PBS 08/06/2008 17:04:42;0002; pbs_mom;n/a;initialize;independent 08/06/2008 17:04:42;0002; pbs_mom;Svr;pbs_mom;Is up 08/06/2008 17:04:42;0002; pbs_mom;Svr;mom_main;MOM executable path and mtime at launch: /usr/sbin/pbs_mom 1211916151 08/06/2008 17:04:42;0002; pbs_mom;n/a;mom_main;hello sent to server master 08/06/2008 17:04:44;0002; pbs_mom;Svr;im_eof;End of File from addr 10.15.0.1:15001 08/06/2008 17:04:44;0002; pbs_mom;n/a;mom_main;hello sent to server master On Wed, Aug 6, 2008 at 8:33 AM, Yanan Sun <[EMAIL PROTECTED]> wrote: > but sometimes, i don't get any *.o and *.e files. > when i run jobs on node001, i got those files, but if i don't specify, > i won't get any and also when i say run jobs on node002, i won't get > any either, i don't know why. > and when i check the pbsnodes -a, everything looks fine for me. > the reason we mount PBS_spool from the master, is that they design the > whole system/network like this, there's nothing i can change about. on > the master computer, there's seperate folder contain all the needed > PBS_spool files for each nodes. each nodes just mounted there. > Thanks. > > Yanan > > On Wed, Aug 6, 2008 at 2:31 AM, Danny Sternkopf <[EMAIL PROTECTED]> wrote: >> Hi, >> >> is my understanding correct if yu set node001 offline then it doesn't work. >> If the job runs on node001 it works, right? >> >> Just to be sure I wanna tell you if you run a two nodes job, on node001 and >> node002, you will only get these PBS outputs from the first node which PBS >> has allocated for this job. The first node from your PBS node list will get >> all outputs from all nodes belonging to the job. You get one *.o and *.e >> file. That's it. For job 132 it doesn't look bad. >> >> Anyway why do you mount the PBS_SPOOL from the master on your nodes? >> >> Best regards, >> >> Danny >> >> Yanan Sun wrote: >>> >>> there's no STDIN.o130 >>> >>> home/ys is home/yanansun sorry about the confusion. >>> >>> i change both mom config file on node001 and node002, and restart >>> pbs_mom on both. it works on node001 which i got the error file >>> STDIN,e# and ouput file STDIN.o#, but it does not work on node002 >>> here is mom_log from node001 >>> 08/05/2008 11:01:48;0002; pbs_mom;Svr;Log;Log opened >>> 08/05/2008 11:01:48;0080; pbs_mom;n/a;add_static;config[1] add name >>> arch value x86 >>> 08/05/2008 11:01:48;0080; pbs_mom;n/a;add_static;config[2] add name >>> opsys value Linux >>> 08/05/2008 11:01:48;0002; pbs_mom;Svr;setloglevel;4 >>> 08/05/2008 11:01:48;0002; pbs_mom;Svr;usecp;master.perceus.centos:/home >>> /home >>> 08/05/2008 11:01:48;0002; pbs_mom;n/a;initialize;independent >>> 08/05/2008 11:01:48;0002; pbs_mom;Svr;pbs_mom;Is up >>> 08/05/2008 11:01:48;0002; pbs_mom;Svr;mom_main;MOM executable path >>> and mtime at launch: /usr/sbin/pbs_mom 1211916151 >>> 08/05/2008 11:01:48;0002; >>> pbs_mom;n/a;init_server_stream;init_server_stream: added connection to >>> master port 15001 >>> 08/05/2008 11:01:48;0002; pbs_mom;n/a;mom_main;hello sent to server >>> master >>> 08/05/2008 11:01:50;0008; pbs_mom;Job;do_rpp;got an inter-server request >>> 08/05/2008 11:01:50;0001; pbs_mom;Job;is_request;stream 0 version 1 >>> 08/05/2008 11:01:50;0001; pbs_mom;Job;is_request;command 2 received >>> 08/05/2008 11:01:50;0001; pbs_mom;Job;is_request;IS_CLUSTER_ADDRS >>> received >>> 08/05/2008 11:01:50;0001; pbs_mom;Job;is_request;is_request: >>> 10.15.0.1 added to okclients >>> 08/05/2008 11:01:50;0001; pbs_mom;Job;is_request;is_request: >>> 10.15.0.21 added to okclients >>> 08/05/2008 11:01:50;0001; pbs_mom;Job;is_request;is_request: >>> 10.15.0.173 added to okclients >>> 08/05/2008 11:01:51;0002; pbs_mom;n/a;is_update_stat;status update >>> successfully sent to master >>> 08/05/2008 11:02:04;0008; pbs_mom;Job;process_request;request type >>> QueueJob from host master.perceus.centos allowed >>> 08/05/2008 11:02:04;0008; pbs_mom;Job;process_request;request type >>> JobScript from host master.perceus.centos allowed >>> 08/05/2008 11:02:04;0008; pbs_mom;Job;process_request;request type >>> ReadyToCommit from host master.perceus.centos allowed >>> 08/05/2008 11:02:04;0008; pbs_mom;Job;process_request;request type >>> Commit from host master.perceus.centos allowed >>> 08/05/2008 11:02:04;0001; pbs_mom;Job;job_nodes;0: node001/0 >>> 08/05/2008 11:02:04;0001; pbs_mom;Job;job_nodes;job: >>> 132.master.perceus.centos numnodes=1 numvnod=1 >>> 08/05/2008 11:02:04;0008; >>> pbs_mom;Job;132.master.perceus.centos;evaluating limits for job >>> 08/05/2008 11:02:04;0001; >>> pbs_mom;Job;132.master.perceus.centos;phase 2 of job launch >>> successfully completed >>> 08/05/2008 11:02:04;0001; pbs_mom;Job;TMomFinalizeJob3;read start >>> return code=0 session=7371 >>> 08/05/2008 11:02:04;0001; pbs_mom;Job;TMomFinalizeJob3;job >>> 132.master.perceus.centos started, pid = 7371 >>> 08/05/2008 11:02:04;0001; pbs_mom;Job;132.master.perceus.centos;job >>> successfully started >>> 08/05/2008 11:02:04;0008; pbs_mom;Job;132.master.perceus.centos;job >>> 132.master.perceus.centos reported successful start on 1 node(s) >>> 08/05/2008 11:02:04;0008; pbs_mom;Job;process_request;request type >>> StatusJob from host master.perceus.centos allowed >>> 08/05/2008 11:02:04;0080; >>> pbs_mom;Job;132.master.perceus.centos;sending "send flagged" attr: >>> session_id >>> 08/05/2008 11:02:04;0008; pbs_mom;Job;process_request;request type >>> ModifyJob from host master.perceus.centos allowed >>> 08/05/2008 11:02:04;0002; >>> pbs_mom;n/a;mom_set_limits;mom_set_limits(132.master.perceus.centos,alter) >>> entered >>> 08/05/2008 11:02:04;0002; pbs_mom;n/a;mom_set_limits;setting limit >>> for attribute 'neednodes' >>> 08/05/2008 11:02:04;0002; pbs_mom;n/a;mom_set_limits;setting limit >>> for attribute 'nodes' >>> 08/05/2008 11:02:04;0002; pbs_mom;n/a;mom_set_limits;setting limit >>> for attribute 'walltime' >>> 08/05/2008 11:02:04;0008; pbs_mom;Job;132.master.perceus.centos;Job >>> Modified at request of [EMAIL PROTECTED] >>> 08/05/2008 11:02:04;0008; pbs_mom;Job;scan_for_terminated;for job >>> 132.master.perceus.centos, task 1, pid=7371, exitcode=126 >>> 08/05/2008 11:02:04;0080; >>> pbs_mom;Job;132.master.perceus.centos;scan_for_terminated: job >>> 132.master.perceus.centos task 1 terminated, sid 7371 >>> 08/05/2008 11:02:04;0008; pbs_mom;Job;132.master.perceus.centos;job >>> was terminated >>> 08/05/2008 11:02:04;0008; pbs_mom;Req;send_sisters;sending command >>> KILL_JOB for job 132.master.perceus.centos (2) >>> 08/05/2008 11:02:04;0080; >>> pbs_mom;Job;132.master.perceus.centos;local task termination detected. >>> killing job >>> 08/05/2008 11:02:04;0008; pbs_mom;Job;132.master.perceus.centos;kill_job >>> 08/05/2008 11:02:04;0080; >>> pbs_mom;Job;132.master.perceus.centos;performing job clean-up >>> 08/05/2008 11:02:04;0008; pbs_mom;Job;scan_for_terminated;pid 7379 >>> not tracked, exitcode=0 >>> 08/05/2008 11:02:04;0008; pbs_mom;Job;process_request;request type >>> CopyFiles from host master.perceus.centos allowed >>> 08/05/2008 11:02:04;0008; >>> pbs_mom;Job;132.master.perceus.centos;attempting to copy file >>> 'master.perceus.centos:/home/yanansun/STDIN.o132' >>> 08/05/2008 11:02:04;0008; >>> pbs_mom;Job;132.master.perceus.centos;forking to user, uid: 556 gid: >>> 100 homedir: '/home/yanansun' >>> 08/05/2008 11:02:04;0008; pbs_mom;Job;process_request;request type >>> DeleteJob from host master.perceus.centos allowed >>> 08/05/2008 11:02:04;0080; >>> pbs_mom;Job;132.master.perceus.centos;deleting job >>> 132.master.perceus.centos in state EXITED >>> 08/05/2008 11:02:04;0008; pbs_mom;Job;scan_for_terminated;pid 7380 >>> not tracked, exitcode=0 >>> 08/05/2008 11:02:36;0002; pbs_mom;n/a;is_update_stat;status update >>> successfully sent to master >>> 08/05/2008 11:03:21;0002; pbs_mom;n/a;is_update_stat;status update >>> successfully sent to master >>> 08/05/2008 11:04:06;0002; pbs_mom;n/a;is_update_stat;status update >>> successfully sent to master >>> 08/05/2008 11:04:51;0002; pbs_mom;n/a;is_update_stat;status update >>> successfully sent to master >>> 08/05/2008 11:05:36;0002; pbs_mom;n/a;is_update_stat;status update >>> successfully sent to master >>> 08/05/2008 11:06:21;0002; pbs_mom;n/a;is_update_stat;status update >>> successfully sent to master >>> 08/05/2008 11:07:06;0002; pbs_mom;n/a;is_update_stat;status update >>> successfully sent to master >>> 08/05/2008 11:07:51;0002; pbs_mom;n/a;is_update_stat;status update >>> successfully sent to master >>> 08/05/2008 11:08:36;0002; pbs_mom;n/a;is_update_stat;status update >>> successfully sent to master >>> 08/05/2008 11:09:21;0002; pbs_mom;n/a;is_update_stat;status update >>> successfully sent to master >>> 08/05/2008 11:10:06;0002; pbs_mom;n/a;is_update_stat;status update >>> successfully sent to master >>> >>> >>> here is the mom_log from node002 >>> 08/05/2008 09:32:07;0002; pbs_mom;Svr;Log;Log opened >>> 08/05/2008 09:32:07;0001; pbs_mom;Job;TMomFinalizeJob3;job not >>> started, Failure job exec failure, before files staged, no retry >>> 08/05/2008 09:32:07;0008; pbs_mom;Req;send_sisters;sending ABORT to >>> sisters >>> 08/05/2008 09:32:07;0008; pbs_mom;Job;128.master.perceus.centos;Job >>> Modified at request of [EMAIL PROTECTED] >>> 08/05/2008 11:02:31;0001; pbs_mom;Job;TMomFinalizeJob3;job not >>> started, Failure job exec failure, before files staged, no retry >>> 08/05/2008 11:02:31;0008; pbs_mom;Req;send_sisters;sending ABORT to >>> sisters >>> 08/05/2008 11:02:31;0008; pbs_mom;Job;133.master.perceus.centos;Job >>> Modified at request of [EMAIL PROTECTED] >>> 08/05/2008 11:03:29;0002; pbs_mom;Svr;pbs_mom;caught signal 2: >>> leaving jobs running, just exiting >>> 08/05/2008 11:03:29;0002; pbs_mom;Svr;pbs_mom;Is down >>> 08/05/2008 11:03:29;0002; pbs_mom;Svr;Log;Log closed >>> 08/05/2008 11:03:52;0002; pbs_mom;Svr;Log;Log opened >>> 08/05/2008 11:03:52;0080; pbs_mom;n/a;add_static;config[1] add name >>> arch value x86 >>> 08/05/2008 11:03:52;0080; pbs_mom;n/a;add_static;config[2] add name >>> opsys value Linux >>> 08/05/2008 11:03:52;0002; pbs_mom;Svr;usecp;master.perceus.centos:/home >>> /home >>> 08/05/2008 11:03:52;0002; pbs_mom;n/a;initialize;independent >>> 08/05/2008 11:03:52;0002; pbs_mom;Svr;pbs_mom;Is up >>> 08/05/2008 11:03:52;0002; pbs_mom;Svr;mom_main;MOM executable path >>> and mtime at launch: /usr/sbin/pbs_mom 1211916151 >>> 08/05/2008 11:03:52;0002; pbs_mom;n/a;mom_main;hello sent to server >>> master >>> 08/05/2008 11:04:02;0001; pbs_mom;Job;TMomFinalizeJob3;job not >>> started, Failure job exec failure, before files staged, no retry >>> 08/05/2008 11:04:02;0008; pbs_mom;Req;send_sisters;sending ABORT to >>> sisters >>> 08/05/2008 11:04:02;0008; pbs_mom;Job;134.master.perceus.centos;Job >>> Modified at request of [EMAIL PROTECTED] >>> >>> >>> both node001 and node002 mounted PBS_spool file on masters, but each >>> nodes every their own PBS_spool folder even though the content are the >>> same. >>> Thanks a lot. >>> >>> Yanan >>> >>> On Tue, Aug 5, 2008 at 10:55 AM, Danny Sternkopf >>> <[EMAIL PROTECTED]> wrote: >>>> >>>> Hi, >>>> >>>> check if file master.perceus.centos:/home/yanansun/STDIN.o130 exists. >>>> >>>> What is /home/ys? >>>> >>>> Can you try in your MOM cinfig file: >>>> $usecp *:/home /home >>>> >>>> Best regards, >>>> >>>> Danny >>>> >>>> Yanan Sun wrote: >>>>> >>>>> here is the new mom_log >>>>> >>>>> 08/05/2008 10:26:06;0002; pbs_mom;Svr;pbs_mom;caught signal 2: >>>>> leaving jobs running, just exiting >>>>> 08/05/2008 10:26:06;0002; pbs_mom;Svr;pbs_mom;Is down >>>>> 08/05/2008 10:26:06;0002; pbs_mom;Svr;Log;Log closed >>>>> 08/05/2008 10:26:16;0002; pbs_mom;Svr;Log;Log opened >>>>> 08/05/2008 10:26:16;0080; pbs_mom;n/a;add_static;config[1] add name >>>>> arch value x86 >>>>> 08/05/2008 10:26:16;0080; pbs_mom;n/a;add_static;config[2] add name >>>>> opsys value Linux >>>>> 08/05/2008 10:26:16;0002; pbs_mom;Svr;setloglevel;4 >>>>> 08/05/2008 10:26:16;0002; >>>>> pbs_mom;Svr;usecp;master.perceus.centos:/home /mnt/PBS >>>>> 08/05/2008 10:26:16;0002; pbs_mom;n/a;initialize;independent >>>>> 08/05/2008 10:26:16;0002; pbs_mom;Svr;pbs_mom;Is up >>>>> 08/05/2008 10:26:16;0002; pbs_mom;Svr;mom_main;MOM executable path >>>>> and mtime at launch: /usr/sbin/pbs_mom 1211916151 >>>>> 08/05/2008 10:26:16;0002; >>>>> pbs_mom;n/a;init_server_stream;init_server_stream: added connection to >>>>> master port 15001 >>>>> 08/05/2008 10:26:16;0002; pbs_mom;n/a;mom_main;hello sent to server >>>>> master >>>>> 08/05/2008 10:26:18;0008; pbs_mom;Job;do_rpp;got an inter-server >>>>> request >>>>> 08/05/2008 10:26:18;0001; pbs_mom;Job;is_request;stream 0 version 1 >>>>> 08/05/2008 10:26:18;0001; pbs_mom;Job;is_request;command 2 received >>>>> 08/05/2008 10:26:18;0001; pbs_mom;Job;is_request;IS_CLUSTER_ADDRS >>>>> received >>>>> 08/05/2008 10:26:18;0001; pbs_mom;Job;is_request;is_request: >>>>> 10.15.0.1 added to okclients >>>>> 08/05/2008 10:26:18;0001; pbs_mom;Job;is_request;is_request: >>>>> 10.15.0.21 added to okclients >>>>> 08/05/2008 10:26:18;0001; pbs_mom;Job;is_request;is_request: >>>>> 10.15.0.173 added to okclients >>>>> 08/05/2008 10:26:19;0002; pbs_mom;n/a;is_update_stat;status update >>>>> successfully sent to master >>>>> 08/05/2008 10:26:44;0008; pbs_mom;Job;process_request;request type >>>>> QueueJob from host master.perceus.centos allowed >>>>> 08/05/2008 10:26:44;0008; pbs_mom;Job;process_request;request type >>>>> JobScript from host master.perceus.centos allowed >>>>> 08/05/2008 10:26:44;0008; pbs_mom;Job;process_request;request type >>>>> ReadyToCommit from host master.perceus.centos allowed >>>>> 08/05/2008 10:26:44;0008; pbs_mom;Job;process_request;request type >>>>> Commit from host master.perceus.centos allowed >>>>> 08/05/2008 10:26:44;0001; pbs_mom;Job;job_nodes;0: node001/0 >>>>> 08/05/2008 10:26:44;0001; pbs_mom;Job;job_nodes;job: >>>>> 130.master.perceus.centos numnodes=1 numvnod=1 >>>>> 08/05/2008 10:26:44;0008; >>>>> pbs_mom;Job;130.master.perceus.centos;evaluating limits for job >>>>> 08/05/2008 10:26:44;0001; >>>>> pbs_mom;Job;130.master.perceus.centos;phase 2 of job launch >>>>> successfully completed >>>>> 08/05/2008 10:26:44;0001; pbs_mom;Job;TMomFinalizeJob3;read start >>>>> return code=0 session=7024 >>>>> 08/05/2008 10:26:44;0001; pbs_mom;Job;TMomFinalizeJob3;job >>>>> 130.master.perceus.centos started, pid = 7024 >>>>> 08/05/2008 10:26:44;0001; pbs_mom;Job;130.master.perceus.centos;job >>>>> successfully started >>>>> 08/05/2008 10:26:44;0008; pbs_mom;Job;130.master.perceus.centos;job >>>>> 130.master.perceus.centos reported successful start on 1 node(s) >>>>> 08/05/2008 10:26:44;0008; pbs_mom;Job;process_request;request type >>>>> StatusJob from host master.perceus.centos allowed >>>>> 08/05/2008 10:26:44;0080; >>>>> pbs_mom;Job;130.master.perceus.centos;sending "send flagged" attr: >>>>> session_id >>>>> 08/05/2008 10:26:44;0008; pbs_mom;Job;process_request;request type >>>>> ModifyJob from host master.perceus.centos allowed >>>>> 08/05/2008 10:26:44;0002; >>>>> >>>>> pbs_mom;n/a;mom_set_limits;mom_set_limits(130.master.perceus.centos,alter) >>>>> entered >>>>> 08/05/2008 10:26:44;0002; pbs_mom;n/a;mom_set_limits;setting limit >>>>> for attribute 'neednodes' >>>>> 08/05/2008 10:26:44;0002; pbs_mom;n/a;mom_set_limits;setting limit >>>>> for attribute 'nodes' >>>>> 08/05/2008 10:26:44;0002; pbs_mom;n/a;mom_set_limits;setting limit >>>>> for attribute 'walltime' >>>>> 08/05/2008 10:26:44;0008; pbs_mom;Job;130.master.perceus.centos;Job >>>>> Modified at request of [EMAIL PROTECTED] >>>>> 08/05/2008 10:26:44;0008; pbs_mom;Job;scan_for_terminated;for job >>>>> 130.master.perceus.centos, task 1, pid=7024, exitcode=126 >>>>> 08/05/2008 10:26:44;0080; >>>>> pbs_mom;Job;130.master.perceus.centos;scan_for_terminated: job >>>>> 130.master.perceus.centos task 1 terminated, sid 7024 >>>>> 08/05/2008 10:26:44;0008; pbs_mom;Job;130.master.perceus.centos;job >>>>> was terminated >>>>> 08/05/2008 10:26:44;0008; pbs_mom;Req;send_sisters;sending command >>>>> KILL_JOB for job 130.master.perceus.centos (2) >>>>> 08/05/2008 10:26:44;0080; >>>>> pbs_mom;Job;130.master.perceus.centos;local task termination detected. >>>>> killing job >>>>> 08/05/2008 10:26:44;0008; >>>>> pbs_mom;Job;130.master.perceus.centos;kill_job >>>>> 08/05/2008 10:26:44;0080; >>>>> pbs_mom;Job;130.master.perceus.centos;performing job clean-up >>>>> 08/05/2008 10:26:44;0008; pbs_mom;Job;scan_for_terminated;pid 7032 >>>>> not tracked, exitcode=0 >>>>> 08/05/2008 10:26:44;0008; pbs_mom;Job;process_request;request type >>>>> CopyFiles from host master.perceus.centos allowed >>>>> 08/05/2008 10:26:44;0008; >>>>> pbs_mom;Job;130.master.perceus.centos;attempting to copy file >>>>> 'master.perceus.centos:/home/yanansun/STDIN.o130' >>>>> 08/05/2008 10:26:44;0008; >>>>> pbs_mom;Job;130.master.perceus.centos;forking to user, uid: 556 gid: >>>>> 100 homedir: '/home/yanansun' >>>>> 08/05/2008 10:26:52;0008; pbs_mom;Job;scan_for_terminated;pid 7033 >>>>> not tracked, exitcode=0 >>>>> 08/05/2008 10:26:52;0008; pbs_mom;Job;process_request;request type >>>>> DeleteJob from host master.perceus.centos allowed >>>>> 08/05/2008 10:26:52;0080; >>>>> pbs_mom;Job;130.master.perceus.centos;deleting job >>>>> 130.master.perceus.centos in state EXITED >>>>> 08/05/2008 10:27:04;0002; pbs_mom;n/a;is_update_stat;status update >>>>> successfully sent to master >>>>> >>>>> >>>>> [EMAIL PROTECTED] ~]# df -k >>>>> Filesystem 1K-blocks Used Available Use% Mounted on >>>>> none 4093988 0 4093988 0% /dev/shm >>>>> /dev/sda2 55652312 184400 52640876 1% /tmp >>>>> 10.15.0.1:/usr 14769088 13010432 1008416 93% /usr >>>>> 192.168.44.232:/home 71485088 45707168 22189408 68% /home >>>>> 10.15.0.7:/raidd/raida >>>>> 7324868512 5028359424 2296509088 69% /raida >>>>> 10.15.0.7:/raidd/raidb >>>>> 7324868512 5028359424 2296509088 69% /raidb >>>>> 10.15.0.1:/mnt/PBS 9361184 181504 8704160 3% /mnt/PBS >>>>> >>>>> >>>>> Thanks. >>>>> >>>>> Yanan >>>>> >>>>> >>>>> On Tue, Aug 5, 2008 at 10:10 AM, Danny Sternkopf >>>>> <[EMAIL PROTECTED]> wrote: >>>>>> >>>>>> Hi, >>>>>> >>>>>> there is no error visible. >>>>>> Can you increase verbosity level of your MOM. >>>>>> In ../mom_priv/config set: >>>>>> >>>>>> $logevent 255 >>>>>> $loglevel 4 >>>>>> >>>>>> Then restart your MOM. Then try again and send me the MOM log file and >>>>>> 'df >>>>>> -k' from node001. >>>>>> >>>>>> Best regards, >>>>>> >>>>>> Danny >>>>>> >>>>>> >>>>>> >>>>>> Yanan Sun wrote: >>>>>>> >>>>>>> 08/05/2008 09:22:10;0002; pbs_mom;Svr;Log;Log opened >>>>>>> 08/05/2008 09:22:10;0001; pbs_mom;Svr;pbs_mom;Resource temporarily >>>>>>> unavailable (11) in pbs_mom, cannot lock >>>>>>> '/mnt/PBS/node001/PBS_spool/mom_priv/mom.lock' >>>>>>> - another mom running >>>>>>> 08/05/2008 09:23:34;0002; pbs_mom;Svr;Log;Log opened >>>>>>> 08/05/2008 09:23:34;0001; pbs_mom;Svr;pbs_mom;Resource temporarily >>>>>>> unavailable (11) in pbs_mom, cannot lock >>>>>>> '/mnt/PBS/node001/PBS_spool/mom_priv/mom.lock' >>>>>>> - another mom running >>>>>>> 08/05/2008 09:24:57;0002; pbs_mom;Svr;Log;Log opened >>>>>>> 08/05/2008 09:24:57;0080; pbs_mom;n/a;add_static;config[1] add name >>>>>>> arch value x86 >>>>>>> 08/05/2008 09:24:57;0080; pbs_mom;n/a;add_static;config[2] add name >>>>>>> opsys value Linux >>>>>>> 08/05/2008 09:24:57;0002; >>>>>>> pbs_mom;Svr;usecp;master.perceus.centos:/home /mnt/PBS >>>>>>> 08/05/2008 09:24:57;0002; pbs_mom;n/a;initialize;independent >>>>>>> 08/05/2008 09:24:57;0002; pbs_mom;Svr;pbs_mom;Is up >>>>>>> 08/05/2008 09:24:57;0002; pbs_mom;Svr;mom_main;MOM executable path >>>>>>> and mtime at launch: /usr/sbin/pbs_mom 1211916151 >>>>>>> 08/05/2008 09:24:57;0002; pbs_mom;n/a;mom_main;hello sent to server >>>>>>> master >>>>>>> 08/05/2008 09:36:50;0001; pbs_mom;Job;TMomFinalizeJob3;job >>>>>>> 129.master.perceus.centos started, pid = 6452 >>>>>>> 08/05/2008 09:36:50;0008; pbs_mom;Job;129.master.perceus.centos;Job >>>>>>> Modified at request of [EMAIL PROTECTED] >>>>>>> 08/05/2008 09:36:50;0080; >>>>>>> pbs_mom;Job;129.master.perceus.centos;scan_for_terminated: job >>>>>>> 129.master.perceus.centos task 1 terminated, sid 6452 >>>>>>> 08/05/2008 09:36:50;0008; pbs_mom;Job;129.master.perceus.centos;job >>>>>>> was terminated >>>>>>> >>>>>>> that's the log for today when i run the job, i could tell where the >>>>>>> problems are. could you help me with that, please? >>>>>>> thanks. >>>>>>> >>>>>>> Yanan >>>>>>> >>>>>>> On Tue, Aug 5, 2008 at 9:45 AM, Danny Sternkopf >>>>>>> <[EMAIL PROTECTED]> >>>>>>> wrote: >>>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> your $PBS_SPOOL/mom_logs/<logfile> will tell you why it is not there. >>>>>>>> >>>>>>>> Best regards, >>>>>>>> >>>>>>>> Danny >>>>>>>> >>>>>>>> Yanan Sun wrote: >>>>>>>>> >>>>>>>>> -bash-3.1$ echo "perl helloworld.pl node001" |qsub -q short -l >>>>>>>>> nodes=node001 >>>>>>>>> 129.master.perceus.centos >>>>>>>>> -bash-3.1$ qstat -f >>>>>>>>> Job Id: 129.master.perceus.centos >>>>>>>>> Job_Name = STDIN >>>>>>>>> Job_Owner = [EMAIL PROTECTED] >>>>>>>>> resources_used.cput = 00:00:00 >>>>>>>>> resources_used.mem = 0kb >>>>>>>>> resources_used.vmem = 0kb >>>>>>>>> resources_used.walltime = 00:00:00 >>>>>>>>> job_state = E >>>>>>>>> queue = short >>>>>>>>> server = master.perceus.centos >>>>>>>>> Checkpoint = u >>>>>>>>> ctime = Tue Aug 5 09:37:01 2008 >>>>>>>>> Error_Path = master.perceus.centos:/home/ys/STDIN.e129 >>>>>>>>> exec_host = node001/0 >>>>>>>>> Hold_Types = n >>>>>>>>> Join_Path = n >>>>>>>>> Keep_Files = n >>>>>>>>> Mail_Points = a >>>>>>>>> mtime = Tue Aug 5 09:37:02 2008 >>>>>>>>> Output_Path = master.perceus.centos:/home/ys/STDIN.o129 >>>>>>>>> Priority = 0 >>>>>>>>> qtime = Tue Aug 5 09:37:01 2008 >>>>>>>>> Rerunable = True >>>>>>>>> Resource_List.nodect = 1 >>>>>>>>> Resource_List.nodes = node001 >>>>>>>>> Resource_List.walltime = 00:05:00 >>>>>>>>> session_id = 6452 >>>>>>>>> Variable_List = PBS_O_HOME=/home/ys,PBS_O_LANG=en_US.UTF-8, >>>>>>>>> PBS_O_LOGNAME=ys, >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> PBS_O_PATH=/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/home/ys/Desktop/cmake-2.6.0-Linux-i386/bin, >>>>>>>>> PBS_O_MAIL=/var/spool/mail/ys,PBS_O_SHELL=/bin/bash, >>>>>>>>> PBS_O_HOST=master.perceus.centos,PBS_O_WORKDIR=/home/ys, >>>>>>>>> PBS_O_QUEUE=short >>>>>>>>> etime = Tue Aug 5 09:37:01 2008 >>>>>>>>> exit_status = 126 >>>>>>>>> submit_args = -q short -l nodes=node001 >>>>>>>>> >>>>>>>>> so the error file should be at /home/ys/ >>>>>>>>> but it was not there >>>>>>>>> >>>>>>>>> thanks. >>>>>>>>> >>>>>>>>> Yanan >>>>>>>>> >>>>>>>>> >>>>>>>>> On Fri, Aug 1, 2008 at 3:28 AM, Danny Sternkopf >>>>>>>>> <[EMAIL PROTECTED]> >>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> check you MOM log file in $PBS_SPOOL/mom_logs/ on the node which >>>>>>>>>> was >>>>>>>>>> set >>>>>>>>>> offline. >>>>>>>>>> >>>>>>>>>> So there is a difference between node001 and node002. >>>>>>>>>> >>>>>>>>>> If you run 'qstat -f' on the job you see where the *.o and *.o >>>>>>>>>> files >>>>>>>>>> will >>>>>>>>>> be >>>>>>>>>> stored. Check the name of the target host if it can be accessed >>>>>>>>>> from >>>>>>>>>> these >>>>>>>>>> two nodes. >>>>>>>>>> >>>>>>>>>> Best regards, >>>>>>>>>> >>>>>>>>>> Danny >>>>>>>>>> >>>>>>>>>> Yanan Sun wrote: >>>>>>>>>>> >>>>>>>>>>> i added two nodes on the cluster, node001 and node002. >>>>>>>>>>> if i keep both free, i don't get any STDIN.e# and STDIN.o# files. >>>>>>>>>>> if i put one offline, i got both files. >>>>>>>>>>> anyone knows why? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Yanan >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> mauiusers mailing list >>>>>>>>>>> [email protected] >>>>>>>>>>> http://www.supercluster.org/mailman/listinfo/mauiusers >>>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> torqueusers mailing list >>>>>>>>>> [EMAIL PROTECTED] >>>>>>>>>> http://www.supercluster.org/mailman/listinfo/torqueusers >>>>>>>>>> >>>>>>>> -- >>>>>>>> Danny Sternkopf http://www.nec.de/hpc [EMAIL PROTECTED] >>>>>>>> HPCE Division Germany phone: +49-711-68770-35 fax: +49-711-6877145 >>>>>>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>>>>>>> NEC Deutschland GmbH, Hansaallee 101, 40549 Düsseldorf >>>>>>>> Geschäftsführer Yuya Momose >>>>>>>> Handelsregister Düsseldorf HRB 57941; VAT ID DE129424743 >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> mauiusers mailing list >>>>>>>> [email protected] >>>>>>>> http://www.supercluster.org/mailman/listinfo/mauiusers >>>>>>>> >>>>>> -- >>>>>> Danny Sternkopf http://www.nec.de/hpc [EMAIL PROTECTED] >>>>>> HPCE Division Germany phone: +49-711-68770-35 fax: +49-711-6877145 >>>>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>>>>> NEC Deutschland GmbH, Hansaallee 101, 40549 Düsseldorf >>>>>> Geschäftsführer Yuya Momose >>>>>> Handelsregister Düsseldorf HRB 57941; VAT ID DE129424743 >>>>>> >>>>>> >>>> -- >>>> Danny Sternkopf http://www.nec.de/hpc [EMAIL PROTECTED] >>>> HPCE Division Germany phone: +49-711-68770-35 fax: +49-711-6877145 >>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>>> NEC Deutschland GmbH, Hansaallee 101, 40549 Düsseldorf >>>> Geschäftsführer Yuya Momose >>>> Handelsregister Düsseldorf HRB 57941; VAT ID DE129424743 >>>> >>>> >>> >> >> -- >> Danny Sternkopf http://www.nec.de/hpc [EMAIL PROTECTED] >> HPCE Division Germany phone: +49-711-68770-35 fax: +49-711-6877145 >> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >> NEC Deutschland GmbH, Hansaallee 101, 40549 Düsseldorf >> Geschäftsführer Yuya Momose >> Handelsregister Düsseldorf HRB 57941; VAT ID DE129424743 >> >> > _______________________________________________ mauiusers mailing list [email protected] http://www.supercluster.org/mailman/listinfo/mauiusers
