Hi Jiajun, You are right, it seems to be the reason, the doesnt connect..
Each node write the checkpoint to a directory that is mount with NFS in other two machines. I can see the files in both nodes. I set DMTCP_CHECKPOINT_DIR to that folder. Anyway, I try to wirte the locally, but it is the same problem. I copy the output of 'ls' command on the checkpoint folder on both nodes: node m110a (the one that connects, where the coordinator runs): hpcpro@m110a:~/nfs$ ls -l total 26536 -rw------- 1 hpcpro hpcpro 5122030 Nov 19 12:31 ckpt_app_heat_512_1310c955e7a-71000-564e0755.dmtcp drwxr-xr-x 2 hpcpro hpcpro 4096 Nov 19 12:31 ckpt_app_heat_512_1310c955e7a-71000-564e0755_files -rw------- 1 hpcpro hpcpro 5122218 Nov 19 12:31 ckpt_app_heat_512_1310c955e7a-74000-564e0755.dmtcp drwxr-xr-x 2 hpcpro hpcpro 4096 Nov 19 12:31 ckpt_app_heat_512_1310c955e7a-74000-564e0755_files -rw------- 1 hpcpro hpcpro 5119200 Nov 19 12:31 ckpt_app_heat_512_1310c955e7a-76000-564e0755.dmtcp drwxr-xr-x 2 hpcpro hpcpro 4096 Nov 19 12:31 ckpt_app_heat_512_1310c955e7a-76000-564e0755_files -rw------- 1 hpcpro hpcpro 5119579 Nov 19 12:31 ckpt_app_heat_512_1310c955e7a-78000-564e0755.dmtcp drwxr-xr-x 2 hpcpro hpcpro 4096 Nov 19 12:31 ckpt_app_heat_512_1310c955e7a-78000-564e0755_files -rw------- 1 hpcpro hpcpro 2736819 Nov 19 12:31 ckpt_dmtcp_ssh_1310c955e7a-67000-564e0754.dmtcp -rw------- 1 hpcpro hpcpro 3899774 Nov 19 12:31 ckpt_orterun_1310c955e7a-66000-564e0754.dmtcp drwxr-xr-x 2 hpcpro hpcpro 4096 Nov 19 12:31 ckpt_orterun_1310c955e7a-66000-564e0754_files -rwxr--r-- 1 hpcpro hpcpro 12091 Nov 19 12:31 dmtcp_restart_script_1310c955e7a-66000-564e0754.sh lrwxrwxrwx 1 hpcpro hpcpro 67 Nov 19 12:31 dmtcp_restart_script.sh -> /home/hpcpro/nfs/dmtcp_restart_script_1310c955e7a-66000-564e0754.sh -rw-r--r-- 1 hpcpro hpcpro 254 Nov 19 12:31 jtimings.csv node m111a (the one doesnt connect) hpcpro@m111a:~/nfs$ ls -l total 26348 -rw------- 1 hpcpro hpcpro 5117116 Nov 19 12:31 ckpt_app_heat_512_1310c955fc5-72000-564e0755.dmtcp drwxr-xr-x 2 hpcpro hpcpro 4096 Nov 19 12:31 ckpt_app_heat_512_1310c955fc5-72000-564e0755_files -rw------- 1 hpcpro hpcpro 5116818 Nov 19 12:31 ckpt_app_heat_512_1310c955fc5-73000-564e0755.dmtcp drwxr-xr-x 2 hpcpro hpcpro 4096 Nov 19 12:31 ckpt_app_heat_512_1310c955fc5-73000-564e0755_files -rw------- 1 hpcpro hpcpro 5116096 Nov 19 12:31 ckpt_app_heat_512_1310c955fc5-75000-564e0755.dmtcp drwxr-xr-x 2 hpcpro hpcpro 4096 Nov 19 12:31 ckpt_app_heat_512_1310c955fc5-75000-564e0755_files -rw------- 1 hpcpro hpcpro 5115075 Nov 19 12:31 ckpt_app_heat_512_1310c955fc5-77000-564e0755.dmtcp drwxr-xr-x 2 hpcpro hpcpro 4096 Nov 19 12:31 ckpt_app_heat_512_1310c955fc5-77000-564e0755_files -rw------- 1 hpcpro hpcpro 2736548 Nov 19 12:31 ckpt_dmtcp_sshd_1310c955fc5-69000-564e0755.dmtcp -rw------- 1 hpcpro hpcpro 3742556 Nov 19 12:31 ckpt_orted_1310c955fc5-70000-564e0755.dmtcp drwxr-xr-x 2 hpcpro hpcpro 4096 Nov 19 12:31 ckpt_orted_1310c955fc5-70000-564e0755_files I cant figure out what can it be... Thanks for your help, regards Marina On 11/18/15, Jiajun Cao <jia...@ccs.neu.edu> wrote: > Hi Marina, > > Where are checkpoint images stored? Are they stored in a shared file > system, or to local storage? From what I can tell from the log, there're 12 > processes before checkpoint, and hence 12 checkpoint images. On restart, > only 6 of them connect to the coordinator. It may be the fact that the > restart script couldn't find the rest images. Could you verify that? > > Best, > Jiajun > > On Wed, Nov 18, 2015 at 6:03 PM, Marina Moran > <esperandoelmila...@gmail.com> > wrote: > >> Hi All! I'm back using DMTCP! >> >> I'm having a problem when restarting a checkpoint. >> >> I have two nodes (PCs) in an ethernet lan, with: >> -Debian 8 Jessi, >> -DMTCP 2.4.2 (configure with -enable-timing) >> -OpenMPI 1.10.1. >> >> I do: >> $ dmtcp_launch mpirun -np 8 -hostfile hosts app_heat_512 >> >> On the console where the coordinator is running, I press 'c' to >> checkpoint. After that, I killed the application or it finished, and >> then, from the same directory where the checkpoints are stored I run >> the restarting script, with the following output: >> >> $ ./dmtcp_restart_script.sh >> [75000] WARNING at socketconnection.cpp:540 in postRestart; >> REASON='JWARNING(_real_bind(_fds[0], (sockaddr*) >> &_bindAddr,_bindAddrlen) == 0) failed' >> (strerror((*__errno_location ()))) = Address already in use >> id() = 1310c955e7a-75000-564d1a58(99506) >> Message: Bind failed. >> [77000] WARNING at socketconnection.cpp:540 in postRestart; >> REASON='JWARNING(_real_bind(_fds[0], (sockaddr*) >> &_bindAddr,_bindAddrlen) == 0) failed' >> (strerror((*__errno_location ()))) = Address already in use >> id() = 1310c955e7a-77000-564d1a58(99517) >> Message: Bind failed. >> >> On the coordinator console it outputs this: >> >> [762] NOTE at dmtcp_coordinator.cpp:1137 in >> validateRestartingWorkerProcess; REASON='FIRST dmtcp_restart >> connection. Set numPeers. Generate timestamp' >> numPeers = 12 >> curTimeStamp = 23166315138 >> compId = 1310c955e7a-66000-564d1a57 >> [762] WARNING at jtimer.h:81 in start; REASON='JWARNING(!_isStarted) >> failed' >> _name = restart >> [762] NOTE at dmtcp_coordinator.cpp:1079 in onConnect; REASON='worker >> connected' >> hello_remote.from = 1310c955e7a-66000-564d1a57 >> [762] NOTE at dmtcp_coordinator.cpp:1079 in onConnect; REASON='worker >> connected' >> hello_remote.from = 1310c955e7a-67000-564d1a57 >> [762] NOTE at dmtcp_coordinator.cpp:1079 in onConnect; REASON='worker >> connected' >> hello_remote.from = 1310c955e7a-71000-564d1a58 >> [762] NOTE at dmtcp_coordinator.cpp:1079 in onConnect; REASON='worker >> connected' >> hello_remote.from = 1310c955e7a-73000-564d1a58 >> [762] NOTE at dmtcp_coordinator.cpp:1079 in onConnect; REASON='worker >> connected' >> hello_remote.from = 1310c955e7a-77000-564d1a58 >> [762] NOTE at dmtcp_coordinator.cpp:1079 in onConnect; REASON='worker >> connected' >> hello_remote.from = 1310c955e7a-75000-564d1a58 >> >> >> And when I pressed L to show connected nodes: >> l >> Client List: >> #, PROG[virtPID:realPID]@HOST, DMTCP-UNIQUEPID, STATE >> 64, orterun[66000:1530]@m110a, 1310c955e7a-66000-564d1a57, CHECKPOINTED >> 65, dmtcp_ssh[67000:1618]@m110a, 1310c955e7a-67000-564d1a57, CHECKPOINTED >> 66, app_heat_512[71000:1619]@m110a, 1310c955e7a-71000-564d1a58, >> CHECKPOINTED >> 67, app_heat_512[73000:1620]@m110a, 1310c955e7a-73000-564d1a58, >> CHECKPOINTED >> 68, app_heat_512[77000:1622]@m110a, 1310c955e7a-77000-564d1a58, >> CHECKPOINTED >> 69, app_heat_512[75000:1621]@m110a, 1310c955e7a-75000-564d1a58, >> CHECKPOINTED >> >> >> It seems ti hangs... It never ends. >> >> Hope this is something I forgot... >> >> Thanks all in advance, >> Regards >> Marina >> >> >> ------------------------------------------------------------------------------ >> _______________________________________________ >> Dmtcp-forum mailing list >> Dmtcp-forum@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/dmtcp-forum >> > ------------------------------------------------------------------------------ _______________________________________________ Dmtcp-forum mailing list Dmtcp-forum@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dmtcp-forum
