Hi Manuel,
It's not entirely clear what you're asking, but let me review some
concepts from DMTCP. I apologize if you already know this.
1. The dmtcp_restart_script.sh can be executed from any machine. It is
independent of the location of the coordinator and independent of the
location of the checkpoint images.
2. Start a coordinator explicitly on the desired host.
3. The important lines in the script (executed on any host) will be of the
form:
COORD_HOST= ...
COORD_PORT= ...
ssh HOST1 dmtcp_restart --coord-host COORD_HOST --coord-port COORD_PORT
full_path_to_checkpoint_image
... (same for HOST2, etc.) ...
3. In this restart script, it assumes that the checkpoint image for HOST1
is already on the machine HOST1.
In a shared filesystem, this is easy. Otherwise, we must copy the
checkpoint image there.
If all processes are on the same host, one does not need to start an
explicit coordinator. In this special case, dmtcp_restart will create a
local host if none exists already.
If the above context is also what you were assuming, could you re-phrase
your question using the above description?
Our best guess is the following. In your case, the checkpoint image is in
/tmp. Since /tmp is not part of any shared filesystem, the checkpoint
image is visible only on the host where it was created at checkpoint time.
If you restart with the same processes on the same hosts as at the time of
checkpoint, then everything will work fine.
Otherwise, you will need to copy the checkpoint images to the path given in
the restart script (or copy the checkpoint images into the shared
filesystem, and modify the restart script accordingly).
Thanks,
- Jiajun and Gene
Best,
Jiajun
On Fri, Oct 9, 2015 at 1:36 PM, Manuel Rodríguez Pascual <
manuel.rodriguez.pasc...@gmail.com> wrote:
> Good morning all,
>
> I am working on trying to reduce the checkpoint time of MPI
> applications. Our basic scenario is a multi node cluster. Every node
> has local storage (fast) and network storage (slow). Network storage
> is employed to create shared folders, accesible by all the nodes on
> the cluster. Local storage is only visible by the node.
>
> A basic approach is to configure DMTCP to employ this shared storage.
> When a checkpoint operation is requested, all nodes write the
> information to the specified folder. Then, on the restart step, data
> is accesible by everyone (most important, by the coordinator) and it
> works great.
>
> However, a second -and more interesting step- would be to employ local
> storage for the storage of checkpoints. This way, both checkpoint and
> restart operations are faster.
>
> This is not a problem with serial applications: as the coordinator can
> access local files, any location can be chosen and the restart works
> too. This is not only faster but also reduces network overhead, which
> is good.
>
> The problem comes with MPI jobs. Checkpoint operation is OK: if a
> given folder (let's say /tmp) exists on all nodes, the checkpoint
> files are written there. If I look at dmtcp_restart.sh file, I can
> see (simplified for legibility):
>
> worker_ckpts='
> :: host1 :bg: /tmp/my_files....
> :: host2 :bg: /tmp/my_files...
>
> which is correct. All these files exist and are accesible by the user.
>
> However, when a restart operation is requested, it does not work.
> Looking into dmtcp_script.sh, this is the operation where it gets
> stuck.
>
> + dmtcp_restart --join --coord-host host1 --coord-port 7779
> /tmp/ckpt_dmtcp_srun_helper_1b69d09fb3238b30-41000-5617e813.dmtcp
> /tmp/ckpt_mpiexec.hydra_1b69d09fb3238b30-40000-5617e813.dmtcp
>
> If I go to host 1,
>
> # dmtcp_command -s
> Coordinator:
> Host: localhost
> Port: 7779 (default port)
> Status...
> NUM_PEERS=2
> RUNNING=no
> CKPT_INTERVAL=0 (checkpoint manually)
>
> # dmtcp_command -k
> This causes the dmtcp_restart operation to finish, so I assume it is
> kind of working.
>
> So, any idea or suggestion?
>
> Just in case it is useful, please find below the full dmtcp_restart.sh
> trace.
>
> Cheers,
>
>
> Manuel
>
>
> ''''''
>
> + /bin/bash ./dmtcp_restart_script.sh
>
> + usage_str='USAGE:
>
> dmtcp_restart_script.sh [OPTIONS]
>
>
> OPTIONS:
>
> --coord-host, -h, (environment variable DMTCP_COORD_HOST):
>
> Hostname where dmtcp_coordinator is running
>
> --coord-port, -p, (environment variable DMTCP_COORD_PORT):
>
> Port where dmtcp_coordinator is running
>
> --hostfile <arg0> :
>
> Provide a hostfile (One host per line, "#" indicates comments)
>
> --ckptdir, -d, (environment variable DMTCP_CHECKPOINT_DIR):
>
> Directory to store checkpoint images
>
> (default: use the same directory used in previous checkpoint)
>
> --restartdir, -d, (environment variable DMTCP_RESTART_DIR):
>
> Directory to read checkpoint images from
>
> --tmpdir, -t, (environment variable DMTCP_TMPDIR):
>
> Directory to store temporary files (default: $TMDPIR or /tmp)
>
> --no-strict-uid-checking:
>
> Disable uid checking for the checkpoint image. This allows the
>
> checkpoint image to be restarted by a different user than the one
>
> that create it. (environment variable DMTCP_DISABLE_UID_CHECKING)
>
> --interval, -i, (environment variable DMTCP_CHECKPOINT_INTERVAL):
>
> Time in seconds between automatic checkpoints
>
> (Default: Use pre-checkpoint value)
>
> --help:
>
> Print this message and exit.'
>
> + ckpt_timestamp='Fri Oct 9 18:15:39 2015'
>
> + coord_host=acme12.ciemat.es
>
> + test -z acme12.ciemat.es
>
> + coord_port=7779
>
> + test -z 7779
>
> + checkpoint_interval=
>
> + test -z ''
>
> + checkpoint_interval=0
>
> + export DMTCP_CHECKPOINT_INTERVAL=0
>
> + DMTCP_CHECKPOINT_INTERVAL=0
>
> + '[' 0 -gt 0 ']'
>
> + dmt_rstr_cmd=/home/localsoft/dmtcp/bin/dmtcp_restart
>
> + which /home/localsoft/dmtcp/bin/dmtcp_restart
>
> + which /home/localsoft/dmtcp/bin/dmtcp_restart
>
> + which /home/localsoft/dmtcp/bin/dmtcp_restart
>
> + worker_ckpts='
>
> :: acme12.ciemat.es :bg:
> /tmp/ckpt_dmtcp_srun_helper_1b69d09fb3238b30-41000-5617e813.dmtcp
> /tmp/ckpt_hydra_pmi_proxy_1b69d09fb3238b30-43000-5617e813.dmtcp
> /tmp/ckpt_mpiexec.hydra_1b69d09fb3238b30-40000-5617e813.dmtcp
> /tmp/ckpt_helloWorldMPI_1b69d09fb3238b30-44000-5617e813.dmtcp
>
> :: acme13.ciemat.es :bg:
> /tmp/ckpt_hydra_pmi_proxy_54385264162a2589-45000-5617e813.dmtcp
> /tmp/ckpt_helloWorldMPI_54385264162a2589-46000-5617e814.dmtcp
>
> '
>
> ++ which ibrun
>
> + ibrun_path=
>
> + '[' '!' -n '' ']'
>
> ++ which dmtcp_discover_rm
>
> + discover_rm_path=/home/localsoft/dmtcp/bin/dmtcp_discover_rm
>
> + '[' -n /home/localsoft/dmtcp/bin/dmtcp_discover_rm ']'
>
> ++ dmtcp_discover_rm -t
>
> + eval RES_MANAGER=SLURM 'manager_resources="*acme12:2' '"'
>
> ++ RES_MANAGER=SLURM
>
> ++ manager_resources='*acme12:2 '
>
> ++ which srun
>
> + srun_path=/home/localsoft/slurm/soft/bin/srun
>
> ++ which dmtcp_rm_loclaunch
>
> + llaunch=/home/localsoft/dmtcp/bin/dmtcp_rm_loclaunch
>
> + '[' SLURM = SLURM ']'
>
> + '[' -n /home/localsoft/slurm/soft/bin/srun ']'
>
> ++ dmtcp_discover_rm -n '
>
> :: acme12.ciemat.es :bg:
> /tmp/ckpt_dmtcp_srun_helper_1b69d09fb3238b30-41000-5617e813.dmtcp
> /tmp/ckpt_hydra_pmi_proxy_1b69d09fb3238b30-43000-5617e813.dmtcp
> /tmp/ckpt_mpiexec.hydra_1b69d09fb3238b30-40000-5617e813.dmtcp
> /tmp/ckpt_helloWorldMPI_1b69d09fb3238b30-44000-5617e813.dmtcp
>
> :: acme13.ciemat.es :bg:
> /tmp/ckpt_hydra_pmi_proxy_54385264162a2589-45000-5617e813.dmtcp
> /tmp/ckpt_helloWorldMPI_54385264162a2589-46000-5617e814.dmtcp
>
> '
>
> + eval RES_MANAGER=SLURM 'manager_resources="*acme12:2' '"'
> 'input_config="*acme12.ciemat.es:1' acme13.ciemat.es:1 '"'
> 'DMTCP_DISCOVER_PM_TYPE='\''HYDRA'\''' 'DMTCP_LAUNCH_CKPTS='\'''
> /tmp/ckpt_dmtcp_srun_helper_1b69d09fb3238b30-41000-5617e813.dmtcp
> '/tmp/ckpt_mpiexec.hydra_1b69d09fb3238b30-40000-5617e813.dmtcp'\'''
> DMTCP_REMLAUNCH_NODES=1 DMTCP_REMLAUNCH_0_SLOTS=1
>
> 'DMTCP_REMLAUNCH_0_0='\''/tmp/ckpt_helloWorldMPI_1b69d09fb3238b30-44000-5617e813.dmtcp'
> /tmp/ckpt_hydra_pmi_proxy_1b69d09fb3238b30-43000-5617e813.dmtcp
> /tmp/ckpt_helloWorldMPI_54385264162a2589-46000-5617e814.dmtcp
> /tmp/ckpt_hydra_pmi_proxy_54385264162a2589-45000-5617e813.dmtcp ''\'''
>
> ++ RES_MANAGER=SLURM
>
> ++ manager_resources='*acme12:2 '
>
> ++ input_config='*acme12.ciemat.es:1 acme13.ciemat.es:1 '
>
> ++ DMTCP_DISCOVER_PM_TYPE=HYDRA
>
> ++ DMTCP_LAUNCH_CKPTS='
> /tmp/ckpt_dmtcp_srun_helper_1b69d09fb3238b30-41000-5617e813.dmtcp
> /tmp/ckpt_mpiexec.hydra_1b69d09fb3238b30-40000-5617e813.dmtcp'
>
> ++ DMTCP_REMLAUNCH_NODES=1
>
> ++ DMTCP_REMLAUNCH_0_SLOTS=1
>
> ++
> DMTCP_REMLAUNCH_0_0='/tmp/ckpt_helloWorldMPI_1b69d09fb3238b30-44000-5617e813.dmtcp
> /tmp/ckpt_hydra_pmi_proxy_1b69d09fb3238b30-43000-5617e813.dmtcp
> /tmp/ckpt_helloWorldMPI_54385264162a2589-46000-5617e814.dmtcp
> /tmp/ckpt_hydra_pmi_proxy_54385264162a2589-45000-5617e813.dmtcp '
>
> + '[' -n '' ']'
>
> + export DMTCP_REMLAUNCH_NODES=1
>
> + DMTCP_REMLAUNCH_NODES=1
>
> + bound=0
>
> ++ seq 0 0
>
> + for i in '$(seq 0 $bound)'
>
> + eval 'val=${DMTCP_REMLAUNCH_0_SLOTS}'
>
> ++ val=1
>
> + export DMTCP_REMLAUNCH_0_SLOTS=1
>
> + DMTCP_REMLAUNCH_0_SLOTS=1
>
> + bound2=0
>
> ++ seq 0 0
>
> + for j in '$(seq 0 $bound2)'
>
> + eval 'ckpts=${DMTCP_REMLAUNCH_0_0}'
>
> ++ ckpts='/tmp/ckpt_helloWorldMPI_1b69d09fb3238b30-44000-5617e813.dmtcp
> /tmp/ckpt_hydra_pmi_proxy_1b69d09fb3238b30-43000-5617e813.dmtcp
> /tmp/ckpt_helloWorldMPI_54385264162a2589-46000-5617e814.dmtcp
> /tmp/ckpt_hydra_pmi_proxy_54385264162a2589-45000-5617e813.dmtcp '
>
> + export
> 'DMTCP_REMLAUNCH_0_0=/tmp/ckpt_helloWorldMPI_1b69d09fb3238b30-44000-5617e813.dmtcp
> /tmp/ckpt_hydra_pmi_proxy_1b69d09fb3238b30-43000-5617e813.dmtcp
> /tmp/ckpt_helloWorldMPI_54385264162a2589-46000-5617e814.dmtcp
> /tmp/ckpt_hydra_pmi_proxy_54385264162a2589-45000-5617e813.dmtcp '
>
> +
> DMTCP_REMLAUNCH_0_0='/tmp/ckpt_helloWorldMPI_1b69d09fb3238b30-44000-5617e813.dmtcp
> /tmp/ckpt_hydra_pmi_proxy_1b69d09fb3238b30-43000-5617e813.dmtcp
> /tmp/ckpt_helloWorldMPI_54385264162a2589-46000-5617e814.dmtcp
> /tmp/ckpt_hydra_pmi_proxy_54385264162a2589-45000-5617e813.dmtcp '
>
> + '[' HYDRA = HYDRA ']'
>
> ++ mktemp ./tmp.XXXXXXXXXX
>
> + export DMTCP_SRUN_HELPER_SYNCFILE=./tmp.wCpQ2NYVjA
>
> + DMTCP_SRUN_HELPER_SYNCFILE=./tmp.wCpQ2NYVjA
>
> + rm ./tmp.wCpQ2NYVjA
>
> + dmtcp_srun_helper -r /home/localsoft/slurm/soft/bin/srun
> /home/localsoft/dmtcp/bin/dmtcp_rm_loclaunch
>
> + '[' '!' -f ./tmp.wCpQ2NYVjA ']'
>
> + . ./tmp.wCpQ2NYVjA
>
> ++ export DMTCP_SRUN_HELPER_ADDR=/tmp/srun_helper_usock.ezhOdw
>
> ++ DMTCP_SRUN_HELPER_ADDR=/tmp/srun_helper_usock.ezhOdw
>
> + pass_slurm_helper_contact '
> /tmp/ckpt_dmtcp_srun_helper_1b69d09fb3238b30-41000-5617e813.dmtcp
> /tmp/ckpt_mpiexec.hydra_1b69d09fb3238b30-40000-5617e813.dmtcp'
>
> + LOCAL_FILES='
> /tmp/ckpt_dmtcp_srun_helper_1b69d09fb3238b30-41000-5617e813.dmtcp
> /tmp/ckpt_mpiexec.hydra_1b69d09fb3238b30-40000-5617e813.dmtcp'
>
> + '[' -n '' ']'
>
> + '[' -n '' ']'
>
> ++ whoami
>
> ++ hostname
>
> + CURRENT_TMPDIR=/tmp/dmtcp-sl...@acme12.ciemat.es
>
> + '[' '!' -d /tmp/dmtcp-sl...@acme12.ciemat.es ']'
>
> + for CKPT_FILE in '$LOCAL_FILES'
>
> + SUFFIX=/tmp/ckpt_dmtcp_srun_helper_1b69d09fb3238b30-41000-5617e813
>
> + SLURM_ENV_FILE=/tmp/
> dmtcp-sl...@acme12.ciemat.es/slurm_env_1b69d09fb3238b30-41000-5617e813
>
> + echo DMTCP_SRUN_HELPER_ADDR=/tmp/srun_helper_usock.ezhOdw
>
> + for CKPT_FILE in '$LOCAL_FILES'
>
> + SUFFIX=/tmp/ckpt_mpiexec.hydra_1b69d09fb3238b30-40000-5617e813
>
> + SLURM_ENV_FILE=/tmp/
> dmtcp-sl...@acme12.ciemat.es/slurm_env_1b69d09fb3238b30-40000-5617e813
>
> + echo DMTCP_SRUN_HELPER_ADDR=/tmp/srun_helper_usock.ezhOdw
>
> + rm ./tmp.wCpQ2NYVjA
>
> + dmtcp_restart --join --coord-host acme12.ciemat.es --coord-port 7779
> /tmp/ckpt_dmtcp_srun_helper_1b69d09fb3238b30-41000-5617e813.dmtcp
> /tmp/ckpt_mpiexec.hydra_1b69d09fb3238b30-40000-5617e813.dmtcp
>
> + exit 0
>
>
> '''''''
>
>
>
>
>
> --
> Dr. Manuel Rodríguez-Pascual
> skype: manuel.rodriguez.pascual
> phone: (+34) 913466173 // (+34) 679925108
>
> CIEMAT-Moncloa
> Edificio 22, desp. 1.25
> Avenida Complutense, 40
> 28040- MADRID
> SPAIN
>
>
> ------------------------------------------------------------------------------
> _______________________________________________
> Dmtcp-forum mailing list
> Dmtcp-forum@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/dmtcp-forum
>
------------------------------------------------------------------------------
_______________________________________________
Dmtcp-forum mailing list
Dmtcp-forum@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dmtcp-forum
