Hello Lorenzo, Unfortunately, Carpet will always write one checkpoint file per MPI rank, there is no way to change that.
As you learned the option out_proc_every only affects the out3D_vars output (and possible out_vars 3D output) but never checkpoints. In my opinion, you should be impossible to stress the file system, of a reasonably provisioned cluster, with the checkpoints. Even when running on 32k MPI ranks (and 4k nodes) on BW, checkpoint-recovery was very quick (1min or so) and barely made a blip on the system monitoring radar. Any cluster with sufficiently many nodes to run at scale at 1 file per rank (for a sane number of ranks ie some OpenMP threads) should have a file system capable of taking checkpoints. Of course 1 rank per core is no longer "sane" once you go beyond a couple hundred cores. Now writing 1 file per output variable and per MPI rank may be a different thing.... In that case out_proc_every should help with out3D_vars. I would also suggest one_file_per_group or even one_file_per_rank for this (see CarpetIOHDF5's param.ccl), which will have less of a performance (no communication) impact than out_proc_every != 1. If the issue is opening many files (again, only for out3D_vars regular output), then you may also see benefits from the different options in: https://bitbucket.org/eschnett/carpet/pull-requests/34 https://bitbucket.org/einsteintoolkit/tickets/issues/2364 Yours, Roland > Hello, > In order to avoid stressing the filesystem on the cluster I'm running on, I > was suggested to avoid writing one output/checkpoint file per MPI process > and instead collecting data from multiple processes before > outputting/checkpointing happens. I found the combination of parameters > > IO::out_mode = "np" > IO::out_proc_every = 8 > > does the job for output files, but I still have one checkpoint file per > process. Is there a similar parameter, or combination of parameters, which > can be used for checkpoint files? > > Thank you very much, > Lorenzo Ennoggi -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://keys.gnupg.net.
pgpi_0xD0z5KP.pgp
Description: OpenPGP digital signature
_______________________________________________ Users mailing list [email protected] http://lists.einsteintoolkit.org/mailman/listinfo/users
