Hi Iosif,
The different output-xxxx folders are mostly intended to be used when
recovering from checkpoint files for run that is longer than the queue
time on a machine. However, the recovery from checkpoint files have to be
activated by parameter settings in the parameter file. You can see an
example of that in the Checkpoint/Recovery: section of the binary neutron
star gallery parameter file (see
https://www.einsteintoolkit.org/gallery/bns/bns.par). As you say using
create-submit will create a new simulation and the output from that run
will be in output-0000. If you have checkpoint/recovery activated in your
parameter file (here the IO::recover = "autoprobe" is crucial) if you then
does a simfactory submit, a new run will be started starting from the last
checkpoint file in output-0000 with the output being placed in
output-0001. Another simfactory submit will then create a new run that
starts from the last checkpoint file in output-0001 and produces output in
output-0002.
If you instead want to rerun from the beginning I would rather create a
new simulation either with a new name or deleting the old simulation
first (simfactory has a delete with create-submit.
If you rerun using simfactory submit, you don't have to specify the same
number of processors. Cactus supports recovering on a different number
of processors than the run that created the checkpoint files.
Cactus supports changing some (not all) parameters on recovery. It depends
on whether the parameters are declared (in the param.ccl file) as being
steerable. If you look at the param.ccl file for CarpetIOHDF5 you can see
that the parameters that chooses the output variables all have
STEERABLE = ALWAYS, so yes, you can change what variables are output on
recovery. It's been a while since I have done this, but I suspect you have
to modify your parameter file and then specify --parfile=<your modified
parameter file> when you do the simfactory submit command to restart.
Finally in order to look at the data in from multiple output-xxxx
directories, Cactus itself does nothing to combine the data from different
output directories. However, Kuibit and SimulationTools, have been
developed to handle stuff like this automatically for you. Kuibit is
written in python, while SimulationTools is written in Mathematica.
You can find these at:
https://sbozzolo.github.io/kuibit/
https://simulationtools.org/
I hope this answered your questions.
Cheers,
Peter Diener
Assistant Professor Research
Center for Computation & Technology and
Department of Physics & Astronomy
Louisiana State University
On Thu, 18 Sep 2025, IOSIF PANAGIOTIS wrote:
Hi all,
I am not sure I understand what the output-xxxx folders do under each
individual simulation folder.
Suppose we "create-submit" a new simulation named test.
The results will appear in the output-0000 folder under the folder
"home/username/simulations/test/".
OK, great.
Suppose now we want to re-run that simulation.
And suppose we want to re-run it from scratch and not continue from a
previous checkpoint (e.g. maybe we forgot to save some output we needed).
The skeleton is already created, so we can just use the command submit this
time.
The results will appear in the output-0001 folder.
Is the above understanding correct?
Questions:
* When we submit again the simulation for the re-run, do we have to
specify the same computational resources?
* I.e. is this info something already encoded in the simulation's skeleton
and therefore we should keep it fixed for consistency?
* Does the skeleton of the simulation also fix what quantities are saved
and therefore if we forgot to save some quantity we have to create a new
simulation skeleton?
Additionally, from what I understand, the output-xxxx folders mainly handle
different checkpoints of a larger simulation that could take weeks or
months.
* Does the user have to do anything to combine all the information from
the checkpoints, or is this handled automatically by ETK / SimFactory?
* Is there any advice on best practices regarding this scenario?
Best,
Panayotis
-------------------
Panagiotis Iosif
postdoctoral researcher
Department of Physics, University of Trieste
Via Alfonso Valerio 2, Trieste 34127
Italy
-------------------
_______________________________________________
Users mailing list
[email protected]
http://lists.einsteintoolkit.org/mailman/listinfo/users
