Re: [Users] Implementing a simple 1+1 BSSN in Python
e, Chris Dr Chris Stevens Lecturer in Applied Mathematics Rm 602, Jack Erskine building School of Mathematics and Statistics T: +64 3 369 0396 (Internal 90396) University of Canterbury | Te Whare Wānanga o Waitaha Private Bag 4800, Christchurch 8140, New Zealand http://www.chrisdoesmaths.com<http://www.chrisdoesmaths.com/> Director SCRI Ltd http://www.scri.co.nz<http://www.scri.co.nz/> ___ Users mailing list Users@einsteintoolkit.org<mailto:Users@einsteintoolkit.org> http://lists.einsteintoolkit.org/mailman/listinfo/users -- Erik Schnetter mailto:schnet...@gmail.com>> http://www.perimeterinstitute.ca/personal/eschnetter/ ___ Users mailing list Users@einsteintoolkit.org<mailto:Users@einsteintoolkit.org> http://lists.einsteintoolkit.org/mailman/listinfo/users -- Ian Hinder Research Software Engineer University of Manchester, UK ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] MPI killing issue
Hi, I don't know why the mailing list formatted my email in that way! It has added line breaks where there should not be any. Here is another attempt: I use the parameter TwoPunctures::TP_epsilon= 1e-6 to ensure that the NaN never appears, no matter where I place the BHs. This replaces a denominator sqrt (pow (r2_plus, 2)) with sqrt (pow (r2_plus, 2) + pow (TP_epsilon, 4)) See https://bitbucket.org/einsteintoolkit/einsteininitialdata/src/master/TwoPunctures/src/Equations.c#lines-91 so adds a fixed source of error to your evolution, but the effect of this is negligible. -- Ian Hinder Research Software Engineer University of Manchester, UK ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] MPI killing issue
On 23 Dec 2020, at 13:30, Roland Haas mailto:rh...@illinois.edu>> wrote: Hello Karima, I am not a black hole evolution expert, but I can give this a try. Hi, I use the parameter TwoPunctures::TP_epsilon = 1e-6 to ensure that the NaN never appears, no matter where I place the BHs. This replaces a denominator sqrt (pow (r2_plus, 2)) with sqrt (pow (r2_plus, 2) + pow (TP_epsilon, 4)) See https://bitbucket.org/einsteintoolkit/einsteininitialdata/src/master/TwoPunctures/src/Equations.c#lines-91 so adds a fixed source of error to your evolution, but the effect of this is negligible. -- Ian Hinder Research Software Engineer University of Manchester, UK ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] Meeting MInutes 06-11-2020
On 11 Jun 2020, at 16:28, Hamilton, Maria mailto:bab...@marshall.edu>> wrote: * Peter starts the meeting with the first item on the agenda: replacement of the cacuscode.org<http://cacuscode.org/> web page, and asked Seven to explain the situation. Steven explained that that he wants a resolution to this this issue and will keep adding it to the agenda until it is resolved, and the old web page replaced. Roland mentioned that Ian Hinder worked on a replacement for the web site, and gave the links: https://test.cactuscode.org/ https://github.com/einsteintoolkit/cactuscode.org/issues [https://avatars0.githubusercontent.com/u/6198340?s=400&v=4]<https://github.com/einsteintoolkit/cactuscode.org/issues> EinsteinToolkit/cactuscode.org<https://github.com/einsteintoolkit/cactuscode.org/issues> Holds the content of the CactusCode website. Contribute to EinsteinToolkit/cactuscode.org<http://cactuscode.org/>development by creating an account on GitHub. github.com<http://github.com/> The problem is that there are about 30 or more things that need to be fixed and he asked for volunteers to help pushing things forward. Steven asked if is absolutely necessary to address each on of them or only the essential ones. Hi all, The full list of remaining issues that I identified is here: https://github.com/EinsteinToolkit/cactuscode.org/issues but they are split into "bug" and "enhancement". Loosely speaking, the bugs are issues with the conversion, and the enhancements are things which would make the content better; e.g. to update things which haven't been updated in 10 years. In order to switch, we only need to fix the bugs. There are 11 of these: https://github.com/EinsteinToolkit/cactuscode.org/issues?q=is%3Aopen+is%3Aissue+label%3Abug<https://github.com/EinsteinToolkit/cactuscode.org/issues?q=is:open+is:issue+label:bug> Note that they do not all need to be implemented; we might just decide that some of them are unimportant. Atul Kedia was interested in the location of the the new web page, and if all the information from the old web page will be kept. The new webpage is available to browse at the temporary location: https://test.cactuscode.org It is generated automatically when commits are pushed to the master branch of the source repository here: https://github.com/EinsteinToolkit/cactuscode.org Steven encouraged Atul to take a look and compare the info, warning that the new web page will not have the same functionalities, not being written in php. The idea of the new website is to maintain as much of the existing content from cactuscode.org<http://cactuscode.org> as possible. The reason for the move is that the old website is hosted on a webserver that we have to maintain, whereas the new website is hosted on github pages, which means we don't have to maintain a server. The old website used dynamic PHP scripts on the server for a small amount of functionality, and this is not possible on github pages, where the content is static. Some of the functionality can probably be replicated with client-side scripting or site-build-time configuration. -- Ian Hinder Research Software Engineer University of Manchester, UK ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] Meeting Minutes 2020-06-04 [fixed links]
On 8 Jun 2020, at 14:48, Ian Hinder
mailto:ian.hin...@manchester.ac.uk>> wrote:
On 4 Jun 2020, at 16:39, Bill Gabella
mailto:b.gabe...@vanderbilt.edu>> wrote:
** [ZE] NRPyPN we use two punctures for initial data in ETK. Looking
for low ecc BBH intial data parameter solvers. Wrote one based on
literature. Option in Two Punctures that you sepcify the 8 input params
and outputs the initial data, P_t and P_r. If close to ecc =0, as close
as Post-Newtonian allows. ES-Good enough params or need to iterate?
ZE-Paper by Ramos-Buades et al. https://arxiv.org/abs/1810.00036 . They
say with one iteration can drop ecc to order 1/10^3 . RH-Good to have
the notebook in the utils folder for Two Punctures. ZE-Tedious, but the
C version is not too hard and integrate with Two Punctures. RH-Very low
eccentricity requires man iterations, a glitch shows that ecc energy
goes up. Scheme gives back P_t and P_r. PN in C does iteration zero,
but not later ones. RH&ZE-Iteration greater than 1 is not in NRPyPN but
RH's student has been doing that. RH-I will reach out to principals and
discuss this. Does two orbits and sees what to update, and then
re-submits itself, and want ecc < 1e-6 . First bit is all eccentricity
reduction. ZE-Found typos in the original paper. dE_GW/dt was very
different than other groups use. RH-Recevied their Mathematica
notebook, so hopefully better than the paper.
Some links to NRPyPN, Low-eccentricity Post-Newtonian BBH initial data
parameters (for Two Punctures):
https://nbviewer.jupyter.org/github/zachetienne/nrpytutorial/blob/master/NRPyPN/NRPyPN.ipynb
https://github.com/zachetienne/nrpytutorial/tree/master/NRPyPN (source
codes)
Hi,
I developed an infrastructure for obtaining low-eccentricity parameters for
BBH. It is implemented and freely-available in SimulationTools for
Mathematica. See
https://bitbucket.org/simulationtools/simulationtools/src/master/EccentricityReduction.m.
I haven't used it in a while, but can't think why it wouldn't still work. It
lacks documentation - if someone is interested in using it or developing it
further, I can see if I can find some time to write up some docs.
You can use it to generate an initial guess from PN
(QuasiCircularParametersFromPostNewtonian[{m_, q_, chi1_, chi2_, om_}]) for any
aligned-spin case (you can probably use chi_z for precessing cases). This will
give you the separation, orbital angular momentum (r * py) and radial linear
momentum (px). You then run a simulation with these parameters for a few
orbits, and analyse the results. The method uses the time derivative of the
radial separation for its eccentricity estimator. You use
BinaryEccentricityFromSeparationDerivative, passing it the separation as a
function of time (which you can get from ReadBinarySeparation), and a time
window in which to measure the eccentricity. You can get a suitable window
from EccentricityFitWindow, which calculates it using a simple heuristic from
the initial orbital frequency to give two orbits. You can then use
ReduceEccentricity with the results of
BinaryEccentricityFromSeparationDerivative which gives you updated TwoPunctures
parameters.
There are a number of higher level functions designed to fit this into an
automated workflow. I was using it with SimFactory 3, which supports the idea
of post-simulation scripts and "termination reasons". I had it set up so that
when Cactus terminates during an eccentricity reduction run, it would run a
script
(https://bitbucket.org/simulationtools/simulationtools/src/master/Scripts/AnalyseEccentricity)
which would call SimulationTools to estimate the next separation and radial
momentum (I chose to keep the angular momentum L fixed, since that corresponds
to fixed omega at 0 PN (maybe 1 PN?)). The post-simulation script is
https://bitbucket.org/ianhinder/simfactory3/src/master/simfactory/etc/appdb/scripts/post-simulation,
and it has a number of heuristics for what to do in different cases.
There are a lot of parts to this system, but it used to run extremely well, and
was fully automated, and was used to produce many production simulations,
including the very high spin simulations used in
https://arxiv.org/abs/1810.10585, with chi_z = 0.9 and mass ratios up to 5.
I forgot to mention the crucial part! For those simulations, it's very
challenging to get the eccentricity down; typically I could only get it down to
~1e-3, but that is usually enough. For simple cases, like equal mass,
non-spinning, if you are at a large separation (e.g. 15 orbits), the method was
able to iterate the eccentricity down to something like 1e-6.
--
Ian Hinder
Research Software Engineer
University of Manchester, UK
___
Users mailing list
Users@einsteintoolkit.org
http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] Meeting Minutes 2020-06-04 [fixed links]
ovements to understandability of the graphs. Newest version and try to make the XY plots, should be quite the same and produce nice views of the horizons. And work for the GW graphics. AK-Looking at the BBH gallery and discussion of order of the Gallery page and thinking about for people starting the first time with these examples. Some graphics use Wizard, some SimulationTools, some with VisIt, and have not be able to duplicate some. What is "Wizard"? Leave the animation as is...cannot regenerate yet. Barry Wardell (barry.ward...@ucd.ie<mailto:barry.ward...@ucd.ie>) made the movie; I'm not sure where the scripts for that are; maybe you could contact him? So re-arrange and add text. ES-Would be good to collect the scripts that generate these images. A single "Make" would be nice. RH-Do these changes. ZE&BG-Looks good. BG-Would be nice to also have the steps/scripts for each of the tools generating the graphics in case the user does not have access to them all. RH-Look at http://einsteintoolkit.org/gallery/bns/scripts.tar.gz Look at https://bitbucket.org/einsteintoolkit/www/src/master/gallery/bbh/Makefile. 🙂 That builds all the plots I contributed. The VisIt ones and the movie were not so simple to script, but it might be possible given an investment of time. The VisIt ones were done by Eloisa Bentivenga (eloisa.bentive...@ibm.com<mailto:eloisa.bentive...@ibm.com>). -- Ian Hinder Research Software Engineer University of Manchester, UK ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] Error in exe/cactus_sim
On 22 Apr 2020, at 12:03, Rishank Diwan mailto:rishank2...@gmail.com>> wrote: Hello Ian, I am getting "gcc -I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi -I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/opal/mca/event/libevent2022/libevent -I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/opal/mca/event/libevent2022/libevent/include -I/usr/lib/x86_64-linux-gnu/openmpi/include -pthread -L/usr//lib -L/usr/lib/x86_64-linux-gnu/openmpi/lib -lmpi" as result for "mpicc -show" and "/usr/bin/mpicc" for "which mpicc" which exactly the same as you mentioned. Hmm. Please can you delete the config: rm -rf configs/sim and rebuild, sim build --thornlist= >build.log 2>&1 and send us build.log (e.g. via https://pastebin.com, as it will be quite large). A log of the exact commands you ran would be good as well. Maybe the MPI thorn is getting confused about which MPI to use and trying to build its own. -- Ian Hinder Research Software Engineer University of Manchester, UK ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] Error in exe/cactus_sim
On 21 Apr 2020, at 21:19, Rishank Diwan
mailto:rishank2...@gmail.com>> wrote:
Hello Roland and Ian,
I did check for OpenMPI and MPICH package, it seems there is only one installed
I am attaching the result so you can see.
I also checked the other two command to know the compatibility but they seem to
have different path.
"dpkg -S $(readlink -f $(ldd exe/cactus_sim | gawk
'/libmpi.so/{print$3}'<http://libmpi.so/%7Bprint$3%7D'>)) " gives
"libopenmpi2:amd64: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so.20.10.1" as
output.
"dpkg -S $(readlink -f $(which mpirun))" gives "openmpi-bin: /usr/bin/orterun"
as output.
They are both openmpi, so that doesn't immediately throw up any red flags.
I have tried "simfactory/bin/sim run static_tov --parfile=par/static_tov.par
--procs=2" and "./simfactory/bin/sim create-run helloworld --parfile
arrangements/CactusExamples/HelloWorld" and "exe/cactus_sim" in all the above
cases I got the same error message.
I am also attaching the mpi.log file you asked for.
>From the dpkg output, you have both mpich and openmpi installed.
ii mpich 3.3~a2-4
amd64Implementation of the MPI Message Passing
Interface standard
ii openmpi-bin2.1.1-8
amd64high performance message passing library --
binaries
ii openmpi-common 2.1.1-8
all high performance message passing library --
common files
Can you tell us the output of
which mpicc
It should be /usr/bin/mpicc
Then type
mpicc -show
to see which MPI package it is using. It should be something like
gcc -I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi
-I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/opal/mca/event/libevent2022/libevent
-I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/opal/mca/event/libevent2022/libevent/include
-I/usr/lib/x86_64-linux-gnu/openmpi/include -pthread -L/usr//lib
-L/usr/lib/x86_64-linux-gnu/openmpi/lib -lmpi
so we can see which one is the default? When you build the ET, the MPI thorn
build script runs mpicc to find out which MPI to use.
It's possible that you might have to uninstall mpich, but I'm not sure. There
*shouldn't* be a fundamental reason why you can't have both installed and just
use one of them, but there might be technical reasons why it doesn't work.
--
Ian Hinder
Research Software Engineer
University of Manchester, UK
___
Users mailing list
Users@einsteintoolkit.org
http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] Error in exe/cactus_sim
On 20 Apr 2020, at 15:38, Rishank Diwan mailto:rishank2...@gmail.com>> wrote: Dear Sir/Madam I am trying to compile a Cactus executable on my laptop, which has an ubuntu-based operating system(Ubuntu 18.04). I am using the latest version i.e; ET_2019_10. I am using simfactory and following the Simulation Factory Advanced Tutorial. Hi Rishank, Are you referring to this? https://docs.einsteintoolkit.org/et-docs/Simulation_Factory_Advanced_Tutorial This was last updated in 2018; it might have been superseded by the Jupyter tutorials described at https://einsteintoolkit.org/documentation/new-user-tutorial.html. Roland, can you confirm? For building, I am using generic configuration files which are also the default files. Here are the steps I am using: simfactory/bin/sim setup ./simfactory/bin/sim build -j2 --thornlist ../einsteintoolkit.th<http://einsteintoolkit.th/> Compilation produces an executable, but when I go for running the example file using the command, exe/cactus_sim I get the following error. "" Fatal error in PMPI_Comm_rank: Invalid communicator, error stack: PMPI_Comm_rank(110): MPI_Comm_rank(comm=0x8202540, rank=0x7ffcee11d818) failed PMPI_Comm_rank(68).: Invalid communicator [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=805918213 : system msg for write_line failure : Bad file descriptor "" I am not able to find any relevant solution on the internet. Can you please help me find in the solution. Can you try running the executable via simfactory, as per the tutorial section "Running a simulation" (https://docs.einsteintoolkit.org/et-docs/Simulation_Factory_Advanced_Tutorial#Running_a_Simulation)? simfactory/bin/sim run static_tov --configuration sim-debug --parfile=par/static_tov.par --procs=8 This will hopefully make sure that the MPI used to build the code is used to run it as well (though I have my suspicions that even then it's not guaranteed). You say that you are running the executable with exe/cactus_sim but that is not how the tutorial says to run it. In general, Cactus executables will need certain environment variables set, modules loaded, etc, and to be run with the correct mpirun command. That being said, running the executable directly should work on a ubuntu system like this, but it's better to use sim run. -- Ian Hinder Research Software Engineer University of Manchester, UK ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] Determination of the final spin of the remnant of a binary neutron star merger
On 5 Mar 2020, at 15:42, Erik Schnetter mailto:schnet...@gmail.com>> wrote: Beyhan The QuasiLocalMeasures thorn can examine not only horizons, but also other 2-surfaces. You can set up a surface that is large and which encloses both the remnant and surrounding matter, but which is still inside the emitted gravitational wave train. QuasiLocalMeasures can then calculate the angular momentum contained inside that sphere. I'm not familiar with the method as applied to neutron stars, but for a black hole system, I would probably try to do this by computing the "ADM angular momentum" of the spacetime, as well as the "Bondi angular momentum loss", their difference being the "remaining" angular momentum in the system. I think this is fairly rigorous when done with masses, but I put the quotes around the angular momenta as I don't think these quantities are on as firm a footing. In practice, one *should* be able to compute the "ADM angular momentum" on the initial data slice by evaluating the formula on a set of finite-radius spheres using QuasiLocalMeasures, similar to what Erik mentioned, and then extrapolating to spatial infinity. I don't know if there are reasons why this won't work for neutron star initial data. The "Bondi angular momentum loss" could be calculated by measuring the angular momentum flux in the emitted gravitational waves. This is technically very challenging to get accurate. You need quite a lot of resolution, and wave extraction far enough out that you can cleanly extrapolate it to future null infinity. There are also severe complications due to junk radiation. So this approach is quite hard to implement. -- Ian Hinder Research Software Engineer University of Manchester, UK ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] Documents lack details
On 26 Feb 2020, at 15:09, 刘昊阳 mailto:liuhaoyan...@mails.ucas.edu.cn>> wrote: Dear colleague: I'm Liu Haoyang, a fresher of EinsteinToolkit from University of Chinese Academy of Science. I found that some Thorn's documents in Thorn Guide lack detailed description about their variables' function, such as "Carpet" and "ReflectionSymmetry". And it seems that access to https://carpetcode.org/ is restricted. Is there any other material for such Thorns I can learn about? And how can I have access to the website of Carpet? Hi, The best place to look at an overview of the documentation is probably https://einsteintoolkit.org/documentation/ThornGuide.php It's true that the Carpet thorn doesn't have any documentation, but the Carpet arrangement does. This applies to the whole arrangement of Carpet thorns. You can read it here: https://einsteintoolkit.org/arrangementguide/Carpet/documentation.html I'm not sure what happened to carpetcode.org<http://carpetcode.org>. Erik Schnetter is the original author of Carpet, and his website still has that link (https://www.perimeterinstitute.ca/personal/eschnetter/), so I assume the website has gone down and is unnoticed. I will ask him in a separate thread. Yes, the symmetry thorns don't appear to be documented. Often, you can work out how to use a thorn by looking at the available parameters in params.ccl: https://bitbucket.org/cactuscode/cactusnumerical/src/master/ReflectionSymmetry/param.ccl That, at least, is less likely to be out of date than any documentation! The main idea, if you want to use reflection symmetry, is that you need to set up your domain, e.g. with CoordBase parameters, to contain only the portion you want to simulate. E.g. if you have a symmetry z -> -z, then you could set zmin = 0. You then activate ReflectionSymmetry and set ReflectionSymmetry::reflection_z = "yes", ReflectionSymmetry::avoid_origin_z = "no" (the need for the latter is due to an unfortunate default, from what I remember). -- Ian Hinder Research Software Engineer University of Manchester, UK ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
[Users] carpetcode.org
Hi Erik, It looks like carpetcode.org<http://carpetcode.org> is down. Do you know what the situation is? We can add it to uptimerobot if you like. As I remember, it was hosted on stellarcollapse.org<http://stellarcollapse.org/>. It could also be converted to github pages, if we don't want to continue having to use a self-maintained server. -- Ian Hinder Research Software Engineer University of Manchester, UK ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
[Users] cactuscode.org website
On 11 Feb 2020, at 16:29, Steven R. Brandt mailto:sbra...@cct.lsu.edu>> wrote: OK, I have this imported from SVN: https://github.com/EinsteinToolkit/cactuscode Some of you already have commit rights on it. I'm happy to have more. We apparently have a docker image in https://github.com/stevenrbrandt/et-websites already... and apparently I made it, though I don't remember doing so. Maybe I'm older than I think. What I need is someone to help with the content. Hi Steve, Thanks for importing it! I was just about to ask... An alternative to hosting a web server with a docker image is to use Github Pages. The site is essentially static, and shouldn't need us to maintain a web server. This would remove any dependence on CCT infrastructure apart from the domain name, and remove any associated maintenance. I've made a proof-of-principle attempt at this. - Created a gh-pages branch in https://github.com/EinsteinToolkit/cactuscode on which to do my testing - Wrote a script to convert the PHP files to markdown; could have kept HTML but I prefer markdown as it enforces separation of the content from the presentation and is easier to edit - Ran the script on the current PHP files - Converted the site to use jekyll for the headers, footers, includes, etc - The site is available at https://einsteintoolkit.github.io/cactuscode.org/, but there are lots of assumptions in the files that it is served from "/", whereas here it is served from /cactuscode.org<http://cactuscode.org>. - To fix this, rather than editing all the URLs in the site, I've instead pointed my own domain to it. You can see the result at http://cactuscode.ianhinder.net (https will start working in up to 24 hours, apparently). If we want to go ahead with this method, and if you have access to the cactuscode.org<http://cactuscode.org> domain, maybe you could create a CNAME record pointing test.cactuscode.org<http://test.cactuscode.org> to einsteintoolkit.github.io<http://einsteintoolkit.github.io> so we don't have to use my domain? The automated conversion from PHP to markdown wasn't perfect; we will need to tidy it up a bit. But the idea now is that you can edit the markdown files, push to the repository, and they will automatically appear on the site. You can also preview it locally, jekyll serve if you have jekyll installed, or through docker, if you have docker installed: docker run --rm -p 4000:4000 --volume="$PWD:/srv/jekyll" -it jekyll/jekyll jekyll serve Visit http://localhost:4000 in either case to see the site. If this sounds like a good way to go forward, we can start working through the files and fixing up the php to markdown conversion, deleting old content, etc. -- Ian Hinder Research Software Engineer University of Manchester, UK ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] meeting minutes for 2020-02-06
On 10 Feb 2020, at 16:37, Steven R. Brandt mailto:sbra...@cct.lsu.edu>> wrote: So the question is, what do we do about the website? We don't have the time, person-power, or budget to maintain it. At present, the website is full of things that are horribly out of date and likely does more harm than good for anyone reading it. Hi Steve, For time, person power and budget, how would this change if it were moved to be under the ET website? Is the issue related to keeping the content up-to-date, or infrastructure issues like maintaining a separate webserver etc? Deleting content which is out of date would help. We could make a website that's just a few well-chosen pages with a similar template to the ETK. Reducing the amount of administrative duplication is certainly desirable. -- Ian Hinder Research Software Engineer University of Manchester, UK ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] meeting minutes for 2020-02-06
On 6 Feb 2020, at 16:03, Erik Schnetter mailto:schnet...@cct.lsu.edu>> wrote: On Thu, Feb 6, 2020 at 10:47 AM Roland Haas mailto:rh...@illinois.edu>> wrote: * Are we ready to turn off CactusCode.org<http://CactusCode.org>? ** all in favor for moving I notice that this topic has been mentioned in the notes on the Einstein Toolkit mailing list, but has not been announced (in an email of its own), nor has it been mentioned on the Cactus mailing lists at all. I understand that most of the active development these days is happening on the context of the Einstein Toolkit, and that maintaining the cactuscode.org<http://cactuscode.org> domain web server seems like a burden, but Cactus is usable by itself, and publicly stating that Cactus is subsumed by the Einstein Toolkit greatly reduces the impact of the Toolkit on the computational science side. This probably won't affect physics funding, but this will make it more difficult to obtain funding from non-astrophysics sources. For example, CISE might ask why they should fund an astrophysics-only project where the maintainers decided deliberately to restrict the target audience of their software. I agree. For some of the non-NR projects I am now working on, I feel like I'm constantly missing and reinventing things that Cactus provides. Some of these projects might benefit from being Cactus thorns. Already, selling Cactus to people might be a little difficult, due to the learning curve and the extra "baggage" that you need to learn. This wouldn't be helped by Cactus being perceived as "a relativity code". -- Ian Hinder Research Software Engineer University of Manchester, UK ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] [ET Trac] #2064: Add GiRaFFE to the Einstein Toolkit
On 4 Nov 2019, at 19:10, Zach Etienne mailto:zache...@gmail.com>> wrote: Thanks for this very nice summary, Roland. Note that all of these rely on the client to do the right thing. With bitbucket you cannot use the pre-receive hooks that git offers to reject commits that contain output (see... Cool. I have heard of folks adding black (https://github.com/psf/black) hooks to git to ensure that Python codes are consistently formatted at every git push. I want NRPy+ development to be as kind as possible to the developer, while maintaining rigorous standards for documentation, modularity, validation tests, CI, etc. Maybe we can discuss further in a future ETK call or offline. One approach would be for the CI pipeline to check all the things you want it to check, and flag the problem (e.g. output committed in notebooks when you don't want it to be). This would be displayed on a web page along with the CI results, but wouldn't "break the build" or be rejected. So people can commit freely, but the problems are tracked, and can be cleaned up later. This is similar to how the compiler warnings are handled in Jenkins for the ET. -- Ian Hinder Research Software Engineer University of Manchester, UK ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] Missing file for configuring build ubuntu.cfg
On 1 Nov 2019, at 17:56, Steven R. Brandt mailto:sbra...@cct.lsu.edu>> wrote: You are using an old version of the einsteintoolkit. Newer versions do not have ubuntu.cfg. All flavors of linux should compile with generic.cfg. Hi, I think Steve means that you are using an old tutorial / instructions. The old instructions will tell you to use ubuntu.cfg, which will work with old versions of the toolkit. Newer versions of the toolkit do not need this (and do not provide it). Please make sure "mpicc" is in your PATH before you compile. Thanks. --Steve On 10/31/2019 6:33 PM, David N Bradley wrote: Hello All: I am in process of installing the ET system, and can not find the file ubuntu.cfg needed for configuring for Mint. ./simfactory/bin/sim setup-silent --optionlist=ubuntu.cfg --runscript debian.sh recently reformated and installed mint 19.1 and lost my old setups. David ___ Users mailing list Users@einsteintoolkit.org<mailto:Users@einsteintoolkit.org> http://lists.einsteintoolkit.org/mailman/listinfo/users ___ Users mailing list Users@einsteintoolkit.org<mailto:Users@einsteintoolkit.org> http://lists.einsteintoolkit.org/mailman/listinfo/users -- Ian Hinder Research Software Engineer University of Manchester, UK ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] Loop over the grid points of the base grid
On 25 Oct 2019, at 09:23, Erik Rodrígo Jiménez Vázquez
mailto:erj...@ciencias.unam.mx>> wrote:
Dear all
I want to subtract an array at t-dt and t times only at the base grid
(refinement level 0), following
http://lists.einsteintoolkit.org/pipermail/users/2013-May/003004.html, I
implemented the suggestion in my code as
BEGIN_GLOBAL_MODE(cctkGH) {
ENTER_LEVEL_MODE(cctkGH, 0) {
for(int k=0;khttps://einsteintoolkit.org/usersguide/UsersGuidech9.html, search for
cctk_levfac):
cctk_levfac
An array of cctk_dim integer factors by which the local grid is refined in the
corresponding direction with respect to the base grid.
--
Ian Hinder
Research Software Engineer
University of Manchester, UK
___
Users mailing list
Users@einsteintoolkit.org
http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] Output file for one specific grid point
On 15 Oct 2019, at 22:05, Haas, Roland mailto:rh...@illinois.edu>> wrote: Hello Severin, However if I want to choose a point inside the spherical part (outside of the cartesian grid) I can not gather any data. I assume that the output thorn is only aware of the cartesian coordinates? Do you (or anyone else) know if there is a way to bypass this issue? The output thorn is CarpetIOASCII, though I am not familiar enough with it to know exactly how it finds out which point to output. It may well not be aware of Llama (it would then use the "local" coordinates instead of the "global", Cartesian ones). You could write some C code to use Cactus's interpolator to get interpolate the data. If the coordinate given is the location of an actual grid point then there is (effectively) no interpolation. For an example on how to use it, please see the Cactus docs: https://www.einsteintoolkit.org/referencemanual/ReferenceManualch2.html#x4-11A2 You could (ab)use CactusNumerical's InterpToArray thorn which lets you do the same without writing code. For an example use, please see its tests though for your use you will want to use the "scalar_vars" etc parameters instead of "array1d_vars". Hi, You can use CarpetIOASCII and CarpetIOHDF5 to do what you want, more or less. Each of the 6 angular patches will be output in separate files. Each one has a mapping x,y,z->rho,sigma,r, where rho and sigma are two of the angular coordinates (different for each patch), and r is *always* the radial coordinate. So, if you want to output a line in the radial direction, you need to use CarpetIOASCII::out1d_vars = "myvar" CarpetIOASCII::out1d_z = yes I believe this is enough. You will get the angular coordinates (0,0) for each patch, corresponding to the +x, -x, +y, -y and +z, -z axes, and the "z" coordinate in the 1D output file will be the radial direction. This *might* depend on the value of n_angular; the angular coordinates chosen will come from out1D_zline_x and out1D_zline_y, which default to 0. If the coordinates are such that 0 is between two grid points, then you might not get any data. This will depend on whether n_angular is even or odd, but I don't remember the details from memory. It's also possible that CarpetIOASCII will give you the nearest point in that case; I'm not sure. You can do the same trick for output at a constant radius, either 1D or 2D, BUT you have to watch out for the fact that the radius will be chosen as, eg. for 2D output, the value of CarpetIOASCII::out2D_xyplane_z, which defaults to 0, where there is no data since the angular patches start at a finite radius r > 0. So if you want a 2D surface at r = 100, you need to set CarpetIOASCII::out2D_xyplane_z = 100. I believe the same works with HDF5. It should also work for a single point; 0d output; see the parameters in CarpetIOASCII/param.ccl. Just remember that the radial coordinate will be called "z"! -- Ian Hinder Research Software Engineer University of Manchester, UK ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] Question about GW150914 and other simulations
On 18 Sep 2019, at 04:14, 黎旭翔 mailto:jukco...@pku.edu.cn>> wrote: Dear friends, We are trying to use those examples listed on the website to test if it works on our HPC. https://einsteintoolkit.org/gallery.html It goes well with a simple tov equation, which takes 2-5 min. But when we use it to simulate GW150914 BH merger or solve tov equation with high precision and long time, it seems that it will stop at an iteration point without any further output, even an error. What really confuses us is that for different tests it stops at different points. Could you please help us find out what goes wrong with the simulation? I will attach the log.txt and parameter files to this mail. Thanks for your time! Here we used a partition with 144 nodes. Sometimes with a specific —procs and —num-threads number in the shell file, the simulation finished successfully. In other time it came across the problem above. In the two neutron star output files, the job stoped at two different iteration points, and was cancelled due to time out or by hand. Hi, Can you tell us the simfactory command line that you used to submit the simulation? From the log file, it looks like it might be wrong, or the machine might not be set up correctly in simfactory. [LOG:2019-06-01 16:07:23] restart.userRun(simulationName)::numprocs= 8 [LOG:2019-06-01 16:07:23] restart.userRun(simulationName)::nodeprocs = 8 [LOG:2019-06-01 16:07:23] restart.userRun(simulationName)::numthreads = 18 [LOG:2019-06-01 16:07:23] restart.userRun(simulationName)::hostname= b01.hpc.pku.edu.cn<http://b01.hpc.pku.edu.cn> [LOG:2019-06-01 16:07:23] restart.userRun(simulationName)::ppn = 144 [LOG:2019-06-01 16:07:23] restart.userRun(simulationName)::ppnused = 144 [LOG:2019-06-01 16:07:23] restart.userRun(simulationName)::procsrequested = 144 [LOG:2019-06-01 16:07:23] restart.userRun(simulationName)::pbsSimulationName= GW150914- [LOG:2019-06-01 16:07:23] restart.userRun(simulationName)::cpufreq = [LOG:2019-06-01 16:07:23] restart.userRun(simulationName)::user= 1801110076 [LOG:2019-06-01 16:07:23] restart.userRun(simulationName)::memory = 0 [LOG:2019-06-01 16:07:23] restart.userRun(simulationName)::nodes = 1 [LOG:2019-06-01 16:07:23] restart.userRun(simulationName)::procs = 144 [LOG:2019-06-01 16:07:23] restart.userRun(simulationName)::numsmt = 1 In particular, ppn = 144 looks wrong. Erik, can you confirm? If it's trying to run on too few nodes, it will run out of memory, as Steve suggested. -- Ian Hinder Research Software Engineer University of Manchester, UK ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] Norm2 calculation on different grid levels
On 15 Jun 2019, at 18:10, Haas, Roland mailto:rh...@illinois.edu>> wrote: Hello Ian, Perhaps a mask would be of use here? You mean: being able to pass a mask to the reduce function in Cactus? Would be nice, unfortunately as far as I know, the "old" reduction interface using CCTK_Reduce (http://einsteintoolkit.org/usersguide/UsersGuidech9.html#verbatim-64) which Carpet uses does not let you pass in a param_table_handle the way that CCTK_ReduceGridArrays (http://einsteintoolkit.org/referencemanual/ReferenceManualch2.html#x4-165000A2) does. So one would first have to convert Carpet to support the "new" reduction interface (which is probably simple as the differences seem minor to me). I meant to use CarpetMask; isn't this a way to mask points from reductions? -- Ian Hinder Research Software Engineer University of Manchester, UK ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] Norm2 calculation on different grid levels
On 15 Jun 2019, at 17:22, Haas, Roland mailto:rh...@illinois.edu>> wrote: The alternative is to introduce a helper grid function where you manually zero out the regions you do not care about then compute the norm of that grid function. Perhaps a mask would be of use here? -- Ian Hinder Research Software Engineer University of Manchester, UK ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
[Users] Jenkins build and test system
Hi, Unfortunately, I have reason to believe that the ET Jenkins build and test system has again been compromised by cryptocurrency mining malware, so it has been taken offline. We have backups from before the intrusion, but need to identify the security flaw which allowed this before restoring the last good backup, changing all security tokens, and bringing the system back online. -- Ian Hinder Research Software Engineer University of Manchester, UK ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] coupling Llama to an evolution code
On 10 May 2019, at 10:22, Miguel Zilhão mailto:miguel.zilhao.nogue...@tecnico.ulisboa.pt>> wrote: hi Ian, yes, i guess the CartesianCoordinate thorn could work. this is not part of the ET, though, is it? Hi Miguel, You're right; I hadn't realised this. It's in the CTGamma arrangement: https://bitbucket.org/llamacode/ctgamma/src/master/ but is not part of the toolkit. Just clone CTGamma into arrangements and add CTGamma/CartesianCoordinates to the thornlist and recompile. -- Ian Hinder Research Software Engineer University of Manchester, UK ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] coupling Llama to an evolution code
Hi Miguel,
There is also the CartesianCoordinate thorn, which implements "Coordinates",
but which just provides a Cartesian grid. Would this be sufficient?
On 9 May 2019, at 00:31, Haas, Roland
mailto:rh...@illinois.edu>> wrote:
Hello Miguel,
There may be two issues:
1. You do not have to inherit from Coordinates to use functions
provided by Coordinates.
2. You do however need to inherit to get easy access to the Jacobian
though. To avoid this you need to call the (low level) function
CCTK_VarDataPtr to get access to the Jacobian at runtime depending on
whether you want to use it. Then when you need an if statement based on
this decision to apply / not apply the Jacobian. This then is a bit
more complex.
Basically (this is inspired by McLachlan):
const CCTK_REAL *J11 = use_jacobian ?
CCTK_VarDataPtr(cctkGH, 0, "Coordinates::J11") : NULL;
for(ijk) {
CCTK_REAL phi_x = phi[i+1] - phi[i-1];
CCTK_REAL phi_y = phi[j+1] - phi[j-1];
if (use_jacobian) {
// apply jacobian to derivatives (or so)
CCTK_REAL Jac_phi_x = J11 * phi_x + J12 * phi_y;
CCTK_REAL Jac_phi_y = J12 * phi_x + J22 * phi_y;
phi_x = Jac_phi_x;
phi_y = Jac_phi_y;
}
}
Yours,
Roland
hi again,
i have a follow-up question regarding this. i'm following Roland's
implementation of the WaveToy code with Llama, and i'm running into the
following issue.
when i inherit the Coordinates thorn, the function
MultiPatch_GetDomainSpecification becomes aliased, and this becomes a problem
if i want to use the same thorn and *not* use Llama. in order words, when adding
inherits: Coordinates
to a thorn's interface.ccl file, one then needs to activate the Coordinates
thorn in the parfile upon running the code whether or not one wants to use
Llama. but then, if multipatch is not used (ie, with
Carpet::domain_from_coordbase = yes), the following error occurs:
void Carpet::get_domain_specification(const cGH*, int, const ivect&,
CarpetLib::rvect&, CarpetLib::rvect&, CarpetLib::rvect&): Assertion `not
CCTK_IsFunctionAliased("MultiPatch_GetDomainSpecification")' failed.
is there a simple way of having a Llama-aware thorn which can also run without
multipatch if so desired?
i've found a previous discussion with a similar issue
(http://lists.einsteintoolkit.org/pipermail/users/2015-December/004656.html)
when using CTGamma, where the suggestion was to activate the thorn
CTGamma/CartesianCoordinates when not using multipatch. i'm guessing that this
thorn provides all the grid functions that Coordinates provides?
is this then the only solution, ie, creating a helper thorn with a "trivial"
Coordinates implementation?
thanks,
Miguel
On 22/04/19 21:45, Miguel Zilhão wrote:
thanks Roland!
this should be enough to get me started. i'll report back if i run into any
difficulty.
cheers,
Miguel
On 22/04/19 13:32, Haas, Roland wrote:
Hello Miguel,
I gave a tutorial on this (for a WaveToy code) at the NCSA ET meeting:
https://drive.google.com/open?id=0B4gNfWainf-5dGcxQzNuOUtEUFk
The code is (likely, given its name) in the the "rhaas/llama" branch of
the cactusexample repo:
cd repos/cactusexamples
git checkout rhaas/llama
should get them for you.
Yours,
Roland
hi all,
i have a few evolution codes that i would like to make Llama-aware. one of them
would be the
LeanBSSNMoL thorn, that was included in the latest ET release.
is there a canonical procedure to do this, or any documentation that i should
follow? i understand
that the main thing to change are the finite differencing operations... is
there a standard way of
performing this change? or anything else i should be aware of?
thanks,
Miguel
___
Users mailing list
Users@einsteintoolkit.org
http://lists.einsteintoolkit.org/mailman/listinfo/users
___
Users mailing list
Users@einsteintoolkit.org<mailto:Users@einsteintoolkit.org>
http://lists.einsteintoolkit.org/mailman/listinfo/users
--
My email is as private as my paper mail. I therefore support encrypting
and signing email messages. Get my PGP key from http://pgp.mit.edu .
___
Users mailing list
Users@einsteintoolkit.org<mailto:Users@einsteintoolkit.org>
http://lists.einsteintoolkit.org/mailman/listinfo/users
--
Ian Hinder
Research Software Engineer
University of Manchester, UK
___
Users mailing list
Users@einsteintoolkit.org
http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] BSSN, 8th order accurate -- how much dissipation?
On 8 Apr 2019, at 18:03, Erik Schnetter mailto:eschnet...@perimeterinstitute.ca>> wrote: I am trying to set up a binary black hole simulation that uses 8th order accurate finite differencing. (I need the black holes to be highly resolved, and I don't need the wave zone, thus I am not use AMR.) On the other hand, I am using Cartoon2D, and I extended it to support 8th order interpolation. My problem is that I cannot get the simulation started well. After a few (about 70) time steps, the code aborts due to nans. Of course, having modified thorn Cartoon2D, I cannot exclude that I made an error there. On the other hand, I did find some pre-existing errors that I corrected. I wonder: What are good settings for numerical dissipation in this case (ML_BSSN with fdOrder=8)? I see a sample parameter file that uses epsDiss = 0.1; should I expect this to work? Would puncture initial conditions need any kind of special treatment? (I'm using TwoPunctures.) Hi Erik, I did experiments with this a while ago with a production BBH simulation (multipatch), and the limit seemed to be about 0.08 in my tests, and I use 0.05 for safety. I don't have access to the data right now (though can dig it out if you want), but I think it was for a Courant factor of 0.45, though it might have been 0.5. It is strongly dependent on the Courant factor. You can probably get away with larger dissipation strengths for a short time or for low resolution, or if you don't have very sharp features (e.g. junk radiation from high spin or high q configurations), but you probably don't want to rely on this. For reference, the parameters I use are below. With Cartoon, I think you will have an effective very small grid spacing for small r, which means that to maintain the Courant condition, you would need a smaller timestep. Is that right? I've never used Cartoon, but it seems like this is something that should be taken into account. Puncture initial conditions shouldn't need any special treatment. Without Cartoon, a resolution of something like 20 grid cells across the radius of the settled AH (which could be a factor of 2 larger than the initial radius) is probably a good minimum to use. # Spatial finite differencing SummationByParts::order = 8 # Drop order instead of using upwinded stencils, only for advection derivatives SummationByParts::sbp_upwind_deriv = no SummationByParts::sbp_1st_deriv = yes SummationByParts::sbp_2nd_deriv = no SummationByParts::onesided_interpatch_boundaries = no SummationByParts::onesided_outer_boundaries = yes SummationByParts::use_dissipation= no GlobalDerivative::use_dissipation= yes SummationByParts::scale_with_h = yes SummationByParts::dissipation_type = "Kreiss-Oliger" # Stability limit seems to be about 0.08 SummationByParts::epsdis = 0.05 # Variables for dissipation SummationByParts::vars = " ML_BSSN::ML_log_confac ML_BSSN::ML_metric ML_BSSN::ML_trace_curv ML_BSSN::ML_curv ML_BSSN::ML_Gamma ML_BSSN::ML_lapse ML_BSSN::ML_shift ML_BSSN::ML_dtlapse ML_BSSN::ML_dtshift " -- Ian Hinder Research Software Engineer University of Manchester, UK ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
[Users] Multiple emails from bitbucket issues
Hi, I've noticed that I'm getting two copies of some BitBucket issue notifications. The first is from t...@einsteintoolkit.org<mailto:t...@einsteintoolkit.org> because I am subscribed to that mailing list, and all the issue notifications go to it. The second is directly from BitBucket, and it is sent because I have interacted with an issue, so BitBucket assumes I am interested and "watches" the issue for me. I think the solution is to unsubscribe from t...@einsteintoolkit.org<mailto:t...@einsteintoolkit.org> and instead enable issue notifications from the tickets repository in bitbucket. Given this situation, is there still value in having trac@einsteintoolkit,org? I would imagine that people who want to see *all* ticket activity from the ET project will likely have a bitbucket account so they can interact with the code, and hence can enable direct bitbucket notifications. -- Ian Hinder Research Software Engineer University of Manchester, UK ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] weird behavior with some ET simulations
On 20 Mar 2019, at 07:52, Antoni Ramos Buades mailto:antoniramosbua...@gmail.com>> wrote: Hi, setting the parameter CarpetRegrid2::min_fraction = 1 solves the problem of the noise outburst ( I attach the file waveformMinfraction1.png that you can compare to the older waveforms.png where the noise is present). We have checked the problem with the new version of the EinsteinToolkit which is in the website and with version of 2017 which we were using and the noise disappeared in both. In terms of speed, the run with min_fraction=1 is slightly slower with respect to the older with min_fraction=0.9, but not significantly. I would like to thank again everybody for the quick feedback and help. Best Regards, Toni. Hi Toni, Good to hear! -- Ian Hinder Research Software Engineer University of Manchester, UK ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] einsteintoolkit docker organization
On 15 Mar 2019, at 03:51, Erik Schnetter mailto:schnet...@cct.lsu.edu>> wrote: Roland Yes, I own this organization. This hosts an image I created several years ago. See also <https://github.com/eschnett/einsteintoolkit-docker>. Unfortunately, I did not manage to find a good way to have the Cactus executable and simulation result be placed outside the container. I will be happy to transfer ownership to you, or to create a team, etc. Please advise. Ian Hinder mentioned to me that he also created an image, but I assume he did not upload it. It was very large, so I didn't upload it. Steve Brandt also did this. It's not clear exactly how one would use Docker with the ET. I think I set up three different images, inheriting from each other. The first was just the dependencies, the second included the code, and the third included the build. Which one you choose to use would be based on what you needed to be able to do. e.g. for a workshop, maybe you want it already built. For a development environment, maybe you want just the dependencies. For storing the simulation results, one would presumably use a volume. I didn't think of storing the executable outside the container; would this be the whole built config, or just the exe? You could do that with a volume as well, but I'm not sure of the purpose? I have two jenkins containers at https://hub.docker.com/u/ianhinder. – The et-jenkins-slave one is actively used by the build system. – jenkins-einsteintoolkit is for a standalone environment; it gives you a full working jenkins system (without needing separate master/slaves) with all the packages installed. I haven't tested this recently though. Last updated 4 years ago, so I doubt it will work :) -- Ian Hinder Research Software Engineer University of Manchester, UK ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] weird behavior with some ET simulations
On 7 Mar 2019, at 14:50, Antoni Ramos Buades mailto:antoniramosbua...@gmail.com>> wrote: Thanks a lot Zach, Roland and Ian for your comments. We are currently compiling the last version of the EinsteinToolkit and we will make some runs with different values of CarpetRegrid2::min_fraction to check if the noise disappears. We will report you the results in a few days. Hi Toni, I suggest to change just one thing at a time; i.e. take the old version of the code that you ran with before, change just this one parameter, and rerun the simulation. If you change too many things at the same time (and you are thinking of making 2 years worth of changes!), the results might be different for many more reasons, and it's hard to reason about the problem. -- Ian Hinder Research Software Engineer University of Manchester, UK ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] weird behavior with some ET simulations
On 7 Mar 2019, at 07:40, Antoni Ramos Buades
mailto:antoniramosbua...@gmail.com>> wrote:
Hi,
Roland you are right it is forming and destroying a common box each time the
outburst of noise happens. Do you know how one could get rid of this noise, or
if there are some parameters of Carpet which one could add to the parameter
file to avoid such a behaviour?
Hi Antoni,
There is this parameter in CarpetRegrid2:
CCTK_REAL min_fraction "Minimum fraction of required refined points that need
to be present in a refined region" STEERABLE=always
{
0:* :: ""
} 0.9
When two boxes overlap, CarpetRegrid2 has to make a decision about whether to
use the union of the two boxes, or to replace the two boxes with a single
enclosing box. This code is in
carpet/CarpetRegrid2/src/property.cc<http://property.cc> in the function
combine_regions::test_impl. The decision is made based on the number of point
in the union of the two boxes, vs the number of points in a single enclosing
box. It will leave the regions separate if
min_fraction * combined_size > regions_size
which, with the default of 0.9, means that the number of points in the single
enclosing region is greater than 1.11 times the number of points in the
original regions. The logic for this is that having lots of regions means lots
of faces, edges and corners, and more communication and prolongation, which can
affect performance and might also be undesirable due to creating numerical
error features such as reflections. On the other hand, the single enclosing
region will contain more points, and hence will be more expensive to evolve.
min_fraction allows you to decide how many extra points you are willing to
evolve, for the sake of having only a single box. If there would be more than
1/min_fraction times the number of points by combining, the code does not
combine.
If you set
CarpetRegrid2::min_fraction = 1
then Carpet will never create a single enclosing box, but will always give you
a box based on the union of the points in the two boxes.
Ninja'd by Roland. Sigh...
--
Ian Hinder
Research Software Engineer
University of Manchester, UK
___
Users mailing list
Users@einsteintoolkit.org
http://lists.einsteintoolkit.org/mailman/listinfo/users
[Users] Change of job and email address
Hi all, Sorry for the long absence from the list! In November, I started a new job at University of Manchester where I am now employed as a Research Software Engineer, working on a variety of research computing projects across the university, mostly focused on HPC. I will try to keep up with the ET a little, though obviously not as much as before. Please update your address books! ian.hin...@manchester.ac.uk<mailto:ian.hin...@manchester.ac.uk>. My AEI email address will stop working at the end of February. -- Ian Hinder Research Software Engineer University of Manchester, UK ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
[Users] Testing example parameter files
On 21 Feb 2019, at 15:42, Haas, Roland mailto:rh...@illinois.edu>> wrote: Stale example parfiles * suggestion to at least run the example parfiles through cactus_sim -S to test that the parfile are at least parsable * could be made part of the jenkins tests Hi, I did a little bit of work on testing the example parameter files at one of the ET workshops. There are notes at https://docs.einsteintoolkit.org/et-docs/Fixing_examples See also https://bitbucket.org/einsteintoolkit/tickets/issues/641/parameter-files-and-thornlists-could-be, where this is discussed. Unfortunately, the web page with the results that this page points to no longer exists. I have looked briefly for the script I wrote to do this test, and sadly cannot find it. Probably the suggestion to do this in Jenkins is a good one. Most of the parameter files don't need much memory or CPU time. We probably would need to blacklist those that do. PS: there are broken links to trac tickets on the wiki. It would be good if trac.einsteintoolkit.org<http://trac.einsteintoolkit.org> redirected a link such as https://trac.einsteintoolkit.org/ticket/641 to the relevant BitBucket ticket. -- Ian Hinder Research Software Engineer University of Manchester, UK ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] environmental variables in Cactus configuration files
> On 21 Nov 2018, at 14:45, Miguel Zilhão
> wrote:
>
> hi all,
>
> i'm compiling ET on a local cluster that uses the module system. on this
> system, once one does
> "module load <...>", the respective path is added to an environmental
> variable. for instance, doing
>
> $ module load HDF5
>
> sets the environmental variable $EBROOTHDF5:
>
> $ echo $EBROOTHDF5
> /home/share/easybuild/software/HDF5/1.8.20-GCC-7.3.0-2.30-generic
>
> so i was trying to use these in my configuration file, by adding the lines
>
> HDF5_DIR = $EBROOTHDF5
> HDF5_LIB_DIRS = ${EBROOTHDF5}/lib
> HDF5_INC_DIRS = ${EBROOTHDF5}/include
>
> etc, to it. however, these environmental variables don't seem to be correctly
> expanded, as i get
> things like the following:
>
> Running configuration script for thorn HDF5:
> Additional requested language support: Fortran
> WARNING in HDF5 configuration:
> None of H5pubconf.h H5pubconf-64.h H5pubconf-32.h found in ${EBROOTHDF5}/lib
> ${EBROOTHDF5}/include
> Automatic detection of szip/zlib compression not possible
> Finished running configuration script for thorn HDF5.
>
> but if i look into the folder ${EBROOTHDF5}/include, these files are clearly
> there. the compilation
> later fails because of this.
>
> when i specify the full path explicitly in the config file:
>
> HDF5_INC_DIRS =
> /home/share/easybuild/software/HDF5/1.8.20-GCC-7.3.0-2.30-generic/include
>
> i get no such warnings, and the code proceeds to compile just fine. is this
> the expected behaviour?
> shouldn't the environmental variables be correctly expanded? if not, what
> would be the typical
> procedure to compile the code on systems with such module tools?
Hi Miguel,
No, environment variables are not expanded in that way, because optionlists are
not evaluated by a shell. Yes, it would be nice if they were, however :) You
can do it in parameter files, but not in optionlists. Usually, in this case,
we either set the paths explicitly, resulting in a multitude of
almost-identical optionlists, or let the external library automatically detect
the location by finding an executable on the path (MPI does this, but I don't
know if any others do).
I've never used it, but it looks like simfactory provides a solution. In
simsubs.py SubAll, it looks like you can use @ENV(EBROOTHDF5)@. Let us know
how you get on with that.
--
Ian Hinder
https://ianhinder.net
___
Users mailing list
Users@einsteintoolkit.org
http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] Question about GW150914.rpar simulation
ration is not correct. Please could you post the standard output file (GW150914_*.out) to https://pastebin.com so we can take a look at it? > We believe that GW150914.rpar EinsteinToolKit is a great use case to test > OpenShift for HPC, and of course we will reference to EinsteinToolKit is our > final report as a use case for Openshift in HPC mode. Great; it sounds interesting! There are instructions for the papers which should be cited if you use this parameter file and code at the top of the parameter file: # Copyright Barry Wardell, Ian Hinder, Eloisa Bentivegna # We ask that if you make use of the parameter file or the example # data, then please cite # Simulation of GW150914 binary black hole merger using the # Einstein Toolkit - https://doi.org/10.5281/zenodo.155394 # as well as the Einstein Toolkit, the Llama multi-block # infrastructure, the Carpet mesh-refinement driver, the apparent # horizon finder AHFinderDirect, the TwoPunctures initial data code, # QuasiLocalMeasures, Cactus, and the McLachlan spacetime evolution # code, the Kranc code generation package, and the Simulation Factory. # An appropriate bibtex file, etgw150914.bib, is provided with this # parameter file. and the bibtex file is at https://einsteintoolkit.org/gallery/bbh/etgw150914.bib. -- Ian Hinder https://ianhinder.net ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] HDF5 data analysis
> On 15 Oct 2018, at 11:15, 林家暉 wrote: > > Hello, > I am trying to analyze 2D output data (.h5) from ET. I want to get a 2D array > with uniform grid.However HDF5 with AMR contains many layers and each layer > breaks into many components. When I read out the HDF5 data , I do not know > where each component should be. Is there some suggested way to map HDF5 data > into a 2D array ? Hi, What software are you using to analyse the data? There are frameworks for Mathematica (simulationtools.org) and Python (https://bitbucket.org/DrWhat/pycactuset). If you want visualisation rather than more in-depth analysis, then you could try VisIt, which has a reader for ET data. See https://docs.einsteintoolkit.org/et-docs/Analysis_and_post-processing for a list of analysis tools that we know about. If you want to do it manually (and I wouldn't recommend reinventing the wheel), all the information you need to place the individual component in coordinate space is listed in attributes of the datasets. Look at the "origin" attribute for the coordinates of the origin of the component, and the "delta" attribute for the grid spacing. -- Ian Hinder https://ianhinder.net ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] Issue running the default qc0-mclachlan.par
> On 28 Sep 2018, at 18:48, Gomard-Henshaw, Chad wrote: > > Hello, > > When running the default qc0 simulation, I get an error (see attached). This > was run using the following command in the windows linux subshell: > > ./simfactory/bin/sim create-run qc05 \ > --parfile=par/qc0-mclachlan.par > > > The simulation runs for about an hour before aborting; I get partial output > files but only with two data points. Can you please advise on how to address > this issue? Hi, We should have a FAQ... You need to run on at least two processes, due to internal limitations in the code. So add --procs 2 to your create-run command line. [I don't know exactly how your machine is configured in simfactory; if it is configured to use more than one thread per process, then you need to use enough "--procs" (which really means "threads") that at least two MPI processes are used.] -- Ian Hinder https://ianhinder.net ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] Binary Black Hole Gallery Example
> On 27 Aug 2018, at 16:08, Cupp, Samuel wrote: > > Hello all, >Attached is a comparison of my simulation of the BBH gallery example using > the current ETK release with the data provided by the gallery. The actual > Mathematica commands are suppressed in this pdf, but can be provided if > needed. Looks good. Thanks for testing this! -- Ian Hinder https://ianhinder.net ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] [Commits] [barrywardell/EinsteinExact] 86eb5b: Removed unnecessary dependency on Boundary
> On 20 Aug 2018, at 21:23, Cupp, Samuel wrote: > > Hey Ian, >I am aware that we will need to have Kranc generate the files eventually. > However, I am changing the files themselves now to make sure the code works > properly. We will, of course, have to change the Kranc notebooks so that > equivalent code is generated. I want to verify my changes and make sure > presync is functioning and finished with major changes before I try to adjust > Kranc output. Is there a problem with that methodology? Hi Sam, No problem; sounds good! (btw: Kranc doesn't use notebooks; it uses textual .m files, so that they can be version-controlled.) -- Ian Hinder https://ianhinder.net ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] [Commits] [barrywardell/EinsteinExact] 86eb5b: Removed unnecessary dependency on Boundary
Hi Sam, These are auto-generated files from Kranc. You can't just edit them... > On 13 Aug 2018, at 23:48, Samuel Cupp wrote: > > Branch: refs/heads/presync > Home: https://github.com/barrywardell/EinsteinExact > Commit: 86eb5b6da91f914ac02f134195ed7f66dfe4ac51 > > https://github.com/barrywardell/EinsteinExact/commit/86eb5b6da91f914ac02f134195ed7f66dfe4ac51 > Author: Samuel Cupp > Date: 2018-08-13 (Mon, 13 Aug 2018) > > Changed paths: >M GaugeWave/interface.ccl >M KerrSchild/interface.ccl >M Minkowski/interface.ccl >M ModifiedSchwarzschildBL/interface.ccl >M ShiftedGaugeWave/interface.ccl >M Vaidya2/interface.ccl > > Log Message: > --- > Removed unnecessary dependency on Boundary > > > > **NOTE:** This service has been marked for deprecation: > https://developer.github.com/changes/2018-04-25-github-services-deprecation/ > > Functionality will be removed from GitHub.com on January 31st, 2019. > ___ > Commits mailing list > comm...@einsteintoolkit.org > http://lists.einsteintoolkit.org/mailman/listinfo/commits -- Ian Hinder https://ianhinder.net ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] memory leak in Carpet?
> On 8 Aug 2018, at 12:38, Miguel Zilhão > wrote: > > hi Ian, >> The memory problems are very likely strongly related to the machine you run >> on. I don't know that we can take much information from a smaller test run >> on a different machine. We already see from this run that Carpet is not >> "leaking" memory continuously; the curves for allocated memory show what has >> been malloced and not freed, and it remains more or less constant after the >> initial phase. >> I think it's worth trying to get tcmalloc running on the cluster. So this >> means that you have never seen the OOM happen when using tcmalloc. It's >> possible that the improved memory allocation in tcmalloc over glibc would >> entirely solve the problem. > > well, i did have cases where i'd ran out of memory also in my workstation > with tcmalloc (where i've been doing these tests), with this same > configuration and more resolution. i don't have an OOM-killer in the > workstation, though, so at some point the system would just start to swap (at > which point i'd kill the job). OK. >> Sorry, I made a mistake. It should have been pageheap_unmapped, not >> pageheap_free. Sorry! pageheap_free is essentially zero, and cannot >> account for the difference. > > ah, no problem. i'm attaching the updated plot. Good, that looks better. So we see that the rss mostly follows the sum of allocated and unmapped memory. I think one thing I have seen in the past is that a high rss is not necessarily an indication of a problem. Even thought the OS hasn't unmapped the pages from the process' address space, the memory is free if another process (or the current process) needs it. I suspect that the saturation point at iteration ~3000 is the point at which all the processes have a lot of unmapped memory, and the OS needs to start actually unmapping it, which stops the rss from growing any further. >>>> The point that Roland made also applies here: we are looking at the max >>>> across all processes and assuming that every process is the same. It's >>>> possible that one process has a high unmapped curve, but another has a >>>> high rss curve, and we don't see this on the plot. We would have to do 1D >>>> output of the grid arrays and plot each process separately to see the full >>>> detail. One way to see if this is necessary would be to plot both the max >>>> and min instead of just the max. That way, we can see if this is likely >>>> to be an issue. >>> >>> ok, i'm attaching another plot with both the min (dashed lines) and the max >>> (full lines) plotted. i hope it helps. >> Thanks. This shows that the gridfunction usage is more or less similar >> across all processes, which is good. However, there is significant >> variation in most of the other quantities across processes. To understand >> this better, we would have to look at 1D ASCII output of the grid arrays, >> which is a bit painful to plot in gnuplot. Before this, I would definitely >> try to get tcmalloc running and outputting this information on the cluster >> in a run that actually shows the OOM. My guess is that you won't get an OOM >> with tcmalloc, and all will be fine :) > > ok, i could also try to do this on cluster once it's back online (currently > it's down for maintenance). OK. I'll be interested to see the results when you have them. The thing to look out for is generic_current_allocated growing. -- Ian Hinder https://ianhinder.net ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] memory leak in Carpet?
> On 8 Aug 2018, at 11:41, Miguel Zilhão
> wrote:
>
> hi Ian,
>
>> The memory seems to reach a steady state by iteration ~3000. Can you run an
>> example where it dies with an OOM?
>
> the OOM cases that i had were done in our local cluster (where i haven't
> compiled with tcmalloc); those were just higher resolution versions of this
> same simulation, where the OOM would be triggered around the time of one of
> these memory increases (ie, after iteration 2000 in this case, i'd guess).
Hi Miguel,
The memory problems are very likely strongly related to the machine you run on.
I don't know that we can take much information from a smaller test run on a
different machine. We already see from this run that Carpet is not "leaking"
memory continuously; the curves for allocated memory show what has been
malloced and not freed, and it remains more or less constant after the initial
phase.
I think it's worth trying to get tcmalloc running on the cluster. So this
means that you have never seen the OOM happen when using tcmalloc. It's
possible that the improved memory allocation in tcmalloc over glibc would
entirely solve the problem.
>> Can you check this by plotting tcmalloc::generic_current_allocated +
>> tcmalloc::pageheap_free against systemstatistics-process_memory::maxrss? If
>> that is the case, then there is no issue with fragmentation, because even
>> though the address space is fragmented, the "holes" have mostly been
>> returned to the OS for other processes to use ("unmapped").
>
> sure, i've attached a plot with this.
Sorry, I made a mistake. It should have been pageheap_unmapped, not
pageheap_free. Sorry! pageheap_free is essentially zero, and cannot account
for the difference.
>> The point that Roland made also applies here: we are looking at the max
>> across all processes and assuming that every process is the same. It's
>> possible that one process has a high unmapped curve, but another has a high
>> rss curve, and we don't see this on the plot. We would have to do 1D output
>> of the grid arrays and plot each process separately to see the full detail.
>> One way to see if this is necessary would be to plot both the max and min
>> instead of just the max. That way, we can see if this is likely to be an
>> issue.
>
> ok, i'm attaching another plot with both the min (dashed lines) and the max
> (full lines) plotted. i hope it helps.
Thanks. This shows that the gridfunction usage is more or less similar across
all processes, which is good. However, there is significant variation in most
of the other quantities across processes. To understand this better, we would
have to look at 1D ASCII output of the grid arrays, which is a bit painful to
plot in gnuplot. Before this, I would definitely try to get tcmalloc running
and outputting this information on the cluster in a run that actually shows the
OOM. My guess is that you won't get an OOM with tcmalloc, and all will be fine
:)
--
Ian Hinder
https://ianhinder.net
___
Users mailing list
Users@einsteintoolkit.org
http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] memory leak in Carpet?
> On 5 Aug 2018, at 15:52, Miguel Zilhão
> wrote:
>
> hi Ian,
>
> many thanks for your analysis. for that particular run there is no
> significant change in the memory usage for longer times. however, i've now
> rerun the configuration that had originally given me problems (with lower
> resolution so that i'd have enough memory) with tcmalloc and i've plotted
> those same curves. here, the memory used increases further. i'm attaching the
> plot, together with the parfile and stdout output in case it's useful.
Hi Miguel,
The memory seems to reach a steady state by iteration ~3000. Can you run an
example where it dies with an OOM?
The meaning of the different tcmalloc statistics is described at
https://gperftools.github.io/gperftools/tcmalloc.html under "Generic Tcmalloc
Status".
From what I see here, the amount of memory allocated by Cactus grows then
reaches a steady state (the green curve). This is not accounted for in the
gridfunctions, so it must be from something else. I don't know what it might
be.Does this run also have HDF5 output disabled?
It looks like the allocated memory plus the unmapped memory would roughly equal
the rss. That is a little surprising, since I would have expected the rss to
exclude unmapped pages. Maybe until another process needs it, the kernel
doesn't actually unmap it, for performance reasons (mapping it back again would
cause a page fault).
Can you check this by plotting tcmalloc::generic_current_allocated +
tcmalloc::pageheap_free against systemstatistics-process_memory::maxrss? If
that is the case, then there is no issue with fragmentation, because even
though the address space is fragmented, the "holes" have mostly been returned
to the OS for other processes to use ("unmapped").
The point that Roland made also applies here: we are looking at the max across
all processes and assuming that every process is the same. It's possible that
one process has a high unmapped curve, but another has a high rss curve, and we
don't see this on the plot. We would have to do 1D output of the grid arrays
and plot each process separately to see the full detail. One way to see if
this is necessary would be to plot both the max and min instead of just the
max. That way, we can see if this is likely to be an issue.
--
Ian Hinder
https://ianhinder.net
___
Users mailing list
Users@einsteintoolkit.org
http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] memory leak in Carpet?
> On 3 Aug 2018, at 09:49, Miguel Zilhão > wrote: > > hi Ian, > > sorry, i thought it was enough to see the qualitative trend of the curves... > here's what i hope to be a better plot, with all the information. > also, i have no hdf5 output for this run. i'm only saving plain text. > Hi Miguel, So, from this, we can see: - Carpet is using less memory for gridfunctions after the regridding than before. I suppose this could happen if the BHs get closer to each other, and the coarser enclosed grids shrink. I'm a little surprised to see this on the very first regridding, but it's not a big effect anyway. - The amount of *allocated* memory in the tcmalloc heap increases a bit after regridding. I don't know why this would be, since the gridfunction memory should dominate, and this decreases. It would be interesting to see this plot for longer times. If the green curve continues to step up by about 80 MB every 2048 iterations, this could be the reason for running out of memory. - The heap size increases by much more than the additional allocated memory. This suggests that the heap has become fragmented, or that tcmalloc has not attempted to return memory to the OS. The tcmalloc thorn calls tcmalloc to release all memory to the OS in POSTREGRID, so as far as tcmalloc is concerned, no more memory can be returned. This suggests fragmentation; the free blocks are mixed up with allocated blocks so that entire pages cannot be mapped out. Can you set tcmalloc::report_every = 2048? The outputs a short summary of the heap status to stdout. I would again be interested to see whether this continues with a longer run. i.e. whether heap_size - allocated continues to increase. - Interestingly, pageheap_unmapped grows a lot. -- Ian Hinder https://ianhinder.net ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] memory leak in Carpet?
> On 2 Aug 2018, at 18:55, Miguel Zilhão > wrote: > > yes, sorry. here are the labels from the tcmalloc file: > > 3:generic_current_allocated_bytes > 4:generic_heap_size > 5:tcmalloc_pageheap_free_bytes > 6:tcmalloc_pageheap_unmapped_bytes > > column 5 (tcmalloc_pageheap_free_bytes) is not visible in the plot since it's > flat zero. Hi Miguel, It is very hard to see what is going on, because I am having to look in two different plots, with different plot ranges, and a second email with the column number definitions. Please can you help me to help you by organising the information a bit more clearly? It would be good to see a single plot showing all of the data, with a legend that shows what each line is. I assume that $6 in carpet-memory_procs is gridfunctions, but please check that and include the information in the plot. Also, please set the plot y axis range to start at zero; it's hard to see what's going on because the two plots have different ranges, and very confusing that one of them is not zero. From what I can tell at the moment, the new gridfunction data takes less memory than the old after the regridding, but the rss grows dramatically. The allocated memory also grows dramatically, suggesting a memory leak, but not enough to account for the growth in rss. It's possible that something other than regridding is responsible. Do you have any HDF5 output enabled that might coincide with iteration 2048? HDF5 might be allocating buffers on first output that are then not freed because they will be used again later. Can you disable all output and see what happens? -- Ian Hinder https://ianhinder.net ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] memory leak in Carpet?
Can you put labels on the plot for each curve? I don’t know which variables correspond to the different columns in the file. -- Ian Hinder http://members.aei.mpg.de/ianhin > On 2 Aug 2018, at 18:02, Miguel Zilhão > wrote: > > hi Ian, > >> Can you do this with tcmalloc (and activate the tcmalloc thorn that I >> pointed you to), and plot the variables tcmalloc:: >> generic_current_allocated_bytes >> generic_heap_size >> tcmalloc_pageheap_free_bytes >> tcmalloc_pageheap_unmapped_bytes >> This will let us know whether there are actual memory allocations which are >> being made and not freed, or whether the rss is increasing due to >> fragmentation. > > yes, i've just re-ran with tcmalloc. please see the (very crude) plot > attached. does this help? sorry, i don't really know how to interpret these > data :-) > > thanks! > Miguel > > ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] memory leak in Carpet?
> On 2 Aug 2018, at 08:45, Miguel Zilhão > wrote: > > hi all, > > some more information regarding this. i've ran a simulation based on the > provided parfile qc0-mclachlan.par. i'm attaching a crude plot with the > memory consumption, as reported by systemstatistics-process_memory and > carpet-memory_procs, as function of the iteration. > > the jump at iteration ~2048 corresponds to the time where some non-trivial > regridding occurred. if i'm interpreting the plot correctly, while the total > memory consumption of the system (as reported by systemstatistics) increases, > the memory that carpet reports to be using is actually decreasing. is this a > sign that something is probably leaking memory? > > in case it's useful, i'm also attaching the exact parameter file i've used. Can you do this with tcmalloc (and activate the tcmalloc thorn that I pointed you to), and plot the variables tcmalloc:: generic_current_allocated_bytes generic_heap_size tcmalloc_pageheap_free_bytes tcmalloc_pageheap_unmapped_bytes This will let us know whether there are actual memory allocations which are being made and not freed, or whether the rss is increasing due to fragmentation. -- Ian Hinder https://ianhinder.net ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] memory leak in Carpet?
field from mallinfo, > which is supposed to be "Space allocated in mmapped regions (bytes)". This > increases to about 2 GB after a couple of regriddings, but then stays roughly > constant at 2 GB, which seems fine, apart from the fact that I had hoped that > mmap was being used for all the gridfunction data I suspect I got distracted doing my own investigations while I was writing it, and then lost track of it. Typically, I think I run with excess memory for each run, so that by 24 h, it hasn't grown too badly. Further searching finds an email from someone else saying that they had memory leak problems and asking me about it. My reply was: On 5 May 2015, at 17:14, Ian Hinder wrote: > Hi, > > I have added Erik and Roland to CC, as we have been discussing this; I hope > this is OK. > > It sounds very similar. I am running on several hundred cores (<600) and the > simulations often fail with OOM errors after less than a day. First the RSS > grows, then the swap, then the OOM-killer kills it. I have observed this > both on Datura and Hydra. Stopping the simulations and recovering from a > checkpoint usually fixes the problem, and it runs on for another half day or > so. I have done a fair amount of work on this, so I will summarise here. > > Monitor process RSS > > The process resident set size is the amount of address space which is > currently mapped into physical memory by the OS. Thorn SystemStatistics can > be used to measure this and put it into a Cactus variable, which can then be > reduced across processes. I use: > > IOBasic::outInfo_every = 1 > IOBasic::outInfo_reductions = "maximum" > IOBasic::outInfo_vars = " > SystemStatistics::maxrss_mb > SystemStatistics::swap_used_mb > Carpet::gridfunctions > " > > and > > IOScalar::outScalar_every = 128 > IOScalar::outScalar_vars = " > SystemStatistics::process_memory_mb > Carpet::memory_procs > " > > SystemStatistics calls its variable "maxrss" but it should actually be called > "rss", as that is what is output. maxrss is also available from the OS, and > would give the maximum the RSS had ever been during the process lifetime. > > Carpet::gridfunctions (in Carpet::memory_procs) measures the amount of memory > Carpet has allocated in gridfunctions. For me, this remains essentially > flat, whereas maxrss grows after each regridding until it reaches the maximum > available, then the swap starts to grow. This indicates that the problem is > not due to Carpet allocating more and more grid points due to grids changing > size. It could be due to failing to free allocated memory (a leak) or freed > data taking up space which cannot be used for further allocations or returned > to the OS (fragmentation). > > Terminate and checkpoint on OOM > > I have a local change to SystemStatistics which adds parameters for maximum > values of RSS and swap usage, above which it calls CCTK_TerminateNext, so if > you have checkpoint_on_terminate, you get a clean termination and can > continue the run without losing too much CPU time. I have been running with > this for a couple of weeks now, and it works as advertised. I have a branch > with this on, but I just realised it conflicts with a change Erik made. If > you want this, let me know and I will sort it out. > > Memory profiling > > The malloc implementation in glibc provides no usable statistics. mallinfo > is limited to 32 bit integers, which overflow for 64 bit systems. > malloc_info, at least in the version on datura, doesn't include memory > allocated via mmap. Useless. Instead, you need to use an external memory > profiler to see what is going on. I have used "igprof" successfully, and > this shows me that there is no "leak" of allocated memory corresponding to > the increase in RSS. i.e. the problem is not caused by forgetting to free > something. This suggests that the problem is fragmentation, where malloc has > unallocated blocks of memory which it does not or cannot return to the OS. > Malloc allocates memory in two ways: either in its main heap, or by > allocating anonymous mmap regions. I had thought that only the latter could > be fully returned to the OS, but this is not true. Any region of address > space can be marked as unused (internally via the madvise(MADV_DONTNEED) > system call) and a malloc implementation can do this on regions of its > address space which have been freed. If such regions are too small (smaller > than a page), then they could accumulate and not be returned to the OS. > > Alternative malloc implementations > > At the suggestion of
Re: [Users] memory leak in Carpet?
> On 19 Jul 2018, at 17:14, Miguel Zilhão > wrote: > > hi Ian, > >>> i've noticed that my runs (using latest ET release) with CarpetRegrid2 >>> exhibit a significant >>> increase in memory during runtime. this seems to happen immediately after >>> some non-trivial >>> regridding operation is done. the increase is steady, and at some point i >>> run out of memory and the >>> simulation crashes. this is happening both on my workstation (running >>> Ubuntu 18.04) as well as our >>> local cluster (running Debian 9). i was wondering if someone has seen >>> something like this? >>> >>> i have not seen this happen for simulations without CarpetRegrid2. i show >>> below some relevant >>> portions of the stdout file for a standard inspiral BH run (note the last >>> column--maxrss_mb): >>> >> This could be caused by memory fragmentation due to all the freeing and >> mallocing that happens during regridding when the sizes of the grids change. >> Can you try using tcmalloc or jemalloc instead of glibc malloc and >> reporting back? One workaround could be to run shorter simulations (i.e. >> set a walltime of 12 h instead of 24 h). > > thanks for your reply. in one of my cases, for the resolution used and the > available memory, i was out of memory quite quickly -- within 6 hours or > so... so unfortunately it becomes a bit impractical for large simulations... > > what would i need to do in order to use tcmalloc or jemalloc? I have used tcmalloc. I think you will need the following: - Install tcmalloc (https://github.com/gperftools/gperftools), and libunwind, which it depends on. - In your optionlist, link with tcmalloc. I have LDFLAGS = -rdynamic -L/home/ianhin/software/gperftools-2.1/lib -Wl,-rpath,/home/ianhin/software/gperftools-2.1/lib -ltcmalloc This should be sufficient I think for tcmalloc to be used instead of glibc malloc. Try this out, and see if things are better. I also have a thorn which hooks into the tcmalloc API. You can get it from https://bitbucket.org/ianhinder/tcmalloc It's very much a work in progress, and probably has some hard-coded assumptions in it. You can set Cactus parameters to: 1. Report memory statistics periodically 2. Release memory back to the OS periodically 3. Output a memory profile periodically Let us know how it goes! -- Ian Hinder https://ianhinder.net ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] memory leak in Carpet?
> On 19 Jul 2018, at 11:58, Miguel Zilhão > wrote: > > hi all, > > i've noticed that my runs (using latest ET release) with CarpetRegrid2 > exhibit a significant > increase in memory during runtime. this seems to happen immediately after > some non-trivial > regridding operation is done. the increase is steady, and at some point i run > out of memory and the > simulation crashes. this is happening both on my workstation (running Ubuntu > 18.04) as well as our > local cluster (running Debian 9). i was wondering if someone has seen > something like this? > > i have not seen this happen for simulations without CarpetRegrid2. i show > below some relevant > portions of the stdout file for a standard inspiral BH run (note the last > column--maxrss_mb): > This could be caused by memory fragmentation due to all the freeing and mallocing that happens during regridding when the sizes of the grids change. Can you try using tcmalloc or jemalloc instead of glibc malloc and reporting back? One workaround could be to run shorter simulations (i.e. set a walltime of 12 h instead of 24 h). -- Ian Hinder https://ianhinder.net ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] Queuing system and parameter file scripts on personal machine
> On 18 Jul 2018, at 18:12, Federico G Lopez Armengol > wrote: > > Maybe I misunderstood the point of parameter file scripting. I understood > they serve for automatically create several parameter files, and > automatically run (or submit) the corresponding simulations. Hi, Yes, you misunderstood :). The documentation says "A parameter file script is a file with a ”.rpar” extension which, when executed, generates a file in the same place but with a ”.par” extension. The resulting file should be a valid SimFactory parameter file." So you can only generate a single parameter file from the script. You could take a look at the bbh example (https://bitbucket.org/einsteintoolkit/einsteinexamples/raw/master/par/GW150914/GW150914.rpar) to see why this sort of thing is needed. Some of what is done in the python header of that script cannot be done directly in the parameter file, or it would be incredibly tedious and error prone to do so. Having a higher-level mechanism for managing collections of parameter files and simulations would be very useful, but it does not exist at the moment. You will have to implement it yourself, e.g. in a shell script. You should probably use sim create-run rather than sim create-submit so that the simulations don't run in the background at the same time. -- Ian Hinder https://ianhinder.net ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] Printing values when using MPI
> On 21 Jun 2018, at 09:41, Chris Stevens wrote: > > Hi Ian, > > I have just gone to implement your advice but am confused as to what you mean > by Cactus command line. Is this additions to the .sub or .run file? I am > using simfactory to submit my jobs. Hi, The .run file is the script that actually runs Cactus, and that's where the change needs to be made. For example, if you are using generic.run, you would change the line from mpirun -np @NUM_PROCS@ @EXECUTABLE@ -L 3 @PARFILE@ to mpirun -np @NUM_PROCS@ @EXECUTABLE@ -roe -L 3 @PARFILE@ Unfortunately it's all a little convoluted because of the layers of abstractions that simfactory provides. And the interface is not complete, which means do you have to delve into it from time to time. i.e. there is no way to pass through additional options to Cactus from the sim submit command line, as far as I know. Note: in order to make SimFactory use the new runscript, you must rerun "sim build" and give it the runscript again, i.e. sim build [] --runscript generic.run (replacing generic.run with whatever runscript you are using). It does not automatically detect the change and use the new runscript. -- Ian Hinder https://ianhinder.net ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] Printing values when using MPI
> On 17 Jun 2018, at 10:52, Chris Stevens wrote: > > Hi everyone, > > I am currently debugging some code and have been printing out (using either > CCTK_INFO and/or printf) debugging information at particular grid points > (LOCAL mode). > > From what I can tell, Cactus only allows output from one process, say rank 0, > to stop flooding of the output by printing everything once per process. > However if I am wanting data from a particular grid point, and this grid > point is not in the printing process, nothing will output. > > I have been getting around this by turning off MPI (fine for one iteration or > so) but now I am doing longer runs I need to be able to print out this > information with MPI. > > Thus my question: > > How can you force Cactus to print out a statement from any process? > Add -roe to the Cactus command line and you will get separate output and error files from each process. -- Ian Hinder https://ianhinder.net ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] timeouts accessing svn.cactuscode.org using svn
> On 9 Jun 2018, at 20:34, Roland Haas wrote: > > Signed PGP part > Hello all, > > I just tried to check out the Einstein Toolkit and received: > > Could not checkout module ExternalLibraries/BLAS > svn: E170013: Unable to connect to a repository at URL > 'https://svn.cactuscode.org/projects/ExternalLibraries/BLAS/branches/ET_2018_02' > svn: E000110: Error running context: Connection timed out > > Could not checkout module ExternalLibraries/PAPI > svn: E170013: Unable to connect to a repository at URL > 'https://svn.cactuscode.org/projects/ExternalLibraries/PAPI/branches/ET_2018_02' > svn: E000110: Error running context: Connection timed out > > Could not checkout module ExternalLibraries/GSL > svn: E170013: Unable to connect to a repository at URL > 'https://svn.cactuscode.org/projects/ExternalLibraries/GSL/branches/ET_2018_02' > svn: E000110: Error running context: Connection timed out > > There seem to be (possible unrelated) issues with other svn > repositories at LSU right now, so this may not be restricted to just > those three repositories. The Jenkins update script is also failing with Can't create session: Unable to connect to a repository at URL 'https://svn.cct.lsu.edu/repos/numrel/LSUThorns/CPUID': Error running context: Connection timed out at /usr/share/perl5/Git/SVN.pm line 143. It works fine from my laptop, but not from Nebula at NCSA, for some reason. -- Ian Hinder https://ianhinder.net signature.asc Description: Message signed with OpenPGP ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] Toolkit
> On 30 May 2018, at 15:35, Feyisso Sado wrote: > > Dear Ian Hinder > > Many Thanks > > I'm searching google, but still not resolved > > My Ubuntu version: Ubuntu 14.04.5 LTS > and > curl --version: curl 7.52.1 (x86_64-pc-linux-gnu) libcurl/7.52.1 > OpenSSL/1.0.2l zlib/1.2.8 > Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp > smb smbs smtp smtps telnet tftp Hi, (Please can you leave users@einsteintoolkit.org <mailto:users@einsteintoolkit.org> in the CC, with "reply all", so that others will benefit from and contribute to the discussion? Thanks!) OK, that all looks good. This is weird. Can you check that the right host is being connected to with host raw.githubusercontent.com <http://raw.githubusercontent.com/> and also that you can connect using openssl: openssl s_client -connect raw.githubusercontent.com:443 You should see something like Verify return code: 0 (ok) at the end. (Use ctrl-C to exit from the connection). If you get some error message, please post it here. > > With Best Regards!!! > --- > Feyiso Sado Bedecha (PhD), Student > Addis Ababa University, Department of Physics > Cell Phone: +251910626090 > E-mail: bsfey...@gmail.com <mailto:bsfey...@gmail.com> > P.O.Box: 1176 > Ethiopia > > On Tue, May 29, 2018 at 11:43 PM, <mailto:ian.hin...@aei.mpg.de>> wrote: > > >> On 26 May 2018, at 08:06, Feyisso Sado > <mailto:bsfey...@gmail.com>> wrote: >> >> After installing all requirements, I just typed the following commands in >> Ubuntu terminal: >> >> !curl -kLO >> https://raw.githubusercontent.com/gridaphobe/CRL/ET_2018_02/GetComponents >> <https://raw.githubusercontent.com/gridaphobe/CRL/ET_2018_02/GetComponents> >> !chmod a+x GetComponents > >>> Due to my machine failure I'm reinstalling Toolkit, but I faced the >>> following error. >>> curl: (35) error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown >>> protocol >>> > > Hi, > > I copied that command and ran it on Mac OS and Linux, and it worked OK. What > version of Ubuntu are you using? You can get this from the output of > > cat /etc/issue > > Can you send the output of > > curl --version > > A quick google search says that that particular error message occurs if you > try to access an http server with https. Is it possible that there is some > firewall or redirection happening on your network to prevent you from > accessing https servers? > > -- > Ian Hinder > http://members.aei.mpg.de/ianhin <http://members.aei.mpg.de/ianhin> > -- Ian Hinder http://members.aei.mpg.de/ianhin ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] Toolkit
> On 26 May 2018, at 08:06, Feyisso Sado wrote: > > After installing all requirements, I just typed the following commands in > Ubuntu terminal: > > !curl -kLO > https://raw.githubusercontent.com/gridaphobe/CRL/ET_2018_02/GetComponents > <https://raw.githubusercontent.com/gridaphobe/CRL/ET_2018_02/GetComponents> > !chmod a+x GetComponents >> Due to my machine failure I'm reinstalling Toolkit, but I faced the >> following error. >> curl: (35) error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown >> protocol >> Hi, I copied that command and ran it on Mac OS and Linux, and it worked OK. What version of Ubuntu are you using? You can get this from the output of cat /etc/issue Can you send the output of curl --version A quick google search says that that particular error message occurs if you try to access an http server with https. Is it possible that there is some firewall or redirection happening on your network to prevent you from accessing https servers? -- Ian Hinder http://members.aei.mpg.de/ianhin ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
[Users] Fwd: Toolkit
> Begin forwarded message: > > From: Feyisso Sado > Subject: Re: [Users] Toolkit > Date: 26 May 2018 at 08:06:53 BST > To: ian.hin...@aei.mpg.de > > After installing all requirements, I just typed the following commands in > Ubuntu terminal: > > !curl -kLO > https://raw.githubusercontent.com/gridaphobe/CRL/ET_2018_02/GetComponents > <https://raw.githubusercontent.com/gridaphobe/CRL/ET_2018_02/GetComponents> > !chmod a+x GetComponents > > > With Best Regards!!! > --- > Feyiso Sado Bedecha (PhD), Student > Addis Ababa University, Department of Physics > Cell Phone: +251910626090 > E-mail: bsfey...@gmail.com <mailto:bsfey...@gmail.com> > P.O.Box: 1176 > Ethiopia > > On Sat, May 26, 2018 at 3:05 AM, <mailto:ian.hin...@aei.mpg.de>> wrote: > > >> On 24 May 2018, at 08:19, Feyisso Sado > <mailto:bsfey...@gmail.com>> wrote: >> >> Dear sir/madam >> >> How are you doing? >> I'm Sado a PhD student at Addis Ababa University, Ethiopia >> I'm working on compact binary merger and massive star core collapse that >> result in hyperaccreting Kerr hole surrounded accretion disk. I'm simulating >> this magnetorotational collapse using Einstein Toolkit in full general >> relativity. >> Due to my machine failure I'm reinstalling Toolkit, but I faced the >> following error. >> curl: (35) error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown >> protocol >> Please he me resolve > > Please can you tell us the command that you ran to get this error message? > > -- > Ian Hinder > http://members.aei.mpg.de/ianhin <http://members.aei.mpg.de/ianhin> > -- Ian Hinder http://members.aei.mpg.de/ianhin ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] Toolkit
> On 24 May 2018, at 08:19, Feyisso Sado wrote: > > Dear sir/madam > > How are you doing? > I'm Sado a PhD student at Addis Ababa University, Ethiopia > I'm working on compact binary merger and massive star core collapse that > result in hyperaccreting Kerr hole surrounded accretion disk. I'm simulating > this magnetorotational collapse using Einstein Toolkit in full general > relativity. > Due to my machine failure I'm reinstalling Toolkit, but I faced the following > error. > curl: (35) error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol > Please he me resolve Please can you tell us the command that you ran to get this error message? -- Ian Hinder http://members.aei.mpg.de/ianhin ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] restart simulation with checkpoints
> On 23 May 2018, at 08:25, 林家暉 wrote: > > Dear Ian, > I used parameter file in ~/Cactus/par. It seems this parameter is only used > for the first run instead of restarting. And yes, I am using simfactory. I > follow your steps to edit the parameter file in /SIMFACTORY/par > and successfully recover my simulation. Hi, The problem is that simfactory does not use the original parameter file automatically when resubmitting the simulation. You can make it do so by adding the --parfile par/XXX.par parameter to the "sim submit" command line. It does the same with optionlists and thornlists. This is so that the simulation is self-contained, and no longer depends on external files once it has been created unless the user explicitly asks for something to be updated. -- Ian Hinder http://members.aei.mpg.de/ianhin ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] restart simulation with checkpoints
> On 22 May 2018, at 20:11, 林家暉 wrote: > > Dear who it may concern, > I have tried the binary neutron star simulation > (http://einsteintoolkit.org/gallery/bns/index.html > <http://einsteintoolkit.org/gallery/bns/index.html>), and want to finish the > simulation by parts .That is , I would like to recover the simulation from > checkpoints. However I fail to continue my simulation. There are two cases > that I met: > I have the cctk_final_time=2500 and the simulation terminate at time=2500 but > the merge has not yet occur which means I need more time ,perhaps time=8000. > So I want to continue my simulation from time=2500. But after I changed to > cctk_final_time=8000 in parameter file , the simulation still stops at > time=2500, which means it stops just after it begins. How can I finish the > complete simulation? > I have test the same simulation as above but with wall_time=10 min. I have > checkpoint_every=150 ,and checkpoint_on_terminate= "yes" in the parameter > file. However after it finishes with the final iteration=250, the checkpoint > file is not produced. Only the initial one checkpoint.chkpt.it_0.file_0.h5 > exists. How can I produce a checkpoint which enable me to restart simulation? Hi, Which parameter file are you editing? If it is the one INSIDE the simulation output directory, then this will not work, because this is never read, only written. Are you using simfactory? If so, then I usually just edit the parameter file in /SIMFACTORY/par then submit the next segment. -- Ian Hinder http://members.aei.mpg.de/ianhin ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] [ET Trac] [Einstein Toolkit] #1471: Cactus should auto-detect newer versions of GCC from MacPorts
> On 1 May 2018, at 15:47, Steven R. Brandt wrote: > > AFAIK, if it compiles with generic.cfg, it has to use the "gcc" that's in the > path. > > And when homebrew has its own version of gcc, does it put it on the path under the name "gcc"? What does "which gcc" give you? If it is indeed compiling OK with Clang, it's interesting, because when I tried to compile just Cactus, not even the whole ET, using Clang on a Mac, I ran into several incompatibilities. This was a couple of years ago now, and maybe they have all been fixed. -- Ian Hinder http://members.aei.mpg.de/ianhin ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] [ET Trac] [Einstein Toolkit] #1471: Cactus should auto-detect newer versions of GCC from MacPorts
> On 30 Apr 2018, at 16:30, Einstein Toolkit > wrote: > > #1471: Cactus should auto-detect newer versions of GCC from MacPorts > --+- > Reporter: Ian Hinder | Owner: (none) > Type: enhancement | Status: new > Priority: minor| Milestone: > Component: Cactus |Version: development version > Resolution: | Keywords: > --+- > > Comment (by Steven R. Brandt): > > Ian, gcc -version showed that it was clang. And is this the compiler that Cactus picked up? -- Ian Hinder http://members.aei.mpg.de/ianhin ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
[Users] ET website updates
Hi Steve, If I push to the ET website repo, does it automatically get reflected on the web? Or do I need to click the button on http://einsteintoolkit.org/update.php <http://einsteintoolkit.org/update.php> ? What else does that button do? -- Ian Hinder http://members.aei.mpg.de/ianhin ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] 2D configuration with Carpet
> On 14 Feb 2018, at 16:21, Miguel Zilhão > wrote: > > hi Ian, > > many thanks for the parameter file. this allowed me to go a little further (i > was missing CoordBase::boundary_shiftout_z_lower = 1 and > CoordBase::boundary_shiftout_z_upper = 1). > however, i'm still getting the following error: > > ERROR from host meurglysIII process 0 > while executing schedule bin (none), routine (no thorn)::(no routine) > in thorn Carpet, file > ./Cactus/arrangements/Carpet/Carpet/src/SetupGH.cc:2512: > -> There are not enough ghost zones for the desired spatial prolongation > order on map 0, refinement level 0. With a spatial prolongation order of 5, > you need at least 3 ghost zones. > > i'm setting: > > Carpet::prolongation_order_space= 5 > Carpet::prolongation_order_time = 2 > > driver::ghost_size_x = 3 > driver::ghost_size_y = 3 > driver::ghost_size_z = 0 > > i'm guessing that in the example you provided things worked because you only > had one grid? is there any way of doing this with more inner levels? Hi, I have never tried to do this with mesh refinement, no. This might be a limitation in Carpet, because perhaps it tries to prolongate with the same order in every direction. -- Ian Hinder http://members.aei.mpg.de/ianhin ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] 2D configuration with Carpet
> On 13 Feb 2018, at 06:19, Miguel Zilhão > wrote: > > hi Erik, > > thanks for your reply. i've tried to set the parameters as you described, but > i'm getting a Carpet > assertion failure: > > cactus_Lean_ET: Cactus/arrangements/Carpet/CarpetLib/src/gh.cc:61 > <http://gh.cc:61/>: gh::gh(const > std::vector >&, centering, int, centering, const > std::vector > >> &, const i2vect&): Assertion `all(box.shape() / box.stride() >= >> boundary_width[0] + > boundary_width[1])' failed. > > here's how i'm specifying my grid: > > CoordBase::xmin = -48.00 > CoordBase::ymin = -48.00 > CoordBase::zmin = 0.00 > CoordBase::xmax = +48.00 > CoordBase::ymax = +48.00 > CoordBase::zmax = +0.00 > CoordBase::dx= 2.00 > CoordBase::dy= 2.00 > CoordBase::dz= 2.00 > > driver::ghost_size_x = 3 > driver::ghost_size_y = 3 > driver::ghost_size_z = 0 > > CoordBase::boundary_size_x_lower = 3 > CoordBase::boundary_size_y_lower = 3 > CoordBase::boundary_size_z_lower = 0 > CoordBase::boundary_size_x_upper = 0 > CoordBase::boundary_size_y_upper = 0 > CoordBase::boundary_size_z_upper = 0 > > CoordBase::boundary_shiftout_x_lower = 0 > CoordBase::boundary_shiftout_y_lower = 0 > CoordBase::boundary_shiftout_z_lower = 0 > > am i missing something? Hi Miguel, There is an example parameter file in Kranc, for a true 2D Laplace equation (i.e. it doesn't take derivatives in the z direction): https://github.com/ianhinder/Kranc/blob/master/Examples/laplace.par I have not tried this recently, but it worked at one point. -- Ian Hinder http://members.aei.mpg.de/ianhin ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] best practices for groups
> On 12 Feb 2018, at 19:55, Eric West wrote: > > Hi All, > > Do research groups distribute Cactus/EinsteinToolkit files amongst > different users on the same machine? I understand that the Cactus > philosophy is that it is meant to be distributed. But it seems redundant > for every member of a group, using the same machine, to have their own > copies of everything. I am curious how other groups handle this. What > parts of the toolkit are shared amongst multiple users, if any? What > parts are not shared? > > For context, our group is very small (currently myself and a couple of > grad students). The disk quota for the group on our HPC machine is > 150GB. Right now that is not prohibitively small. I'm more concerned > about sprawling config files, parfiles, etc. Although if there are > things that can be done to reduce the overall disk usage, that would be > an added bonus. So I am wondering what practices other groups employ. Hi, I think that usually, people have their own copies of the source code and build it themselves. This is because, historically, there were unlikely to be users who were not also developers. So everyone would be working on their own local changes, and you would not want to conflict with other people. Nowadays, there may be people who run the code without modifying it, and you could imagine a situation where they simply shared a single executable (no need for source code, since it won't be changed). A few GB for source code and build files is not that much, these days, though. Config files and parameter files should be kept in a version controlled repository. Having separate copies for individual users then translates into having multiple checkouts of those repositories, which allows users to make changes which don't conflict with other people, but then also to share those changes in a controlled way. -- Ian Hinder http://members.aei.mpg.de/ianhin ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] questions about compilation
> On 10 Feb 2018, at 06:11, 林家暉 wrote: > > Dear Ian, > I followed the steps in this page: > http://simulationtools.org/Documentation/English/Tutorials/Introduction.html > <http://simulationtools.org/Documentation/English/Tutorials/Introduction.html> > I have installed both SimulationTools and h5mma into my home directory in > Edison, but I cannot move it to ~/Library/Mathematica/Applications which > seems to be a required step : > <螢幕快照 2018-02-10 下午1.31.47.png> > the message is below: > <螢幕快照 2018-02-10 下午1.38.10.png> Hi, The instructions say SimulationTools is available as a tar.gz file containing the SimulationTools application directory. The SimulationTools directory should be placed in ~/Library/Mathematica/Applications on Mac OS, and ~/.Mathematica/Applications on Linux. Since you are using Linux on Edison, you need to place it in ~/.Mathematica/Application, which is in your home directory. There are equivalent download instructions at http://simulationtools.org/download.shtml <http://simulationtools.org/download.shtml> which might clarify this. > By the way, I have installed NX,which is a computer program that handles > remote X Window System connections to run Methematica. And I tried directly > load SimulationTools (maybe there is SimulationTools already installed in > Edison, or maybe I can load SimulationTools even it is located in my home > directory) like below: > <螢幕快照 2018-02-10 下午2.03.30.png> > It failed, so SimulationTools seems to be required to move to Application > directory of Mathematica. Yes, that should fix the problem. NX is good; that should be fast enough to use Mathematica remotely. Good luck, and please keep asking questions if you need help! -- Ian Hinder http://members.aei.mpg.de/ianhin ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] questions about compilation
> On 9 Feb 2018, at 16:33, 林家暉 <mailto:r06222...@ntu.edu.tw>> wrote: > > Dear Ian, > Thanks for your reply. > Besides VisIt, I also tried SimulationTools as another way to analyze data. > However I do not have Mathematica installed in my local machine(which is a > laptop) , so I tried to use the Mathematica in Edison. But it seems that I do > not have permission to load a new tool on the cluster. Is there any method > can I use it on a super cluster? Hi, I don't know what you mean "permission to load a new tool on the cluster". You can install SimulationTools into your home directory, as described in the installation instructions. This should not need any special technical permissions. If this is what you tried, and it doesn't work, can you give some more details about what you did and what went wrong? If you mean that you are not *allowed* to install external software in your home directory, then I don't know how you can use SimulationTools on the cluster, as it would need to be installed. In any case, in order to visualise the data, you will need a graphical display, so presumably you would need to use Mathematica with X forwarding or similar. Unless you have quite a fast connection to the cluster, this might be too slow to be usable. There is no problem with using SimulationTools in Mathematica from the command line or in scripts, but I suspect that's not really what you need. What I usually do is copy the required data from the cluster to my laptop, and visualise it with SimulationTools in Mathematica on my laptop. If the data is too big to transfer, I use Mathematica's "remote kernel" feature, where the notebook interface runs on my laptop, and the kernel runs on the cluster, where it can directly access the data. This way, only the visualisation needs to be sent over the network, not the original raw data. Of course, Mathematica is very expensive, so this might not be an ideal solution for you. -- Ian Hinder http://members.aei.mpg.de/ianhin <http://members.aei.mpg.de/ianhin> ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] questions about compilation
> On 9 Feb 2018, at 16:33, 林家暉 <mailto:r06222...@ntu.edu.tw>> wrote: > > Dear Ian, > Thanks for your reply. > Besides VisIt, I also tried SimulationTools as another way to analyze data. > However I do not have Mathematica installed in my local machine(which is a > laptop) , so I tried to use the Mathematica in Edison. But it seems that I do > not have permission to load a new tool on the cluster. Is there any method > can I use it on a super cluster? Hi, I don't know what you mean "permission to load a new tool on the cluster". You can install SimulationTools into your home directory, as described in the installation instructions. This should not need any special technical permissions. If this is what you tried, and it doesn't work, can you give some more details about what you did and what went wrong? If you mean that you are not *allowed* to install external software in your home directory, then I don't know how you can use SimulationTools on the cluster, as it would need to be installed. In any case, in order to visualise the data, you will need a graphical display, so presumably you would need to use Mathematica with X forwarding or similar. Unless you have quite a fast connection to the cluster, this might be too slow to be usable. There is no problem with using SimulationTools in Mathematica from the command line or in scripts, but I suspect that's not really what you need. What I usually do is copy the required data from the cluster to my laptop, and visualise it with SimulationTools in Mathematica on my laptop. If the data is too big to transfer, I use Mathematica's "remote kernel" feature, where the notebook interface runs on my laptop, and the kernel runs on the cluster, where it can directly access the data. This way, only the visualisation needs to be sent over the network, not the original raw data. Of course, Mathematica is very expensive, so this might not be an ideal solution for you. -- Ian Hinder http://members.aei.mpg.de/ianhin <http://members.aei.mpg.de/ianhin> ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] questions about compilation
> On 9 Feb 2018, at 07:01, 林家暉 wrote: > > Dear Ian and whom it may concern, > Thank you very much and I successfully plot the surface of the 2D topological > spheres ,as showed below. > However I found there is always a line in the plot from the right to the left > and through the center . What is the possible reason of it ? For instance , > resolution or something else. Hi, I think this is a visualisation artefact; i.e. it is a problem in either VisIt or the Carpet plugin for VisIt; it's not a problem in the simulation itself. I seem to remember that there was a way to avoid this by first "slicing" the NxMx1 3D dataset that Carpet produces to be an NxM 2D dataset, so that VisIt knows it is 2D, but I think when I tried this, it didn't work. Perhaps someone who is more expert in VisIt could suggest something? -- Ian Hinder http://members.aei.mpg.de/ianhin ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] Release Meeting, Wednesday, 2/7/2018, 9AM Central Time
> On 6 Feb 2018, at 04:24, Steven R. Brandt wrote: > > The next (and possibly final) release meeting for Tesla will be > Wednesday, 2/5/2018, 9AM Central Time. Please call in to > https://hangouts.google.com/hangouts/_/stevenrbrandt.com/etcall if you > are interested. Slight correction: Wednesday 2/7/2018. -- Ian Hinder http://members.aei.mpg.de/ianhin ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] questions about compilation
> On 1 Feb 2018, at 04:47, 林家暉 wrote: > > Dear Roland and whom it may concern, > I have done the simulation of the example code of GW150914. > However when I tried to visualize the result by VisIt ( > https://docs.einsteintoolkit.org/et-docs/GW150914_VisIt_Tutorial#Installing_VisIt > ), I can not find the .vtk files which represent the shape and properties of > relevant 2D topological spheres in the output data. Only .h5 files are in my > output file. > What is the possible reason for it ? If you look at the parameter file, it has the lines # out3d_every = rl0_every * 2 out3d_every = 0 You need to swap the comment, so that it reads out3d_every = rl0_every * 2 # out3d_every = 0 i.e. you need to have a nonzero out3d_every parameter. The default in the parameter file is NOT to output this 3D data, as it totals 3 TB for the whole simulation, which is more than most people want to generate! Note that this parameter is used for both the 3D gridfunction output, as well as the horizon surface data (in VTK files). If you just want the horizon files, then you can change QuasiLocalMeasures::output_vtk_every = $out3d_every to QuasiLocalMeasures::output_vtk_every = $rl0_every * 2 without changing out3d_every. I'm afraid you will have to rerun the simulation, as the VTK files were not generated when you ran it the first time. > By the way , I have commended some module load in the edison.ini file, listed > below: > #module load cray-petsc/3.7.6.0 > #module load curl/7.28.1 > #module load hwloc/1.7.2 > #module load numactl/2.0.10 > I wonder whether this module commended resulted in the disappearance of .vtk > files. No, this should be unrelated. -- Ian Hinder http://members.aei.mpg.de/ianhin ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] questions about compilation
> On 21 Jan 2018, at 17:16, Roland Haas <mailto:rh...@illinois.edu>> wrote: > > Hello Ian, > >> Shouldn't these changes be backported to the current release? > Sure. Would only be for a couple of weeks though. I am happy to > backport them though. That would allow the machine to be used for science now by people just running "git pull" in the simfactory directory, rather than waiting until the end of February for the next release. -- Ian Hinder http://members.aei.mpg.de/ianhin <http://members.aei.mpg.de/ianhin> signature.asc Description: Message signed with OpenPGP ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] New time for the Einstein Toolkit calls: 7:00am PST / 9:00am CST / 10:00am EST / 15:00 CET
> On 20 Jan 2018, at 00:22, Roland Haas wrote: > > Hello all, > > as pointed out in the ET reminder email, the new meeting time is > 9:00 am US central time on Mondays. For details on how to connect > and what agenda items are to be discussed, use the link below. > > https://docs.einsteintoolkit.org/et-docs/Main_Page#Weekly_Users_Call > <https://docs.einsteintoolkit.org/et-docs/Main_Page#Weekly_Users_Call> Hi, I am unable to join the call; is it happening? -- Ian Hinder http://members.aei.mpg.de/ianhin signature.asc Description: Message signed with OpenPGP ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] questions about compilation
> On 20 Jan 2018, at 16:11, 林家暉 wrote: > > Dear Roland, > Thanks for your answering. > I am looking forward to the next release of ET and I would try it on edison > at that time. > So before then, I would try ET on other machine. > Best regards, > Chia-Hui Lin Hi Roland, Shouldn't these changes be backported to the current release? -- Ian Hinder http://members.aei.mpg.de/ianhin ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
[Users] Release plans
Hi all, As we discussed in the call last Monday, it would be good to finalise the list of tickets that must be addressed for the upcoming release. If you have time, please look through the open tickets and add the ET_2018_02 milestone to those that we should consider. We can then decide which ones we have time to work on for this release, and which ones will be bumped to the next release. Thanks! -- Ian Hinder http://members.aei.mpg.de/ianhin ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] WaveDemo Errors
> On 6 Jan 2018, at 00:52, Andres Mauricio Suarez Mantilla > wrote: > > Hi Everybody. > > I just installed Cactus and einsteintoolkit.th <http://einsteintoolkit.th/> > ran without any problem. However, I tried to run WaveDemo thorn and it showed > some problems (please see the attached image). > > I really don't know how to fix such errors, so I hope you can help me with > this. > > I appreciate your support. > > Pdta: I am working in LinuxMint 18.1 Serena. Hi, (To clarify for other readers, WaveDemo is not in einsteintoolkit.th) Were you following a particular tutorial? If so, can you give more details? If you found a thornlist on one of the Cactus or ET websites along with a WaveDemo tutorial, I'm afraid to say that many of these are unfortunately very out of date, which is likely the reason for these errors. -- Ian Hinder http://members.aei.mpg.de/ianhin ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] Path ET_2011_10
> On 30 Nov 2017, at 04:42, Camilo Alexander Barbosa Doncel > wrote: > > thanks for the answer, > > Yes, is that I need to execute a file of parameters that was made in 2008 and > that in this version does not execute. > I need to run Schwarzschild with attachment of parameters file. Hi, I would strongly recommend trying to get it to work with the current release of the ET instead. The ET is highly backward compatible, and if it doesn't work straight away, it is very likely possible to tweak a couple of parameters and make it work. The old thornlists from 2008 refer to old code repository servers which we do not use any more, and which may not even exist. The code has since been migrated to new servers (hosted on BitBucket). Since the code is stored in git, it is always possible to check out an old version, and if absolutely necessary for a strict reproducibility test, we could probably get the 2008 version of the toolkit running. But this will be a large amount of work. It's probably much less work to get it working with the current release. What error message do you get if you try to use the current release? -- Ian Hinder http://members.aei.mpg.de/ianhin ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] Path ET_2011_10
> On 29 Nov 2017, at 05:52, Camilo Alexander Barbosa Doncel > wrote: > > Hi, > I need to install version 4.0.0 of the Cactus but when I type the command > GetComponents --parallel > https://bitbucket.org/einsteintoolkit/manifest/raw/ET_2011_10/einsteintoolkit.th > > <https://bitbucket.org/einsteintoolkit/manifest/raw/ET_2011_10/einsteintoolkit.th> > asks me to "please enter the path to hg:" and not what is the path... Dear Camilo, It wants you to have Mercurial installed. You would need to install Mercurial. That is an ancient version of the toolkit; is there a reason that you are not using the latest released version, from http://einsteintoolkit.org/download.html <http://einsteintoolkit.org/download.html> ? We don't have the resources to support old versions of the toolkit - we can only really support the current release. The latest release no longer requires Mercurial. -- Ian Hinder http://members.aei.mpg.de/ianhin ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] meeting minutes for 2017-11-27
> On 27 Nov 2017, at 17:33, Steven R. Brandt wrote: > > Roland, I think the new-user-tutorial.html looks great (though it might > be nicer with a small margin around the images). I suggest we link it > from the home page as the simplified tutorial and I start emailing our > users. Are we yet in a position where we can have just a single tutorial, to avoid confusion? -- Ian Hinder http://members.aei.mpg.de/ianhin ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] Error in thorn Vectors
> On 14 Nov 2017, at 01:12, Hamilton, Maria wrote: > > Hello, > > Yes, previously I had the error in thorns Vectors, and I followed the > suggestion to use the current stable version of Cactus. > > I am not using simfactory, but this command: > make ET-config options=simfactory/mdb/optionlists/osx-homebrew.cfg > THORNLIST=thornlists/einsteintoolkit.th > > The error is: > checking for M_PI... no > configure: error: M_PI not defined. Try adding -D_XOPEN_SOURCE to CPPFLAGS. > > Here is what I tried to add to CPPFLAGS: > -D_USE_MATH_DEFINES > -D_XOPEN_SOURCE > -DM_PI = 3.14149265358979323847 > > I also tried to change c11 to c99 at CFLAGS and add > -stdlib=libc++ at LIBCXX. > > None of those worked. > > Please let me know what is to be done. Hi Maria, Have you installed all the homebrew packages listed at the top of osx-homebrew.cfg? -- Ian Hinder http://members.aei.mpg.de/ianhin ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] About the thorn Einstein exact
On 7 Nov 2017, at 17:37, Nisa Amir wrote:
> I am just trying to add this spacetime, I dont want to use it in some other
> thorns of the toolkit. and I am not familiar with xAct package.
Hi,
The below patch modifies EinsteinExact so that it is able to generate thorns
from metrics which are in spherical polar coordinates. It achieves this by
mapping the polar coordinate r,th,ph to the CartGrid3D x,y,z required by
ADMBase. You can use this to generate the existing Schwarzschild metric from
the Metrics package. Note that this means that your x, y, z coordinates in the
simulation are really r, th and ph, and that much of the rest of the toolkit
will not know how to deal with this, but if you really don't want to use any
other thorns, then that shouldn't be a problem.
diff --git a/m/EinsteinExact.m b/m/EinsteinExact.m
index 80eaff2..93b7fff 100644
--- a/m/EinsteinExact.m
+++ b/m/EinsteinExact.m
@@ -247,11 +247,12 @@ idThorn[spacetime_, thorn_] :=
Print["Generating thorn ", thorn, " for ", spacetime, " spacetime."];
(* Load the spacetime: coordinates, metric, inverse metric *)
- coordRule = {t -> T, x -> X, y -> Y, z -> Z, None -> {}};
+ coordRule = {t -> T, x -> X, y -> Y, z -> Z, None -> {},
+r -> X, \[Theta] -> Y, \[Phi] -> Z};
coords = MetricProperty[spacetime, "Coordinates"] /. coordRule;
If[coords =!= {T, X, Y, Z},
-Throw["Error, only metrics in Cartesian coordinates are supported"];
+Throw["Error, only metrics in Cartesian or spherical polar coordinates are
supported"];
];
spatialCoords = coords[[2;;]];
@@ -456,7 +457,8 @@ g_{ab} =
StringForm[doctext]
]
-spacetimes = {"GaugeWave", "KerrSchild", "Minkowski", "ShiftedGaugeWave",
"Vaidya", "ModifiedSchwarzschildBL"};
+(* spacetimes = {"GaugeWave", "KerrSchild", "Minkowski", "ShiftedGaugeWave",
"Vaidya", "ModifiedSchwarzschildBL"}; *)
+spacetimes = {"Schwarzschild"};
thornNameRules = {"Vaidya" -> "Vaidya2"};
thorns = spacetimes /. thornNameRules;
--
Ian Hinder
http://members.aei.mpg.de/ianhin
___
Users mailing list
Users@einsteintoolkit.org
http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] About the thorn Einstein exact
On 7 Nov 2017, at 14:04, Nisa Amir wrote: > Yes, the mathematica package is given but it accepts the metric only in > cartesian coordinates. I want to add the spacetime non kerr which is in polar > coordinates to the mathematica package in Einstein Exact thorn. What > transformations should I made? Hi, You can apply the usual basis transformations in Mathematica to generate the metric in a quasi-Cartesian basis and coordinates. This will then allow you to construct the spacetime numerically in these coordinates, and the thorns in the toolkit which expect quasi-Cartesian coordinates will just work, for example the horizon finder. You would then evolve in 3+1 dimensions. I have done this for Kerr in Boyer-Lindquist coordinates using the xAct tensor manipulation package. This is not straightforward, and it's not something that I would take on lightly if you don't have much experience with EinsteinExact or xAct. However, you have said that you want to store the gridfunctions in polar coordinates, and do the evolution in polar coordinates. I have no experience with this, and many of the thorns in the toolkit which expect quasi-Cartesian coordinates will just not work (e.g. the horizon finder). That is why I asked you what you are trying to do. Please can you answer that, before we go into a lot of detail about how to do it in one particular way, which may in the end not help you? Thanks! -- Ian Hinder http://members.aei.mpg.de/ianhin ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] Joining users email list
On 7 Nov 2017, at 00:44, Roland Haas wrote: > Hello Sam, > >>This is Samuel Cupp from LSU. I would like to be added to the >> users email list. I was told to just email the list to get that to >> happen. This is the email address I would like added. Thanks in >> advance! > The list only lets members post so the fact that you could post would > mean you are already a member (or one of the admins ok'ed the post). Hi Roland, Is that really the case? The instructions at http://einsteintoolkit.org/support.html say The Einstein Toolkit Consortium is aiming to cultivate a self-supporting and open community and we encourage you to write to the users@einsteintoolkit.org mail list with your questions about the use of the toolkit. It doesn't say anything about subscribing, or needing to. Can someone who is an administrator of the list check to see what the setting is? If people need to subscribe, then we should give instructions on the web page. -- Ian Hinder http://members.aei.mpg.de/ianhin signature.asc Description: Message signed with OpenPGP using GPGMail ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] meeting minutes for 2017-11-06
On 6 Nov 2017, at 17:31, Ian Hinder wrote: > > On 6 Nov 2017, at 17:06, Roland Haas wrote: > >> * 6 failing tests >> ** prolongation one is new, need input from Erik, want trac ticket > > It turns out that this was already discussed in the original ticket for the > feature, https://trac.einsteintoolkit.org/ticket/2068, and was fixed in a > pull request > https://bitbucket.org/eschnett/carpet/pull-requests/19/correct-error-in-vertex-centred/diff, > which was reviewed as OK by Roland in > https://trac.einsteintoolkit.org/ticket/2074, but never applied. I just > applied it. Hopefully it will fix the problems. It did. We are now down to three failing tests, all in SphericalHarmonicRecon and SphericalHarmonicReconGen. -- Ian Hinder http://members.aei.mpg.de/ianhin signature.asc Description: Message signed with OpenPGP using GPGMail ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] meeting minutes for 2017-11-06
On 6 Nov 2017, at 17:06, Roland Haas wrote: > * 6 failing tests > ** prolongation one is new, need input from Erik, want trac ticket It turns out that this was already discussed in the original ticket for the feature, https://trac.einsteintoolkit.org/ticket/2068, and was fixed in a pull request https://bitbucket.org/eschnett/carpet/pull-requests/19/correct-error-in-vertex-centred/diff, which was reviewed as OK by Roland in https://trac.einsteintoolkit.org/ticket/2074, but never applied. I just applied it. Hopefully it will fix the problems. > ** SphericalHarmonicReconGen: could be bug in the reader or in PITTNull > code. PITTNull is sensitive to optimization since it includes the full > Einstein field equations. Check if using -O3 instead of -Ofast would > help. > ** have generic.cfg and generic-opt.cfg, get rid of debian.cfg, > ubuntu.cfg etc. I would suggest that generic.cfg should be more conservative, since it is the default probably used by new users, and we don't want them running into edge cases caused by aggressive optimisations. Can we reduce the optimisation level in generic.cfg to -O2? Then we could have a generic-no-opt or something, which really had no optimisations at all. -- Ian Hinder http://members.aei.mpg.de/ianhin signature.asc Description: Message signed with OpenPGP using GPGMail ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] About the thorn Einstein exact
On 5 Nov 2017, at 09:28, Nisa Amir wrote:
> Actually, when in the einsteinexact thorn there is a mathematica package
> named EinsteinExact. The documentation said that if you want to add some new
> space time add the space time in that mathematica package. I want to add non
> kerr which is in polar coordinates to that package, so that it generates a
> new thorn but I dont know how to do that.
> Kindly guide me.
Hi,
Yes, you can add a new spacetime to the Metrics package (in
Cactus/arrangements/EinsteinExact/m/Metrics/metrics). See the examples in
there. For example, Schwarzschild.m:
(* ::Package:: *)
{
"Name" -> "Schwarzschild",
"Description" -> "Schwarzschild spacetime",
"Dimensions" -> 4,
"Coordinates" -> {t, r, \[Theta], \[Phi]},
"Parameters" -> {M},
"Metric" -> {{-1 + 2 M / r, 0, 0, 0},
{0, 1/(1 - 2 M / r), 0, 0},
{0, 0, r^2, 0},
{0, 0, 0, r^2 Sin[\[Theta]]^2}},
"SignDet" -> -1
}
The EinsteinExact package then uses this information to generate an exact
solution thorn, which can be used for initial data in a numerical relativity
evolution. However, it only supports metrics which are explicitly given in
Cartesian coordinates, so for example, the Schwarzschild example won't work
with it (the Metrics package is generic, and is used also in other contexts,
outside EinsteinExact).
There is a check in arrangements/EinsteinExact/m/EinsteinExact.m that the
metric is in Cartesian coordinates. I suspect that the only reason for this
check is that it wouldn't know what Cactus gridfunctions to use for anything
else.
Since you want to do the evolution in polar coordinates, I assume that you are
going to have some mapping between x, y and z, and r, th and ph? I must admit
I have never tried to do something like this.
There are people on this list who have done evolutions in polar coordinates;
maybe one of them could give some advice about whether what you are trying to
do is feasible? Maybe you could give more details about what you are trying to
do?
PS: *please* include users@einsteintoolkit.org in the CC when you reply. If
you reply just to me, then nobody else benefits from the discussion, and nobody
else has the opportunity to help you.
--
Ian Hinder
http://members.aei.mpg.de/ianhin
___
Users mailing list
Users@einsteintoolkit.org
http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] Cactus Tutorial
On 3 Nov 2017, at 19:28, Steven R. Brandt wrote: > The attached file is the html rendering of a Jupyter notebook version of > Cactus tutorials advocated by Ian and Roland. It is my proposal to use this > going forward. > > To run this tutorial, you can either go to the NDS machine (which I propose > should be the default tutorial machine from now on): > https://www.einsteintoolkit.nationaldataservice.org/#/ > > Alternatively, you may run it in docker. > > docker run --name myjupyter -d -p : ndslabs/jupyter-et > > If you don't like Jupyter, you can open a console on the NDS machine, or you > can get a command line interface by doing this > > docker run -it ndslabs/jupyter-et bash > > Which is a machine that already has the requisite cactus packages installed. > You can then use the attached html to enter instructions one by one. > > Please send comments, flames, etc. Thanks. Hi Steve, Very nice! One comment: - Users probably would like, for their operating system, a single command that they can copy and paste into their terminal to install the required packages, rather than going through each of the 23 packages one by one. Also, the table format, while quite attractive, won't scale to adding columns for more operating systems, such as Mac OS. One question: - Is this tutorial in a repository somewhere? -- Ian Hinder http://members.aei.mpg.de/ianhin ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] Cannot access x,y,z,r when inheriting grid in c
On 3 Nov 2017, at 09:22, Chris Stevens wrote:
> Hi Ian,
>
> thanks for the quick reply.
>
> SCHEDULE WorldTubeExtract_TEST AT CCTK_INITIAL AFTER
> WorldTubeExtract_RegisterSlices
> {
> LANG: C
> OPTIONS: LEVEL
> } "TEST"
>
> Also the output of Carpet::is_global_mode() is 0 so definitely not in global
> mode.
>
Hi,
The problem is that it is in level mode ("OPTIONS: LEVEL"). Level mode is
called once per refinement level, and on each refinement level, there may be
multiple components (rectangular blocks of grid points). A function scheduled
in level mode can access quantities which are defined on a given refinement
level (or, for example, call interpolation or reduction functions for that
level), but not those that are defined on a given component, for example
accessing grid data like gridfunctions. When you schedule a function in local
mode ("OPTIONS: LOCAL"), it is called once per component, and then you can
access grid data.
I spent hours recently trying to track down exactly this problem!
--
Ian Hinder
http://members.aei.mpg.de/ianhin
___
Users mailing list
Users@einsteintoolkit.org
http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] Cannot access x,y,z,r when inheriting grid in c
On 3 Nov 2017, at 09:06, Chris Stevens wrote: > Hi all, > > this hopefully is a dumb question, but I cannot for the life of me get access > to the x,y,z,r coordinates defined in CartGrid3D. > > Suppose I have a thorn X that wants to access these coordinates from test.cc, > then the thorn should inherit: grid, the .cc should #include > "cctk_Arguments.h" and the line that accesses, say x[0], should be called > after DECLARE_CCTK_ARGUMENTS. > > Unfortunately doing this results in an error: > > [maths02:86267] *** Process received signal *** > [maths02:86267] Signal: Segmentation fault (11) > [maths02:86267] Signal code: Address not mapped (1) > [maths02:86267] Failing at address: (nil) > Which is analogous to the error message you get by defining a grid variable > in interface.ccl without the accompanying STORAGE: in schedule.ccl, i.e. the > memory is not allocated. > I am using Llama as my coordinate system, but I know this basically is a > wrapper for CartGrid3D and these coordinates are still set. I see the > printout from CartGrid3D. > I try to access these coordinates at, for example, CCTK_INITIAL, when the > grid structure is already setup. I have also tried later time bins such as > CCTK_POSTSTEP. > > Example files that access these coordinates are: > > CTGBase::debug.cc > > ADMDerivatives::calc_derivs.cc > > Coordinates::cylinderinbox.cc > > However I am not having any luck reverse-engineering these. > Any ideas would be appreciated. > Hi, Thanks for the detailed report! What mode are you calling your function in? i.e. can you show us the block in schedule.ccl that schedules your function? If it is not local mode (the default), then you won't be able to access any grid data, and you will get a segfault. -- Ian Hinder http://members.aei.mpg.de/ianhin ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] Carpet options to switch off mesh refinement
On 2 Nov 2017, at 11:20, dumsani wrote: > > > On 02/11/2017 10:46, Ian Hinder wrote: >> On 2 Nov 2017, at 01:41, dumsani wrote: >> >>> I think I have figured it out eventually. Thanks, Ian. >> Good to hear! Would it be useful to summarise the cause of the problem to >> the mailing list, in case it helps others? >> > > The following 3 options were enough, as you suggested. > Carpet::max_refinement_levels = 1 > Carpet::refinement_factor = 1 > CarpetRegrid2::num_centres = 0 > > After setting these properly, it turned out the crash wasn't due to Carpet. > It was now due to inappropriate memory allocation by the scheduler in my > initial data code. Hi, The refinement_factor shouldn't need to be set. You can achieve a unigrid setup either by setting num_centres to 0, or by setting the number of levels in each centre to 1. -- Ian Hinder http://members.aei.mpg.de/ianhin ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] Carpet options to switch off mesh refinement
On 1 Nov 2017, at 19:10, dumsani wrote: > Hi, > > I am to run a simulation with Carpet in "ungrid" mode, that is, without > mesh refinement. I don't seem to get it right. What's options do I need > to set? Apart from setting Carpet::max_refinement_levels = 1, what else > does one have to specify in orer to get a run with only one grid? Hi Dumsani, What goes wrong? Do you get an error message? You don't even have to set max_refinement_levels to 1; you just have to make sure you don't ask for any more than 1 refinement level from CarpetRegrid2. -- Ian Hinder http://members.aei.mpg.de/ianhin ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] About the thorn Einstein exact
On 1 Nov 2017, at 16:43, Nisa Amir wrote: > I want to do the evaluation in polar coordinates. Please include users@einsteintoolkit.org in CC, so that other benefit from the discussion. I am still not sure what you mean: I was asking about the *evolution*, i.e. the solution of the Einstein evolution equations from one time to another time. Do you want to do this in polar coordinates? Or just the "evaluation" of the initial data, where you have polar coordinates available. -- Ian Hinder http://members.aei.mpg.de/ianhin ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] About the thorn Einstein exact
On 31 Oct 2017, at 17:16, Nisa Amir wrote: > Yes exactly, I want to add a space time non kerr to Einstein Exact thorn, but > as you have already said, it considers only cartesian coordinates. What > should I do ? (Please include users@einsteintoolkit.org in CC, so that other benefit from the discussion) Hi, I'm afraid I still don't understand which of the two options you want. Do you want to perform the evolution in polar or Cartesian coordinates? -- Ian Hinder http://members.aei.mpg.de/ianhin ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] About the thorn Einstein exact
On 30 Oct 2017, at 03:03, Nisa Amir wrote: > Hello, > > I am new to Einstein toolkit and m doing my resarch on it for M.phil > dissertion. Can you please guide me how to add a metric in polar coordinates > in Einstein exact thorn in Einstein toolkit and how to generate a new thorn > for this metric using kranc package. > your help will be highly appreciated. Hi, (resent with CC to ET list) Apologies for the long delay in replying. Can you give a bit of background about what you want to do? There are two possibilities. First, you can have a numerical grid in r, th and ph. Since the evolution equations are spatially covariant (almost; the gauge conditions are not always), you can do this, but it might not be what you really want. Most of the Einstein Toolkit assumes that you have an x,y,z type coordinate system, so to make use of it, you probably want to express your initial data in Cartesian-type coordinates. So, for example, if you had Kerr in Boyer-Lindquist coordinates, you would first change coordinates and basis using the usual transformation, and then evaluate the metric and extrinsic curvature on the x,y,z grid. Is that what you want to do? -- Ian Hinder http://members.aei.mpg.de/ianhin ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] defining a grid function constant in time
On 27 Oct 2017, at 10:28, Miguel Zilhão
wrote:
> hi Ian,
>
>>> for the purposes of trying a particular gauge condition, we'd like to have
>>> a grid function (let's
>>> call it "shift_t0") that's set to be equal to the shift at t=0. we don't
>>> want this function to
>>> evolve at all; we just want this grid function to be accessible at all
>>> times (with exactly the same
>>> values).
>>>
>>> is there something in particular that we need to watch out for, when doing
>>> this? we ask because
>>> we've noticed that when doing things in a "naive"* way, sometimes NaNs
>>> start developing at some
>>> point in the evolution (seemingly near refinement boundaries and when
>>> regriding). we noticed that
>>> exactly the same thing happens for the "puncture_u" grid function in
>>> TwoPunctures: this function is
>>> only used at t=0; however, if one activates storage for it and asks for its
>>> output, at some point
>>> NaNs start developing in the output, even though the function is never used
>>> again. so, what would be
>>> the standard way of defining a grid function that is constant in time?
>>>
>>> thanks,
>>> Miguel & Helvi
>>>
>>> * by "naive", we mean essentially the following
>>>
>>> in schedule.ccl we have
>>>
>>> STORAGE: shift_t0[1]
>>>
>>> in interface.ccl we have
>>>
>>> CCTK_REAL shift_t0 type=gf timelevels=1 tags='tensortypealias="U"
>>> tags='prolongation="none"'
>>> {
>>> betax_t0 betay_t0 betaz_t0
>>> } "shift vector at t=0"
>>>
>>> and we set its value (on the whole grid) at a function call at CCTK_INITIAL
>>> after ADMBase_PostInitial
>> Does this happen only after regridding? By not prolongating the refinement
>> boundaries, it's not clear how the "new" bits of the grid are supposed to be
>> filled when the fine grids move. Could this be the problem? If you give
>> the gridfunction 3 timelevels and remove "prolongation=none", do the NaNs go
>> away?
>
> it seems to happen only after regridding, yes. we did try precisely that
> (using 3 timelevels and prolongating) but the NaNs still appeared...
The past timelevels need to be initialised. By default, Carpet will poison
them with NaNs so that you can see when undefined values are used.
Theoretically, if there is no time evolution of this variable, you should be
able to use a single timelevel and use a tag 'prolongation="copy"', which
should perform only spatial prolongation. Let me know what happens if you try
this. I expect you will get an assertion failure from Carpet, and then you
might want to try cherry-picking this commit:
cd repos/Carpet
git cherry-pick d13a6e245a3d7d2b575094fd0def540d510a166c
which disables some code in Carpet that disables spatial prolongation in the
case of "op_copy". The problem here likely resulted in confusion due to the
operator being badly named (https://martinfowler.com/bliki/TwoHardThings.html).
In arrangements/Carpet/CarpetLib/src/operators.hh, the implication is that the
"copy" in op_copy refers to the time interpolation only, but the code in
patched in the above commit assumes that "copy" means don't interpolate at all.
It's not clear how this can make sense when the data comes from another
refinement level, which will always have a different resolution, but maybe
there is some case where this would have made sense.
Erik, should this commit be merged to master?
--
Ian Hinder
http://members.aei.mpg.de/ianhin
___
Users mailing list
Users@einsteintoolkit.org
http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] defining a grid function constant in time
On 26 Oct 2017, at 23:23, Miguel Zilhão
wrote:
> hi all,
>
> for the purposes of trying a particular gauge condition, we'd like to have a
> grid function (let's
> call it "shift_t0") that's set to be equal to the shift at t=0. we don't want
> this function to
> evolve at all; we just want this grid function to be accessible at all times
> (with exactly the same
> values).
>
> is there something in particular that we need to watch out for, when doing
> this? we ask because
> we've noticed that when doing things in a "naive"* way, sometimes NaNs start
> developing at some
> point in the evolution (seemingly near refinement boundaries and when
> regriding). we noticed that
> exactly the same thing happens for the "puncture_u" grid function in
> TwoPunctures: this function is
> only used at t=0; however, if one activates storage for it and asks for its
> output, at some point
> NaNs start developing in the output, even though the function is never used
> again. so, what would be
> the standard way of defining a grid function that is constant in time?
>
> thanks,
> Miguel & Helvi
>
> * by "naive", we mean essentially the following
>
> in schedule.ccl we have
>
> STORAGE: shift_t0[1]
>
> in interface.ccl we have
>
> CCTK_REAL shift_t0 type=gf timelevels=1 tags='tensortypealias="U"
> tags='prolongation="none"'
> {
> betax_t0 betay_t0 betaz_t0
> } "shift vector at t=0"
>
> and we set its value (on the whole grid) at a function call at CCTK_INITIAL
> after ADMBase_PostInitial
Does this happen only after regridding? By not prolongating the refinement
boundaries, it's not clear how the "new" bits of the grid are supposed to be
filled when the fine grids move. Could this be the problem? If you give the
gridfunction 3 timelevels and remove "prolongation=none", do the NaNs go away?
--
Ian Hinder
http://members.aei.mpg.de/ianhin
___
Users mailing list
Users@einsteintoolkit.org
http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] compilation on marenostrum 4
On 19 Oct 2017, at 14:26, hwitek wrote:
> Hi everyone,
>
> we could compile the toolkit on MareNostrum4 without any further
> problems ("unset MPI" did the trick), and I want to run some further
> tests before providing the configuration script.
>
> What is the standard way to include new configuration files into the
> toolkit? Should I send it to someone in particular?
Hi,
Geraint recently (yesterday) sent me simfactory files for marenostrum4, and I
was going to commit them.
Now we have two versions it seems, so I don't know what to do...
For submissions, you can:
1. Ask on the list;
2. Create a bitbucket pull request; or
3. Create a ticket on TRAC.
Up to you! Probably (2) is the preferred option.
--
Ian Hinder
http://members.aei.mpg.de/ianhin
___
Users mailing list
Users@einsteintoolkit.org
http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] Einstein Toolkit Meeting Reminder
On 14 Oct 2017, at 06:38, Steven R. Brandt wrote: > Hi > > Please consider joining the weekly Einstein Toolkit phone call at > 9:30 am US central time on Mondays. For details on how to connect > and what agenda items are to be discussed, use the link below. > > > https://docs.einsteintoolkit.org/et-docs/Main_Page#Weekly_Users_Call Hi Steve, The call this week conflicts with the LIGO press conference, which is at 10:00 am EST (9:00 am CT). I expect many people will not attend; should we just cancel the call? -- Ian Hinder http://members.aei.mpg.de/ianhin ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Re: [Users] Jenkins down due to suspected security compromise
On 3 Jul 2017, at 17:32, Ian Hinder wrote: > > On 20 Jun 2017, at 13:45, Ian Hinder wrote: > >> On Fri, Jun 2, 2017 at 11:25 AM, Ian Hinder wrote: >> >>> Hi all, >>> >>> The security team at NCSA have blocked access to the ET Jenkins server due >>> to a suspected security compromise. We are investigating. >>> >>> If you have in the past configured a jenkins build node which can be >>> accessed from the jenkins master via ssh (i.e. you have added the jenkins >>> public ssh key to an authorized_keys file), then you should immediately >>> remove this key. >>> >>> Note that none of the jenkins build nodes apart from the one also hosted at >>> NCSA was working at the time, so it's unlikely that any further attack was >>> possible to those machines. >>> >>> We have backups from before the incident, so assuming we can fix the >>> vulnerability, we should be able to get the system up and running in a few >>> days. >> >> Hi, >> >> A quick update: >> >> I have recreated the Jenkins master and build nodes from backups, and have >> the new machines running. I am still waiting to hear from the NCSA security >> team concerning exactly what the vulnerability was. I can't make Jenkins >> available publicly until we are confident that the vulnerability is not >> still exposed. > > Hi all, > > We are fairly sure that the problem was due to a security flaw in Jenkins > itself, which was exploited to install a bitcoin miner > (seehttps://security.stackexchange.com/questions/160068/kworker34-malware-on-linux,). > Apart from the presence of this miner, we can't find any other indication > of a problem. Nevertheless, we have restored from backup and upgraded Jenkins. > > This flaw was announced and patched on 26-Apr-2017 > (https://jenkins.io/security/advisory/2017-04-26/), but Jenkins was not being > updated automatically on this machine (other ubuntu security updates were > being applied automatically, but Jenkins was obtained from the Jenkins PPA, > and not included in the unattended-upgrades list). We have added Jenkins to > unattended-upgrades. > > There are a number of outdated plugins which should also be upgraded to be > safe, but these are not all backward compatible, so this needs to be done > with care (and backups). I don't want to make Jenkins accessible publicly > until this is done. > > For ET members with ssh accounts on login.barrywardell.net, you can access > Jenkins using > > ssh -L 8080:192.168.0.28:443 -Nv login.barrywardell.net > > and browsing to https://localhost:8080. You will need to agree to the > mismatched SSL certificate. Hi all, The Einstein Toolkit build and test system "Jenkins" server is online and accessible again via https://build-test.barrywardell.net We still have the 6 failures (https://build-test.barrywardell.net/job/EinsteinToolkit/lastCompletedBuild/testReport/) which were previously reported. I have: - Recovered the build machine from a backup from before the intrusion - Reset the ssh host key - Regenerated the Jenkins ssh private key - Upgraded the OS to Ubuntu 16.04 LTS - Upgraded Jenkins to the latest version - Ensured that automatic security updates are applied to Jenkins (previously they were not, due to an oversight) - Upgraded all plugins to the latest versions - Reset all system and Jenkins passwords If you have a Jenkins account and would like to know the new password, please let me know. -- Ian Hinder http://members.aei.mpg.de/ianhin ___ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
