Yosef, James I'd like to at least try to see whether the McLachlan performance you're seeing is in the right ballpark. Can you post the parameter file you're using?
-erik On Tue, Jul 28, 2015 at 9:19 AM, Yosef Zlochower <[email protected]> wrote: > Hi Erik, > > Our standard choice is to use 8th order centered > differencing with 5th-order dissipation. For this > test, Jim used 8th-order with upwinding and > thus used 5 ghost zones (rather than 4). > We use the full complement of 4*ghost buffer > zones. Near the boundary we reduce the order of the > stencil from 8 to 6 to 4 to 2 and on the boundary > we apply Sommerfeld boundary conditions (implemented as > modifications to RHS). > > For the gauge, we evolve > d0 alpha = -2 alpha K > and > dt beta = 3/4 Gamma^i > (we don't introduce the auxiliary B field). > > We precalculate the mixed derivatives and > only calculate first derivatives and non-mixed second > derivatives in the main loop. We apply > dissipation to all variables, but apply it > one variable at a time (i.e., we go through > a full 3d loop nvars times). > > > The conformal factor is W = sqrt(chi). > > We use the admbase variables alpha and beta > and fill in kij and gij after each ministep > of the RK4 integrator. We similarly enforce > the algebraic constraints at each ministep. > We calculate the constraints using the BSSN > variables, but use a 4th order stencil and implement > mixed derivatives using one dimensional derivatives > along the diagonal (this is less accurate, but > faster). > > > The FD order is a compile time option and > we can test the code with and without upwinding, > with FD orders of 2nd through 8, and we can > use the standard or "quick" mixed derivatives > for the constraint. We can similarly use > 3, 5, 7, 9 order dissipation. > > We use openmp parallelization along the "z" direction > of our loops > > > > > On 07/27/2015 10:21 PM, Erik Schnetter wrote: > >> Jim >> >> By default, McLachlan is configured to be generic, i.e. many choices can >> be made at run time. For example, there are 4 additional gauge variables >> evolved (A and B^i). If you are willing to restrict your run-time >> choices, for example by fixing derivative order, disabling multi-block >> systems, choosing a particular gauge, choice of conformal factor, etc., >> then this will lead to a more efficient code. You can easily do so for >> McLachlan by generating a new version in Kranc. If you look in the >> beginning of McLachlan_BSSN.m in the section "Choose code to generate", >> then you'll see the parameters you can choose. We have made a particular >> set of choices for a configuration "BSSN_bench", but these are >> apparently different from LazEv. For example, we use 4th order finite >> differencing, and evolve B^i in time as well. >> >> Can you specify all the details of your formulation? I notice that you >> didn't respond to my request to point me to the source code, and there >> are quite a few details that you also didn't describe yet (and are also >> not described in your paper), such as your choice of conformal factor, >> whether/how your code is integrated with ADMBase, whether the >> constraints are calculated from the BSSN or the ADM variables, how >> advection derivatives are handled, etc. For a true comparison, this all >> needs to be known. Unless we both know exactly what both the codes do, >> we'll never know where the physics differences are. >> >> -erik >> >> >> On Mon, Jul 27, 2015 at 5:39 PM, James Healy <[email protected] >> <mailto:[email protected]>> wrote: >> >> Hi Erik, >> >> We evolve alpha and beta for the gauge variables as given in >> equations 7a and 7b of http://arxiv.org/pdf/1506.06153.pdf. The >> number I reported for LazEv was using 5th order dissipation applied >> to all 21 evolved variables (alp, beta^i, At_{ij}, gt_{ij}, >> Gammat^{i}, trK, and xi). >> >> LazEv is calculating the Hamiltonian and Momentum constraints as >> well as the constraints on Gammat^i using 4th order stencils. >> >> I also forgot to mention that I did these tests on 16 nodes, and the >> memory usage was relatively low ( ~2 GB/MPI process ). >> >> I will give the internal dissipation for McLachlan a try. >> >> Thanks, >> Jim >> >> >> On 07/27/2015 05:56 PM, Erik Schnetter wrote: >> >>> Jim >>> >>> Thanks for posting the details. >>> >>> Can you give us more details about the LazEv scheme? In particular >>> there may be differences in the gauge. What (gauge) variables do >>> you evolve? What gauge conditions do you use? And what kind of >>> dissipation do you apply? Can you point us to the source code? >>> >>> For the new McLachlan, you would probably use the built-in >>> dissipation instead of thorn Dissipation, which should lead to a >>> small speed-up. >>> >>> -erik >>> >>> On Mon, Jul 27, 2015 at 4:43 PM, James Healy <[email protected] >>> <mailto:[email protected]>> wrote: >>> >>> Hello, >>> >>> I have been running some tests on Stampede comparing the run >>> speed of McLachlan to RIT's evolution thorn LazEv. I started >>> with the qc0-mclachlan.par parameter file included with the >>> Einstein Toolkit, added a few refinement levels, increased the >>> resolution and changed McLachlan to be 8th order (and >>> increased the number of ghost zones to 5). I also increased >>> the initial separation so the finest grids aren't already >>> overlapping. To compare with LazEv, I removed the McLachlan >>> and Dissipation thorns and replaced them with LazEv. >>> Everything else in the parameter file is exactly the same. I >>> tried using both the McLachlan master and rewrite branches. >>> >>> The grid setup is 10 levels of refinement, dx=4M on the >>> coarsest with outer boundary at 400M, M/128 on the finest with >>> r=0.6M, CFL is 0.25. Both use 8th order spatial differencing >>> with ghost_size=5 and 5th order dissipation. >>> >>> Below is a summary of the results as reported at iteration 256 >>> from Carpet::physical_time_per_hour: >>> >>> McLachlan - rewrite branch: 3.0596110 M/hr >>> McLachlan - master branch: 3.8033607 M/hr >>> LazEv - 4.1941544 M/hr >>> >>> I am using the stampede-impi.cfg configuration file in >>> simfactory. "module list" returns: >>> >>> 1) TACC-paths 3) cluster-paths 5) xalt/0.4.6 7) TACC >>> 2) Linux 4) intel/13.0.2.146 <http://13.0.2.146> 6) >>> cluster 8) impi/4.1.0.030 <http://4.1.0.030> >>> >>> >>> Attached is my parameter file. I pasted the McLachlan >>> parameters below. Are there any optimizations that I can use >>> for McLachlan? Are the parameters I am using for it what would >>> be used for production runs? >>> >>> ML_BSSN::harmonicN = 1 # 1+log >>> ML_BSSN::harmonicF = 2.0 # 1+log >>> ML_BSSN::ShiftGammaCoeff = 0.75 >>> ML_BSSN::BetaDriver = 1.0 >>> ML_BSSN::LapseAdvectionCoeff = 1.0 >>> ML_BSSN::ShiftAdvectionCoeff = 1.0 >>> >>> ML_BSSN::MinimumLapse = 1.0e-8 >>> >>> ML_BSSN::my_initial_boundary_condition = "extrapolate-gammas" >>> ML_BSSN::my_rhs_boundary_condition = "NewRad" >>> Boundary::radpower = 2 >>> >>> ML_BSSN::ML_log_confac_bound = "none" >>> ML_BSSN::ML_metric_bound = "none" >>> ML_BSSN::ML_Gamma_bound = "none" >>> ML_BSSN::ML_trace_curv_bound = "none" >>> ML_BSSN::ML_curv_bound = "none" >>> ML_BSSN::ML_lapse_bound = "none" >>> ML_BSSN::ML_dtlapse_bound = "none" >>> ML_BSSN::ML_shift_bound = "none" >>> ML_BSSN::ML_dtshift_bound = "none" >>> >>> ML_BSSN::fdOrder = 8 >>> >>> ActiveThorns = "Dissipation" >>> >>> Dissipation::order = 5 >>> Dissipation::vars = " >>> ML_BSSN::ML_metric >>> ML_BSSN::ML_trace_curv >>> ML_BSSN::ML_curv >>> ML_BSSN::ML_Gamma >>> ML_BSSN::ML_lapse >>> ML_BSSN::ML_shift >>> ML_BSSN::ML_dtlapse >>> ML_BSSN::ML_dtshift >>> " >>> >>> ActiveThorns = "ML_ADMConstraints" >>> >>> >>> Thanks, >>> Jim Healy >>> >>> _______________________________________________ >>> Users mailing list >>> [email protected] <mailto:[email protected]> >>> http://lists.einsteintoolkit.org/mailman/listinfo/users >>> >>> >>> >>> >>> -- >>> Erik Schnetter <[email protected] <mailto:[email protected] >>> >> >>> http://www.perimeterinstitute.ca/personal/eschnetter/ >>> >> >> >> >> >> -- >> Erik Schnetter <[email protected] <mailto:[email protected]>> >> http://www.perimeterinstitute.ca/personal/eschnetter/ >> >> >> _______________________________________________ >> Users mailing list >> [email protected] >> http://lists.einsteintoolkit.org/mailman/listinfo/users >> >> > > -- > Dr. Yosef Zlochower > Center for Computational Relativity and Gravitation > Associate Professor > School of Mathematical Sciences > Rochester Institute of Technology > 85 Lomb Memorial Drive > Rochester, NY 14623 > > Office:74-2067 > Phone: +1 585-475-6103 > > [email protected] > > CONFIDENTIALITY NOTE: The information transmitted, including > attachments, is intended only for the person(s) or entity to which it > is addressed and may contain confidential and/or privileged material. > Any review, retransmission, dissemination or other use of, or taking > of any action in reliance upon this information by persons or entities > other than the intended recipient is prohibited. If you received this > in error, please contact the sender and destroy any copies of this > information. > -- Erik Schnetter <[email protected]> http://www.perimeterinstitute.ca/personal/eschnetter/
_______________________________________________ Users mailing list [email protected] http://lists.einsteintoolkit.org/mailman/listinfo/users
