Dear ARTS developers,

The infrastructure for ARTS line-by-line data will change significantly
with an upcoming update to the development version of ARTS.  Your
simulations will be broken before you adapt to the new infrastructure.  I
will use this email to explain what updates are required to continue your
simulations as before.  I will first justify the necessity of the update.
To follow this in more detail, please see the discussion between various
people involved in the update on the github page.  The discussion is named
"newlinerecord #70".  It might be particularly useful to check how
different *.arts files has had to be updated.


In the past few years the line-by-line data has grown massively.  The
reason for this is the necessity to include quantum numbers to do some of
the more advanced calculations, and the change from only allowing pure
Voigt calculations in air, to allow any species mixture to make up the
broadening bath, to allow line mixing computations, and to allow other
beyond-Voigt line shapes.

The main problem with the old system of keeping all data on a line-by-line
basis was that a lot of this data was shared between several lines.  In
total, we stored close to 1 kB of data per absorption line that could
basically be shared between hundreds or thousands of other lines.  This was
about half of the absorption line memory footprint.  Another problem was
that each line could have their own reference temperature, their own mass,
their own pressure broadening scheme, their own cutoff frequencies, their
own line normalization, and so on.  Thus every individual line must have
many of their parameters computed from scratch even though it was likely
that much of the information of the previous absorption line could just be

The new system intends to keep all the metadata about how the computations
are performed separate from the pure line-by-line data.  This will create
more robust absorption line catalogs, since the meta data about the
computations are stored with the original data itself.


The problem for the controlfiles are however quite large.  There are
several workspace variables that have been removed: abs_lineshape,
abs_species_per_band, lm_p_lim, and zeeman_linerecord_precalc. These are
now dealt with by the new absorption line format itself.  For lm_p_lim,
this means that the entire band has to have a singular pressure limit.  For
abs_lineshape, this means that the entire band has to share the same line
normalization, line cutoff, and line shape calculator.  There are methods
to set these globally on a loaded catalog, either for a specific
ArrayOfSpeciesTag or for the entire line catalog.  The others two just
became superfluous during the changes being made to the line catalog.

There has been one new workspace variables added: lbl_checked.  This
variable has to be set to run any of the line-by-line calculations, i.e.,
*Zeeman and *AddLines.  There is an lbl_checkedCalc() method available for
this purpose.

The most major change is in how the absorption lines are stored.  The old
abs_lines and abs_lines_per_species still exists.  They are now arrays
of AbsorptionLines rather than arrays of LineRecord.  The reading routines
for this has changed.  There are several Read* functions that output the
abs_lines variable.  They should be used to read the data.

Lastly, the NLTE variables has had to become a class of their own instead
of compromised of several matching variables.  This is because the matching
variables used to be kept in the line-by-line data before but cannot do
that now.  The reading routines for the atmospheric data now has to have
several new inputs.  See the artscomponent/nlte folder for details on how
to do this.

Several methods have also been changed or removed.  Particularly, if they
existed only to set some of the workspace variables above, then they are,
perhaps obviously, gone.  But one notable change that might affect some
users is that the abs_xsec_per_speciesAddLines2 no longer exist as its
purpose is now the standard in abs_xsec_per_speciesAddLines.

*Example change*

As an example of the changes you need to do, from the TestClearSky2.arts
test-file, which used to read:

abs_lineshapeDefine( shape="Voigt_Kuntz6", forefactor="VVH",  cutoff=750e9
)  # in general.arts
ReadXML( abs_lines, "abs_lines.xml" )

now reads:

ReadARTSCAT( abs_lines=abs_lines, filename="abs_lines.xml" )
abs_linesSetCutoff(abs_lines, "ByLine", 750e9)
abs_linesSetNormalization(abs_lines, "VVH")
... # after most of the other parts have been defined

Note that a major difference here is that "Voigt_Kuntz6" no longer exist as
an option for these calculations.  The method is simply incompatible with
reliable internal Jacobian calculations because it does not produce the
required phase.  Note that storing this catalog after the two
abs_linesSet*() calls will store the cutoff information and the
normalization information so that if ReadXML() is used on the new catalog,
there will be no need to call the *Set*() functions again.

The exact line catalog reader you need to use depends on your use case.
All previous versions of ARTSCAT can be read, and all possible catalog we
could read in the past we can still read today.


Note that some of you might experience changes in the speed of some
calculations without also adopting other parts of the code.  The parallel
paths through the code has had to change.  We aim to address this in the
future but the code as it will be committed will not be fully
speed-compatible to some of the old use-cases.  In particular, it is
advised for now to switch from iyEmissionStandard to
iyEmissionStandardParallel.  This switch is already implemented in

Generally, however, the new code either outperforms or matches the old code.

*The end?*

I am sure there will be many hiccups and troubles as this major branch is
merged, but I hope they can be addressed without much problem as we go

With hope,
arts_dev.mi mailing list

Reply via email to