Dear ARTS developers, The infrastructure for ARTS line-by-line data will change significantly with an upcoming update to the development version of ARTS. Your simulations will be broken before you adapt to the new infrastructure. I will use this email to explain what updates are required to continue your simulations as before. I will first justify the necessity of the update. To follow this in more detail, please see the discussion between various people involved in the update on the github page. The discussion is named "newlinerecord #70". It might be particularly useful to check how different *.arts files has had to be updated.
*Background* In the past few years the line-by-line data has grown massively. The reason for this is the necessity to include quantum numbers to do some of the more advanced calculations, and the change from only allowing pure Voigt calculations in air, to allow any species mixture to make up the broadening bath, to allow line mixing computations, and to allow other beyond-Voigt line shapes. The main problem with the old system of keeping all data on a line-by-line basis was that a lot of this data was shared between several lines. In total, we stored close to 1 kB of data per absorption line that could basically be shared between hundreds or thousands of other lines. This was about half of the absorption line memory footprint. Another problem was that each line could have their own reference temperature, their own mass, their own pressure broadening scheme, their own cutoff frequencies, their own line normalization, and so on. Thus every individual line must have many of their parameters computed from scratch even though it was likely that much of the information of the previous absorption line could just be copied. The new system intends to keep all the metadata about how the computations are performed separate from the pure line-by-line data. This will create more robust absorption line catalogs, since the meta data about the computations are stored with the original data itself. *Controlfiles* The problem for the controlfiles are however quite large. There are several workspace variables that have been removed: abs_lineshape, abs_species_per_band, lm_p_lim, and zeeman_linerecord_precalc. These are now dealt with by the new absorption line format itself. For lm_p_lim, this means that the entire band has to have a singular pressure limit. For abs_lineshape, this means that the entire band has to share the same line normalization, line cutoff, and line shape calculator. There are methods to set these globally on a loaded catalog, either for a specific ArrayOfSpeciesTag or for the entire line catalog. The others two just became superfluous during the changes being made to the line catalog. There has been one new workspace variables added: lbl_checked. This variable has to be set to run any of the line-by-line calculations, i.e., *Zeeman and *AddLines. There is an lbl_checkedCalc() method available for this purpose. The most major change is in how the absorption lines are stored. The old abs_lines and abs_lines_per_species still exists. They are now arrays of AbsorptionLines rather than arrays of LineRecord. The reading routines for this has changed. There are several Read* functions that output the abs_lines variable. They should be used to read the data. Lastly, the NLTE variables has had to become a class of their own instead of compromised of several matching variables. This is because the matching variables used to be kept in the line-by-line data before but cannot do that now. The reading routines for the atmospheric data now has to have several new inputs. See the artscomponent/nlte folder for details on how to do this. Several methods have also been changed or removed. Particularly, if they existed only to set some of the workspace variables above, then they are, perhaps obviously, gone. But one notable change that might affect some users is that the abs_xsec_per_speciesAddLines2 no longer exist as its purpose is now the standard in abs_xsec_per_speciesAddLines. *Example change* As an example of the changes you need to do, from the TestClearSky2.arts test-file, which used to read: abs_lineshapeDefine( shape="Voigt_Kuntz6", forefactor="VVH", cutoff=750e9 ) # in general.arts ReadXML( abs_lines, "abs_lines.xml" ) now reads: ReadARTSCAT( abs_lines=abs_lines, filename="abs_lines.xml" ) abs_linesSetCutoff(abs_lines, "ByLine", 750e9) abs_linesSetNormalization(abs_lines, "VVH") ... # after most of the other parts have been defined lbl_checkedCalc Note that a major difference here is that "Voigt_Kuntz6" no longer exist as an option for these calculations. The method is simply incompatible with reliable internal Jacobian calculations because it does not produce the required phase. Note that storing this catalog after the two abs_linesSet*() calls will store the cutoff information and the normalization information so that if ReadXML() is used on the new catalog, there will be no need to call the *Set*() functions again. The exact line catalog reader you need to use depends on your use case. All previous versions of ARTSCAT can be read, and all possible catalog we could read in the past we can still read today. *Speed* Note that some of you might experience changes in the speed of some calculations without also adopting other parts of the code. The parallel paths through the code has had to change. We aim to address this in the future but the code as it will be committed will not be fully speed-compatible to some of the old use-cases. In particular, it is advised for now to switch from iyEmissionStandard to iyEmissionStandardParallel. This switch is already implemented in agenda.arts. Generally, however, the new code either outperforms or matches the old code. *The end?* I am sure there will be many hiccups and troubles as this major branch is merged, but I hope they can be addressed without much problem as we go forward. With hope, //Richard
_______________________________________________ arts_dev.mi mailing list arts_dev.mi@lists.uni-hamburg.de https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_dev.mi
