Re: ARTS line absorption v2.3 vs v2.5

[Richard Larsson]

Hi Stuart,

I see.  I think I need yet another update in the future because I don't
support mirroring in the way you want to use it (which is how ARTS 2.3 did
it, so it should probably return).

My intention with the "Manual" option was just to allow checks of negative
line centers to pass.  My intent was that you as a user had mirrored not
just the central frequency but also the line shifts, as these are shifting
in the wrong direction for the mirrored line.  However, this is actually
not how 2.3 did it.  In 2.3, the direction of the shifts were based on the
sign of the frequency.  So manually mirrored lines must manually mirror
themselves.

So two changes must be made on my side:

1) Manual mirroring should return to use options as in ARTS 2.3.
2) The checks on the cutoff should fail for any type of active mirroring
bar Manual.

Give me some hours to prepare a patch and the hours/days/weeks it takes for
this to merge.

With hope,
//Richard

Den tors 10 juni 2021 kl 10:29 skrev Fox, Stuart <
stuart@metoffice.gov.uk>:

> Sorry – for v2.5 I should have said I’m seeing differences between
> mirroring_option=”SameAsLineShape” and mirroring_option=”Manual”.
>
>
>
> *From:* Fox, Stuart
> *Sent:* 10 June 2021 09:26
> *To:* Richard Larsson 
> *Cc:* arts_dev.mi@lists.uni-hamburg.de
> *Subject:* RE: ARTS line absorption v2.3 vs v2.5
>
>
>
> Hi Richard,
>
>
>
> Thanks very much for your reply. I expect the difference I’m seeing
> between the versions when there’s a normalization factor and a pressure
> shift is due to the calculation of the normalization factor on the
> unshifted frequencies. I don’t know enough about the physics to comment
> which is more appropriate.
>
>
>
> However, I still don’t understand the differences I’m seeing with the
> mirror lines, as they are there even with no cutoff and no forefactor. In
> v2.3 I get identical results if I use:
>
>1. abs_lines_per_speciesAddMirrorLines with a single line with no
>pressure shift @ H2O-161 183310107053.569 0.0 2.3119066399931e-16 296
>2.70486699225234e-21 30326.8955785838 153557.646880829 0.71 0.71 296 0
>300 0.1 0.05 0.05 -1 -1 -1
>2. a manual mirror line: @ H2O-161 183310107053.569 0.0
>2.3119066399931e-16 296 2.70486699225234e-21 30326.8955785838
>153557.646880829 0.71 0.71 296 0 300 0.1 0.05 0.05 -1 -1 -1
>
>  @ H2O-161
> -183310107053.569 0.0 2.3119066399931e-16 296 2.70486699225234e-21
> 30326.8955785838 153557.646880829 0.71 0.71 296 0 300 0.1 0.05 0.05 -1
> -1 -1
>
> In v2.5 these two line files give different results (when the first is
> applied with mirroring_option=”None” and the second with
> mirroring_option=”Manual”)
>
>
>
> Note that I’m doing these tests with a Lorentz line shape, so the
> differences in the Voigt function shouldn’t matter.
>
>
>
> Thanks for your time,
>
>
>
> Stuart
>
>
>
> *From:* Richard Larsson 
> *Sent:* 09 June 2021 17:01
> *To:* Fox, Stuart 
> *Cc:* arts_dev.mi@lists.uni-hamburg.de
> *Subject:* Re: ARTS line absorption v2.3 vs v2.5
>
>
>
> *This email was received from an external source.   Always check sender
> details, links & attachments.*
>
> Hi Stuart,
>
>
>
> All the differences between current 2.5 version and old version that I can
> think of are:
>
>
>
> The Doppler broadening is computed on shifted frequencies instead of on
> unshifted frequencies.  This should be more accurate as far as
> I am concerned.
>
>
>
> The normalization factor is computed on the unshifted frequencies instead
> of on the shifted frequencies.  If I have not misunderstood the
> normalization factors, this should also be closer to the physics since you
> shouldn't really be scaling the line strengths with the line frequency but
> by the atmospheric frequency (as in the new LM code).  This is potentially
> an issue.
>
>
>
> The line shape can no longer be a speed-up Voigt function but you have to
> use the full calculations.  The old code was wrong at a scale of 1e-6 if I
> understand the old comments.
>
>
>
> The new code does the mirroring inline unless manually added to the
> catalog.  Before you had to manually add a new line with negative line
> center.  It is possible there's a bug here, since I haven't tested this so
> accurately before.  This changes the cutoff frequency since the old code
> cutoff at CUTOFF - F0 instead at F0 + CUTOFF for the inline mirrored
> lines.  (Thinking about it practically, this makes the line shape
> non-symmetric, so I will have to add a check to the lbl-check that there's
> no cutoff used when mirrored lines are there by inline instead of by
> copy-pasta.)  I would think that the difference in cutoff here is the main
> difference you are seeing.
>
>
>
> I think that's about it.  There's of course entire different algorithm
> involved, so some additional differences may be from there.
>
>
>
> With hope,
>
> //Richard
>
>
>
>
>
>
>
> Den ons 9 juni 2021 kl 17:21 skrev Fox, Stuart <
> stuart@metoff

RE: ARTS line absorption v2.3 vs v2.5

[Fox, Stuart]

Sorry – for v2.5 I should have said I’m seeing differences between 
mirroring_option=”SameAsLineShape” and mirroring_option=”Manual”.

From: Fox, Stuart
Sent: 10 June 2021 09:26
To: Richard Larsson 
Cc: arts_dev.mi@lists.uni-hamburg.de
Subject: RE: ARTS line absorption v2.3 vs v2.5

Hi Richard,

Thanks very much for your reply. I expect the difference I’m seeing between the 
versions when there’s a normalization factor and a pressure shift is due to the 
calculation of the normalization factor on the unshifted frequencies. I don’t 
know enough about the physics to comment which is more appropriate.

However, I still don’t understand the differences I’m seeing with the mirror 
lines, as they are there even with no cutoff and no forefactor. In v2.3 I get 
identical results if I use:

  1.  abs_lines_per_speciesAddMirrorLines with a single line with no pressure 
shift @ H2O-161 183310107053.569 0.0 2.3119066399931e-16 296 
2.70486699225234e-21 30326.8955785838 153557.646880829 0.71 0.71 296 0 300 
0.1 0.05 0.05 -1 -1 -1
  2.  a manual mirror line: @ H2O-161 183310107053.569 0.0 2.3119066399931e-16 
296 2.70486699225234e-21 30326.8955785838 153557.646880829 0.71 0.71 296 0 
300 0.1 0.05 0.05 -1 -1 -1
 @ H2O-161 
-183310107053.569 0.0 2.3119066399931e-16 296 2.70486699225234e-21 
30326.8955785838 153557.646880829 0.71 0.71 296 0 300 0.1 0.05 0.05 -1 -1 -1
In v2.5 these two line files give different results (when the first is applied 
with mirroring_option=”None” and the second with mirroring_option=”Manual”)

Note that I’m doing these tests with a Lorentz line shape, so the differences 
in the Voigt function shouldn’t matter.

Thanks for your time,

Stuart

From: Richard Larsson mailto:ric.lars...@gmail.com>>
Sent: 09 June 2021 17:01
To: Fox, Stuart 
mailto:stuart@metoffice.gov.uk>>
Cc: arts_dev.mi@lists.uni-hamburg.de
Subject: Re: ARTS line absorption v2.3 vs v2.5


This email was received from an external source.   Always check sender details, 
links & attachments.
Hi Stuart,

All the differences between current 2.5 version and old version that I can 
think of are:

The Doppler broadening is computed on shifted frequencies instead of on 
unshifted frequencies.  This should be more accurate as far as I am concerned.

The normalization factor is computed on the unshifted frequencies instead of on 
the shifted frequencies.  If I have not misunderstood the normalization 
factors, this should also be closer to the physics since you shouldn't really 
be scaling the line strengths with the line frequency but by the atmospheric 
frequency (as in the new LM code).  This is potentially an issue.

The line shape can no longer be a speed-up Voigt function but you have to use 
the full calculations.  The old code was wrong at a scale of 1e-6 if I 
understand the old comments.

The new code does the mirroring inline unless manually added to the catalog.  
Before you had to manually add a new line with negative line center.  It is 
possible there's a bug here, since I haven't tested this so accurately before.  
This changes the cutoff frequency since the old code cutoff at CUTOFF - F0 
instead at F0 + CUTOFF for the inline mirrored lines.  (Thinking about it 
practically, this makes the line shape non-symmetric, so I will have to add a 
check to the lbl-check that there's no cutoff used when mirrored lines are 
there by inline instead of by copy-pasta.)  I would think that the difference 
in cutoff here is the main difference you are seeing.

I think that's about it.  There's of course entire different algorithm 
involved, so some additional differences may be from there.

With hope,
//Richard



Den ons 9 juni 2021 kl 17:21 skrev Fox, Stuart 
mailto:stuart@metoffice.gov.uk>>:
Hello all,

Thanks for all the work on the line cutoff behaviour in the latest version of 
ARTS! However,I’m still struggling to reproduce the absorption line behaviour 
between v2.3.1277 and v2.5 even after the latest work on the line cutoff from 
Richard. In fact, even without a line cutoff there still seems to be 
differences caused by (i) the interaction between a line normalization factor 
(e.g. VVH) and the line pressure shift, and (ii) the treatment of mirror lines.

A simple example of (i) using only a single water vapour line with a Lorentz 
profile and VVH pre-factor can be found here, with differences up to about 
0.08K:
https://github.com/stuartfox/arts-cutoff-problem/tree/single-line

For (ii), if I set the normalization to None but add the mirror lines (with the 
SameAsLineShape option) I get differences of 0.001K in this test, but much 
larger values in a more realistic case with more lines (0.5-0.6K below 100GHz).

I’ve no idea which version is more correct, but it would be nice to understand 
the causes of these variations!

Thanks for all your help,

Stuart

RE: ARTS line absorption v2.3 vs v2.5

[Fox, Stuart]

Hi Richard,

Thanks very much for your reply. I expect the difference I’m seeing between the 
versions when there’s a normalization factor and a pressure shift is due to the 
calculation of the normalization factor on the unshifted frequencies. I don’t 
know enough about the physics to comment which is more appropriate.

However, I still don’t understand the differences I’m seeing with the mirror 
lines, as they are there even with no cutoff and no forefactor. In v2.3 I get 
identical results if I use:

  1.  abs_lines_per_speciesAddMirrorLines with a single line with no pressure 
shift @ H2O-161 183310107053.569 0.0 2.3119066399931e-16 296 
2.70486699225234e-21 30326.8955785838 153557.646880829 0.71 0.71 296 0 300 
0.1 0.05 0.05 -1 -1 -1
  2.  a manual mirror line: @ H2O-161 183310107053.569 0.0 2.3119066399931e-16 
296 2.70486699225234e-21 30326.8955785838 153557.646880829 0.71 0.71 296 0 
300 0.1 0.05 0.05 -1 -1 -1
 @ H2O-161 
-183310107053.569 0.0 2.3119066399931e-16 296 2.70486699225234e-21 
30326.8955785838 153557.646880829 0.71 0.71 296 0 300 0.1 0.05 0.05 -1 -1 -1
In v2.5 these two line files give different results (when the first is applied 
with mirroring_option=”None” and the second with mirroring_option=”Manual”)

Note that I’m doing these tests with a Lorentz line shape, so the differences 
in the Voigt function shouldn’t matter.

Thanks for your time,

Stuart

From: Richard Larsson 
Sent: 09 June 2021 17:01
To: Fox, Stuart 
Cc: arts_dev.mi@lists.uni-hamburg.de
Subject: Re: ARTS line absorption v2.3 vs v2.5


This email was received from an external source.   Always check sender details, 
links & attachments.
Hi Stuart,

All the differences between current 2.5 version and old version that I can 
think of are:

The Doppler broadening is computed on shifted frequencies instead of on 
unshifted frequencies.  This should be more accurate as far as I am concerned.

The normalization factor is computed on the unshifted frequencies instead of on 
the shifted frequencies.  If I have not misunderstood the normalization 
factors, this should also be closer to the physics since you shouldn't really 
be scaling the line strengths with the line frequency but by the atmospheric 
frequency (as in the new LM code).  This is potentially an issue.

The line shape can no longer be a speed-up Voigt function but you have to use 
the full calculations.  The old code was wrong at a scale of 1e-6 if I 
understand the old comments.

The new code does the mirroring inline unless manually added to the catalog.  
Before you had to manually add a new line with negative line center.  It is 
possible there's a bug here, since I haven't tested this so accurately before.  
This changes the cutoff frequency since the old code cutoff at CUTOFF - F0 
instead at F0 + CUTOFF for the inline mirrored lines.  (Thinking about it 
practically, this makes the line shape non-symmetric, so I will have to add a 
check to the lbl-check that there's no cutoff used when mirrored lines are 
there by inline instead of by copy-pasta.)  I would think that the difference 
in cutoff here is the main difference you are seeing.

I think that's about it.  There's of course entire different algorithm 
involved, so some additional differences may be from there.

With hope,
//Richard



Den ons 9 juni 2021 kl 17:21 skrev Fox, Stuart 
mailto:stuart@metoffice.gov.uk>>:
Hello all,

Thanks for all the work on the line cutoff behaviour in the latest version of 
ARTS! However,I’m still struggling to reproduce the absorption line behaviour 
between v2.3.1277 and v2.5 even after the latest work on the line cutoff from 
Richard. In fact, even without a line cutoff there still seems to be 
differences caused by (i) the interaction between a line normalization factor 
(e.g. VVH) and the line pressure shift, and (ii) the treatment of mirror lines.

A simple example of (i) using only a single water vapour line with a Lorentz 
profile and VVH pre-factor can be found here, with differences up to about 
0.08K:
https://github.com/stuartfox/arts-cutoff-problem/tree/single-line

For (ii), if I set the normalization to None but add the mirror lines (with the 
SameAsLineShape option) I get differences of 0.001K in this test, but much 
larger values in a more realistic case with more lines (0.5-0.6K below 100GHz).

I’ve no idea which version is more correct, but it would be nice to understand 
the causes of these variations!

Thanks for all your help,

Stuart