Re: [arts-dev] Problem with line absorption in ARTS

[Richard Larsson]

Stefan,

Ok, then 'Mirrored Lorentz' is not a good replacement since it works
differently.  This will also be difficult to consider in any new
implementation.  I thought the policy was to leave such optimizations up to
the user...

The new system I suggest has to remove the lineshapes of today and replace
them with 'best-match' alternatives.  Numerically, the difference should be
nil to very small.  Of course, in the end you should be able to manipulate
individual lines to say what lineshape they should be using, but that will
then have to be stored on a line-by-line basis and not on a
speciestag-by-speciestag basis.  Functions like
SetAllLinesOutsideTheFrequencyGridToUseLorentzLineShape  and
SetAllLinesOfSpeciesTagToUseACutOffOfF(tag="H2O", F=750e9) should be
possible (but sometimes risky choices), and will allow similar controlfile
manipulation of the lineshapes as the abs_lineshapeDefine function.

Computationally, the speed would vary quite a lot.  I have the
lineshapesdata.cc/h files that I made to work with the linemixing code.  It
is about as fast as the normal solutions today --- it is faster if you
deactivate some IEEE-code and tell the compiler to properly optimize for
your system --- but things like VoigtKuntz6 is not working because it only
produces half a lineshape today.

(VoigtKuntz6 could be made to work, though it will be much slower than the
present implementation since the current setup is optimized for it to be
fast --- the other lineshapes are slower today because they have to work
around the VoigtKuntz6-imposed limitations.)

I will try to make an abs_xsec_per_speciesAddLines2 function sometime later
this week or next using the other setup.  This will not be feature-complete
by then but it will be better to have this discussion when some code
exists.

//Richard

2017-10-16 10:56 GMT+02:00 Stefan Buehler :

> Dear Richard,
>
> > Ps. The cutoff is determined based on the negative frequency and not on
> the frequency of the line.  Not sure if that is intentional or not, since I
> still have not read the papers on why the cutoff exists.  This is not a big
> numerical issue but at most a potential mismatch in theory.
>
> This is intentional, since in this way mirror lines that are far away are
> optimised away.
>
> I am fine with moving to a new system altogether, as you propose, but we
> have to make sure the program keeps working. We cannot first remove the
> working lineshapes, and then later introduce new ones.
>
> Best wishes,
>
> Stefan
___
arts_dev.mi mailing list
arts_dev.mi@lists.uni-hamburg.de
https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_dev.mi

Re: [arts-dev] Problem with line absorption in ARTS

[Stefan Buehler]

Dear Richard,

> Ps. The cutoff is determined based on the negative frequency and not on the 
> frequency of the line.  Not sure if that is intentional or not, since I still 
> have not read the papers on why the cutoff exists.  This is not a big 
> numerical issue but at most a potential mismatch in theory.

This is intentional, since in this way mirror lines that are far away are 
optimised away.

I am fine with moving to a new system altogether, as you propose, but we have 
to make sure the program keeps working. We cannot first remove the working 
lineshapes, and then later introduce new ones.

Best wishes,

Stefan

smime.p7s
Description: S/MIME cryptographic signature
___
arts_dev.mi mailing list
arts_dev.mi@lists.uni-hamburg.de
https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_dev.mi

Re: [arts-dev] Problem with line absorption in ARTS

[Richard Larsson]

Stefan,

Mirroring Voigt or HTP internally will not be an issue.  This could happen
by a flag and either trigger parallel computations or companion functions.
(I prefer the latter because then you can set the flag to use Lorentz in
the mirroring if the effects are so far-wing that it should work the same
anyway.)

This will not happen unless we move away from the present line-shape
system.  It has gotten very difficult to work with because of my tinkering.

//Richard

Ps. The cutoff is determined based on the negative frequency and not on the
frequency of the line.  Not sure if that is intentional or not, since I
still have not read the papers on why the cutoff exists.  This is not a big
numerical issue but at most a potential mismatch in theory.

2017-10-15 14:56 GMT+02:00 Stefan Buehler <stefan.bueh...@uni-hamburg.de>:

> Dear Stuart, dear Richard,
>
> Stuart, thanks for finding and reporting this. This is clearly a bug, due
> to an oversight of mine at the time. For mirror lines, the shift should be
> such that they continue to be at the mirror frequency of the original line.
> So, if the original line shifts to higher frequency, then the mirror line
> has to shift to more negative frequencies.
>
> I have committed a bug fix around the code line that Richard has pointed
> out, since I don’t see a good reason not to fix this.
>
> Richard, perhaps we should follow something like your suggestion 2 in the
> longer run. But Mirrored Lorentz alone would not solve the problem, would
> it, since we need something that transitions in a good way to a Voigt for
> low pressure. Can the HTP profile account for mirror lines? Then that would
> be the best long term solution.
>
> Stuart, could you please test whether the new version fixes the problem?
>
> Best wishes,
>
> Stefan
>
> > On 13. Oct 2017, at 17:56, Richard Larsson <ric.lars...@gmail.com>
> wrote:
> >
> > Hi Stuart,
> >
> > You are correct.  The central frequency of the mirrored lines in arts
> are just set to be negative, letting all the other calculations go on as
> usual.  So the addition of pressure shift is not handled correctly.  (It
> even slows some functions down because we have to apply abs(F0), but this
> was not considered in regards to the shifting.)
> >
> > Two possible solutions exists:
> > 1) Change so F0 in line 1350 of absorption.cc has the absolute value
> applied to it before the shift and is then multiplied by its actual sign.
> >
> > 2) Remove the mirrored lines function and enforce F0>0, and tell people
> to use the existing "Mirrored Lorentz" instead when they want to use the
> transformation lineshape.
> >
> > Option 2 means that all simulations will be slightly faster because it
> reduces the number of calculations and we an remove some existing abs(F0)
> complications.  Option 1 means people do not have to change controlfiles
> but it makes all code slightly slower.  I argue we go with option 2.  The
> present implementation of how we deal with the pressure shift is not really
> compatible with HTP, so it will have to change at some point soon anyways.
> >
> > What do others on the developer list say?
> >
> > With hope,
> > //Richard
> >
> > 2017-10-13 15:07 GMT+02:00 Fox, Stuart <stuart@metoffice.gov.uk>:
> > Thanks Richard. The fact that changing to the mirrored Lorentz lineshape
> solves the problem suggests to me that it is highly likely that the
> pressure shift is being added in the wrong direction for the mirrored lines
> if using abs_lines_per_speciesAddMirrorLines, hence the positive pressure
> shift coefficient is shifting the mirror line closer to zero rather than
> further away.
> >
> >
> >
> > I know I’m using an “old” version of ARTS – I’ll get round to updating
> it once I’ve had a chance to get round to somehow installing cmake 3 on our
> systems so that I can actually compile anything newer…
> >
> >
> >
> > Cheers,
> >
> >
> >
> > Stuart
> >
> >
> >
> > From: Richard Larsson [mailto:ric.lars...@gmail.com]
> > Sent: 13 October 2017 14:01
> > To: Fox, Stuart <stuart@metoffice.gov.uk>
> > Cc: arts_dev.mi@lists.uni-hamburg.de
> > Subject: Re: [arts-dev] Problem with line absorption in ARTS
> >
> >
> >
> > Stuart,
> >
> >
> >
> > I am also seeing the problem but I have no time to investigate in
> detail.  As a temporary solution, there exists another lineshape called
> "Mirrored Lorentz" that you can use.  Testing your controlfile with it
> instead, the results are more reasonable.  Simply change your
> >
> >
> >
> > 

Re: [arts-dev] Problem with line absorption in ARTS

[Stefan Buehler]

Dear Stuart, dear Richard,

Stuart, thanks for finding and reporting this. This is clearly a bug, due to an 
oversight of mine at the time. For mirror lines, the shift should be such that 
they continue to be at the mirror frequency of the original line. So, if the 
original line shifts to higher frequency, then the mirror line has to shift to 
more negative frequencies.

I have committed a bug fix around the code line that Richard has pointed out, 
since I don’t see a good reason not to fix this.

Richard, perhaps we should follow something like your suggestion 2 in the 
longer run. But Mirrored Lorentz alone would not solve the problem, would it, 
since we need something that transitions in a good way to a Voigt for low 
pressure. Can the HTP profile account for mirror lines? Then that would be the 
best long term solution.

Stuart, could you please test whether the new version fixes the problem?

Best wishes,

Stefan

> On 13. Oct 2017, at 17:56, Richard Larsson <ric.lars...@gmail.com> wrote:
> 
> Hi Stuart,
> 
> You are correct.  The central frequency of the mirrored lines in arts are 
> just set to be negative, letting all the other calculations go on as usual.  
> So the addition of pressure shift is not handled correctly.  (It even slows 
> some functions down because we have to apply abs(F0), but this was not 
> considered in regards to the shifting.)
> 
> Two possible solutions exists:
> 1) Change so F0 in line 1350 of absorption.cc has the absolute value applied 
> to it before the shift and is then multiplied by its actual sign.
> 
> 2) Remove the mirrored lines function and enforce F0>0, and tell people to 
> use the existing "Mirrored Lorentz" instead when they want to use the 
> transformation lineshape.
> 
> Option 2 means that all simulations will be slightly faster because it 
> reduces the number of calculations and we an remove some existing abs(F0) 
> complications.  Option 1 means people do not have to change controlfiles but 
> it makes all code slightly slower.  I argue we go with option 2.  The present 
> implementation of how we deal with the pressure shift is not really 
> compatible with HTP, so it will have to change at some point soon anyways.
> 
> What do others on the developer list say?  
> 
> With hope,
> //Richard
> 
> 2017-10-13 15:07 GMT+02:00 Fox, Stuart <stuart@metoffice.gov.uk>:
> Thanks Richard. The fact that changing to the mirrored Lorentz lineshape 
> solves the problem suggests to me that it is highly likely that the pressure 
> shift is being added in the wrong direction for the mirrored lines if using 
> abs_lines_per_speciesAddMirrorLines, hence the positive pressure shift 
> coefficient is shifting the mirror line closer to zero rather than further 
> away.
> 
>  
> 
> I know I’m using an “old” version of ARTS – I’ll get round to updating it 
> once I’ve had a chance to get round to somehow installing cmake 3 on our 
> systems so that I can actually compile anything newer…
> 
>  
> 
> Cheers,
> 
>  
> 
> Stuart
> 
>  
> 
> From: Richard Larsson [mailto:ric.lars...@gmail.com] 
> Sent: 13 October 2017 14:01
> To: Fox, Stuart <stuart@metoffice.gov.uk>
> Cc: arts_dev.mi@lists.uni-hamburg.de
> Subject: Re: [arts-dev] Problem with line absorption in ARTS
> 
>  
> 
> Stuart,
> 
>  
> 
> I am also seeing the problem but I have no time to investigate in detail.  As 
> a temporary solution, there exists another lineshape called "Mirrored 
> Lorentz" that you can use.  Testing your controlfile with it instead, the 
> results are more reasonable.  Simply change your 
> 
>  
> 
> abs_lines_per_speciesAddMirrorLines
> 
>  
> 
> to
> 
>  
> 
> abs_lineshapeDefine(shape="Mirrored Lorentz", forefactor="VVW", cutoff=750e9)
> 
>  
> 
> and you will see the results.  (Or just the original "Lorentz" lineshape-call 
> in your main controlfile.)
> 
>  
> 
> With hope,
> 
> //Richard
> 
>  
> 
> Ps. You are running an old version of arts.  The controlfile was not working 
> without changing the order of calls around.  I attach an updated version.  I 
> also recommend that you set ARTS_DATA_PATH to your arts-xml-data path because 
> this makes life easier (export ARTS_DATA_PATH="PATH-TO-DATA" should do the 
> trick).  
> 
>  
> 
> I might have time to look at the details over the weekend and will get back 
> to you later but I hope the temporary solution is good for now.
> 
>  
> 
> 2017-10-13 14:31 GMT+02:00 Fox, Stuart <stuart@metoffice.gov.uk>:
> 
> Hi all,
> 
>  
> 
> I wasn’t sure whether to ask this on arts-user or arts-dev - I’m not sur

Re: [arts-dev] Problem with line absorption in ARTS

[Richard Larsson]

Hi Stuart,

You are correct.  The central frequency of the mirrored lines in arts are
just set to be negative, letting all the other calculations go on as
usual.  So the addition of pressure shift is not handled correctly.  (It
even slows some functions down because we have to apply abs(F0), but this
was not considered in regards to the shifting.)

Two possible solutions exists:
1) Change so F0 in line 1350 of absorption.cc has the absolute value
applied to it before the shift and is then multiplied by its actual sign.

2) Remove the mirrored lines function and enforce F0>0, and tell people to
use the existing "Mirrored Lorentz" instead when they want to use the
transformation lineshape.

Option 2 means that all simulations will be slightly faster because it
reduces the number of calculations and we an remove some existing abs(F0)
complications.  Option 1 means people do not have to change controlfiles
but it makes all code slightly slower.  I argue we go with option 2.  The
present implementation of how we deal with the pressure shift is not really
compatible with HTP, so it will have to change at some point soon anyways.

What do others on the developer list say?

With hope,
//Richard

2017-10-13 15:07 GMT+02:00 Fox, Stuart <stuart@metoffice.gov.uk>:

> Thanks Richard. The fact that changing to the mirrored Lorentz lineshape
> solves the problem suggests to me that it is highly likely that the
> pressure shift is being added in the wrong direction for the mirrored lines
> if using abs_lines_per_speciesAddMirrorLines, hence the positive pressure
> shift coefficient is shifting the mirror line closer to zero rather than
> further away.
>
>
>
> I know I’m using an “old” version of ARTS – I’ll get round to updating it
> once I’ve had a chance to get round to somehow installing cmake 3 on our
> systems so that I can actually compile anything newer…
>
>
>
> Cheers,
>
>
>
> Stuart
>
>
>
> *From:* Richard Larsson [mailto:ric.lars...@gmail.com]
> *Sent:* 13 October 2017 14:01
> *To:* Fox, Stuart <stuart@metoffice.gov.uk>
> *Cc:* arts_dev.mi@lists.uni-hamburg.de
> *Subject:* Re: [arts-dev] Problem with line absorption in ARTS
>
>
>
> Stuart,
>
>
>
> I am also seeing the problem but I have no time to investigate in detail.
> As a temporary solution, there exists another lineshape called "Mirrored
> Lorentz" that you can use.  Testing your controlfile with it instead, the
> results are more reasonable.  Simply change your
>
>
>
> abs_lines_per_speciesAddMirrorLines
>
>
>
> to
>
>
>
> abs_lineshapeDefine(shape="Mirrored Lorentz", forefactor="VVW",
> cutoff=750e9)
>
>
>
> and you will see the results.  (Or just the original "Lorentz"
> lineshape-call in your main controlfile.)
>
>
>
> With hope,
>
> //Richard
>
>
>
> Ps. You are running an old version of arts.  The controlfile was not
> working without changing the order of calls around.  I attach an updated
> version.  I also recommend that you set ARTS_DATA_PATH to your
> arts-xml-data path because this makes life easier (export
> ARTS_DATA_PATH="PATH-TO-DATA" should do the trick).
>
>
>
> I might have time to look at the details over the weekend and will get
> back to you later but I hope the temporary solution is good for now.
>
>
>
> 2017-10-13 14:31 GMT+02:00 Fox, Stuart <stuart@metoffice.gov.uk>:
>
> Hi all,
>
>
>
> I wasn’t sure whether to ask this on arts-user or arts-dev - I’m not sure
> whether there’s a bug in ARTS or something wrong with my inputs!
>
>
>
> I’m trying to do some clear sky brightness temperature simulations for
> Earth using water vapour lines from HITRAN-2016. However, I am getting huge
> (over 20K) differences in brightness temperature for a nadir viewing
> geometry depending on whether I include the mirror (negative frequency)
> lines or not. I don’t get the same thing using HITRAN-2012.
>
>
>
> Attached is a fairly minimal example controlfile, and extracts of H2O
> lines from the two versions of HITRAN. To see the difference then compare
> the two output files depending on which of the line sets is used. (The path
> to the data from arts-xml-data will need to be changed according to your
> setup). For HITRAN 2012 the numbers are pretty much the same. For HITRAN
> 2016 they are very different.
>
>
>
> I’ve done some digging and isolated the main cause of the issue to the
> lowest frequency H2O-162 line in HITRAN-2016 (at 21764932Hz). It seems to
> be a combination of the mirror of this line, the pressure shift and the VVW
> forefactor that causes the difference. My guess is that the pressure shift
> (at the

Re: [arts-dev] Problem with line absorption in ARTS

[Fox, Stuart]

Thanks Richard. The fact that changing to the mirrored Lorentz lineshape solves 
the problem suggests to me that it is highly likely that the pressure shift is 
being added in the wrong direction for the mirrored lines if using 
abs_lines_per_speciesAddMirrorLines, hence the positive pressure shift 
coefficient is shifting the mirror line closer to zero rather than further away.

I know I’m using an “old” version of ARTS – I’ll get round to updating it once 
I’ve had a chance to get round to somehow installing cmake 3 on our systems so 
that I can actually compile anything newer…

Cheers,

Stuart

From: Richard Larsson [mailto:ric.lars...@gmail.com]
Sent: 13 October 2017 14:01
To: Fox, Stuart <stuart@metoffice.gov.uk>
Cc: arts_dev.mi@lists.uni-hamburg.de
Subject: Re: [arts-dev] Problem with line absorption in ARTS

Stuart,

I am also seeing the problem but I have no time to investigate in detail.  As a 
temporary solution, there exists another lineshape called "Mirrored Lorentz" 
that you can use.  Testing your controlfile with it instead, the results are 
more reasonable.  Simply change your

abs_lines_per_speciesAddMirrorLines

to

abs_lineshapeDefine(shape="Mirrored Lorentz", forefactor="VVW", cutoff=750e9)

and you will see the results.  (Or just the original "Lorentz" lineshape-call 
in your main controlfile.)

With hope,
//Richard

Ps. You are running an old version of arts.  The controlfile was not working 
without changing the order of calls around.  I attach an updated version.  I 
also recommend that you set ARTS_DATA_PATH to your arts-xml-data path because 
this makes life easier (export ARTS_DATA_PATH="PATH-TO-DATA" should do the 
trick).

I might have time to look at the details over the weekend and will get back to 
you later but I hope the temporary solution is good for now.

2017-10-13 14:31 GMT+02:00 Fox, Stuart 
<stuart@metoffice.gov.uk<mailto:stuart@metoffice.gov.uk>>:
Hi all,

I wasn’t sure whether to ask this on arts-user or arts-dev - I’m not sure 
whether there’s a bug in ARTS or something wrong with my inputs!

I’m trying to do some clear sky brightness temperature simulations for Earth 
using water vapour lines from HITRAN-2016. However, I am getting huge (over 
20K) differences in brightness temperature for a nadir viewing geometry 
depending on whether I include the mirror (negative frequency) lines or not. I 
don’t get the same thing using HITRAN-2012.

Attached is a fairly minimal example controlfile, and extracts of H2O lines 
from the two versions of HITRAN. To see the difference then compare the two 
output files depending on which of the line sets is used. (The path to the data 
from arts-xml-data will need to be changed according to your setup). For HITRAN 
2012 the numbers are pretty much the same. For HITRAN 2016 they are very 
different.

I’ve done some digging and isolated the main cause of the issue to the lowest 
frequency H2O-162 line in HITRAN-2016 (at 21764932Hz). It seems to be a 
combination of the mirror of this line, the pressure shift and the VVW 
forefactor that causes the difference. My guess is that the pressure shift (at 
the highest pressures in the profile) is pushing the mirror line to a frequency 
very close to zero, possibly causing numerical issues in the 
forefactor/lineshape. (I also wonder if the pressure shift is being added in 
the wrong direction for the mirror lines?)

Does anyone who knows the ARTS internals better than me fancy digging around to 
understand what is happening, or perhaps tell me that I’m doing something wrong!

Cheers,

Stuart


___
arts_dev.mi mailing list
arts_dev.mi@lists.uni-hamburg.de<mailto:arts_dev.mi@lists.uni-hamburg.de>
https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_dev.mi

___
arts_dev.mi mailing list
arts_dev.mi@lists.uni-hamburg.de
https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_dev.mi