Re: [arts-users] Issues with size interpolation - ARTS SSD

2021-06-06 Thread Patrick Eriksson 
Isaac,

I assume that CRTM includes temperatures up to 300K is to cover ice reaching 
the ground even when the surface temperature is well above 0C. In such cases 
the ice particles are not in equilibrium with the environment. Internally the 
temperature will be close to 0C. My suggestion is then to use the data for 0C 
also for higher temperatures. 

Bye,

Patrick

Isaac Moradi  skrev: (6 juni 2021 05:25:38 CEST)
>Hi Jana,
>
>I tried to include the za interpolation but it seems that it needs some
>touch ups as there are some undefined variables. 
>
>I did manage to do all the interpolations using the code that developed
>myself and it seems to be working fine (see the attached file please).
>
>However, I have another issue which is unrelated to the code. In CRTM,
>they have temperatures up to 300 K so I need to do extrapolation as
>current SSD database doesn’t provide those temperatures. Those
>temperatures may not even make sense in the real world but we don’t
>intend to change them at the moment. The extrapolation creates some
>negative values (please see the second attached plot).
>
>I wonder if any of you has any suggestion to avoid this problem? I am
>not going to use those temperatures in my own work, but don’t want to
>through some junks into CRTM coefficients either as someone may
>eventually end up in a situation with T greater than 270 K.
>
>
>The filenames have frequency and temperature included for reference.
>
>Thank you!
>
>
>Best,
>Isaac
>
>
>
>
>
>
>
>
>
>> On Jun 4, 2021, at 12:59 PM, Jana Mendrok 
>wrote:
>> 
>> Hi Patrick et al,
>> 
>> 
>> on the same set of grids. That is, the idea is to use these functions
>
>> first, before any size interpolation. This is at least indicated in
>the 
>> documentation of assp_interp_size.m.
>> 
>>  
>> No time for me to dig into the Python code (and anyhow don't have the
>
>> latest version at hand), and don't remember how closely Jana mimicked
>my 
>> Matlab interface.
>> 
>> Used to follow the Matlab implementation very closely. An
>assp_interp_za exists. However, it's obviously nowhere applied (in the
>original Zenodo published version; I did not check out newer ones).
>> The header of assp_interp_size states the need for common grids.
>Though I agree, that's not very prominently, hence easily missed by a
>user and definitely missed when not called directly by the user.
>Obviously, assp_interp_size misses a proper check that this requirement
>is fulfilled...
>> 
>> I guess, back in the days all size instances of one habit had the
>same number of angles, so it wasn't necessary to apply that (what
>speaks for that is that an IconSnow.rssp file exists; and that had been
>created with exactly the code that was published, ie without
>az-interpolation).
>> 
>> Isaac/Vasileios - get_assp hence needs to be extended with a call of
>assp_interp_za, in between the assp_import_ssdb and
>assp.assp_interp_size (not deep enough into it anymore to judge whether
>exact location matters...).
>> 
>> In any case, Vasileios, thanks for caring for the code (and cleaning
>up after me)!
>> Best,
>> 
>> -- 
>> Jana Mendrok, Ph.D.
>> Deutscher Wetterdienst
>> Offenbach am Main, Germany
>> 
>> +49 (0)69 8062 3139
>> ___
>> arts_users.mi mailing list
>> arts_users.mi@lists.uni-hamburg.de
>> https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi
___
arts_users.mi mailing list
arts_users.mi@lists.uni-hamburg.de
https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi

Re: [arts-users] Issues with size interpolation - ARTS SSD

2021-06-05 Thread Isaac Moradi 
Hi Jana,

I tried to include the za interpolation but it seems that it needs some touch 
ups as there are some undefined variables. 

I did manage to do all the interpolations using the code that developed myself 
and it seems to be working fine (see the attached file please).

However, I have another issue which is unrelated to the code. In CRTM, they 
have temperatures up to 300 K so I need to do extrapolation as current SSD 
database doesn’t provide those temperatures. Those temperatures may not even 
make sense in the real world but we don’t intend to change them at the moment. 
The extrapolation creates some negative values (please see the second attached 
plot).

I wonder if any of you has any suggestion to avoid this problem? I am not going 
to use those temperatures in my own work, but don’t want to through some junks 
into CRTM coefficients either as someone may eventually end up in a situation 
with T greater than 270 K.


The filenames have frequency and temperature included for reference.

Thank you!


Best,
Isaac









> On Jun 4, 2021, at 12:59 PM, Jana Mendrok  wrote:
> 
> Hi Patrick et al,
> 
> 
> on the same set of grids. That is, the idea is to use these functions 
> first, before any size interpolation. This is at least indicated in the 
> documentation of assp_interp_size.m.
> 
>  
> No time for me to dig into the Python code (and anyhow don't have the 
> latest version at hand), and don't remember how closely Jana mimicked my 
> Matlab interface.
> 
> Used to follow the Matlab implementation very closely. An assp_interp_za 
> exists. However, it's obviously nowhere applied (in the original Zenodo 
> published version; I did not check out newer ones).
> The header of assp_interp_size states the need for common grids. Though I 
> agree, that's not very prominently, hence easily missed by a user and 
> definitely missed when not called directly by the user. Obviously, 
> assp_interp_size misses a proper check that this requirement is fulfilled...
> 
> I guess, back in the days all size instances of one habit had the same number 
> of angles, so it wasn't necessary to apply that (what speaks for that is that 
> an IconSnow.rssp file exists; and that had been created with exactly the code 
> that was published, ie without az-interpolation).
> 
> Isaac/Vasileios - get_assp hence needs to be extended with a call of 
> assp_interp_za, in between the assp_import_ssdb and assp.assp_interp_size 
> (not deep enough into it anymore to judge whether exact location matters...).
> 
> In any case, Vasileios, thanks for caring for the code (and cleaning up after 
> me)!
> Best,
> 
> -- 
> Jana Mendrok, Ph.D.
> Deutscher Wetterdienst
> Offenbach am Main, Germany
> 
> +49 (0)69 8062 3139
> ___
> arts_users.mi mailing list
> arts_users.mi@lists.uni-hamburg.de
> https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi

___
arts_users.mi mailing list
arts_users.mi@lists.uni-hamburg.de
https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi

Re: [arts-users] Issues with size interpolation - ARTS SSD

2021-06-04 Thread Jana Mendrok 
Hi Patrick et al,


on the same set of grids. That is, the idea is to use these functions
> first, before any size interpolation. This is at least indicated in the
> documentation of assp_interp_size.m.
>
>

> No time for me to dig into the Python code (and anyhow don't have the
> latest version at hand), and don't remember how closely Jana mimicked my
> Matlab interface.


Used to follow the Matlab implementation very closely. An assp_interp_za
exists. However, it's obviously nowhere applied (in the original Zenodo
published version; I did not check out newer ones).
The header of assp_interp_size states the need for common grids. Though I
agree, that's not very prominently, hence easily missed by a user and
definitely missed when not called directly by the user. Obviously,
assp_interp_size misses a proper check that this requirement is fulfilled...

I guess, back in the days all size instances of one habit had the same
number of angles, so it wasn't necessary to apply that (what speaks for
that is that an IconSnow.rssp file exists; and that had been created with
exactly the code that was published, ie without az-interpolation).

Isaac/Vasileios - get_assp hence needs to be extended with a call of
assp_interp_za, in between the assp_import_ssdb and assp.assp_interp_size
(not deep enough into it anymore to judge whether exact location
matters...).

In any case, Vasileios, thanks for caring for the code (and cleaning up
after me)!
Best,

-- 
Jana Mendrok, Ph.D.
Deutscher Wetterdienst
Offenbach am Main, Germany

+49 (0)69 8062 3139
___
arts_users.mi mailing list
arts_users.mi@lists.uni-hamburg.de
https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi

Re: [arts-users] Issues with size interpolation - ARTS SSD

2021-06-04 Thread Isaac Moradi 
Hi Patrick,

I am not actually directly calling any assp functions. I use rttov.get_assp as 
shown below to get what I need and send the new grids for f, T, and dveq along 
with the habit ID (I have been playing with allow_extrap to just allow to fill 
the gap for now but I know it may not be a good idea to extrapolate). The 
interpolations inside rttov.py are done in the order that Patrick mentioned. I 
didn’t try to call individual functions for interpolation but in my code, I am 
using the same interpolation technique (PchipInterpolator) and of course as 
long as the variables have the correct size then it works just fine.

S,M,mD = rttov.get_assp(habID, orient, size_type='dveq',
size_grid = dveq_mw, s_allow_extrap=True,
t_grid=tgrid, t_allow_extrap=True,
f_grid=fgrid, f_allow_extrap=True)

Best,
Isaac



> On Jun 4, 2021, at 10:59 AM, Patrick Eriksson  
> wrote:
> 
> Hi,
> 
> Some quick input, for clarification.
> 
> At least for the Matlab interface there are functions for interpolating in 
> frequency, angular grid and temperature, to get the data of all sizes on the 
> same set of grids. That is, the idea is to use these functions first, before 
> any size interpolation. This is at least indicated in the documentation of 
> assp_interp_size.m.
> 
> No time for me to dig into the Python code (and anyhow don't have the latest 
> version at hand), and don't remember how closely Jana mimicked my Matlab 
> interface. But I should assume the same principle should be applied in 
> Python. Too bad if that not clear from the documentation.
> 
> Bye,
> 
> Patrick
> 
> 
> On 2021-06-04 16:22, Vasileios Barlakas wrote:
>> Hello Isaac,
>> I had a brief look at assp_interp_size. Indeed there are some bugs in this 
>> function that need to be fixed. I will allocate some time on this next week; 
>> sorry for any inconvenience caused. For the time being, I could give you 
>> some hints:
>> To use this function, all elements in S must have common frequency, 
>> temperature, and angular grid. Note here that, for some habits, e.g. 
>> IconSnow, the given sizes are not available in the same frequency and 
>> angular grids. This is one of the reasons why you get such an error message:
>> (31,5,721,1,1,1,6) into shape (31,5,91,1,1,1,6)
>> Note here that the function defines the dimensions of all the variables from 
>> the first element of the list, i.e., n_freq, n_temp, n_za, 1, 1, 1, 
>> n_phase_matrix_elements. For your case, the dimension at which python 
>> complains is the n_za (angular grid).
>> Following this rule, you could potentially fix any other issues. I hope this 
>> helps.
>> Best regards,
>> Vasileios
>> 
>> *From:* arts_users.mi-boun...@mailman.rrz.uni-hamburg.de 
>>  on behalf of Isaac Moradi 
>> 
>> *Sent:* Wednesday, June 2, 2021 4:36:50 PM
>> *To:* arts_users...@mailman.rrz.uni-hamburg.de
>> *Subject:* [arts-users] Issues with size interpolation - ARTS SSD
>> Hi all,
>> I am trying to do size interpolation using the assp.py included in the 
>> Python data interface but there seems to be multiple issues with the size 
>> interpolation function. There are some undefined variables that I fixed them 
>> but then end up with the following error.
>> I am trying this for IconSnow but the error should exist for any other 
>> habits.
>> Here is the message from Python:
>>   File "/data/users/imoradi/workspace/dda/DataInterfaces/Python/assp.py", 
>> line 1154, in assp_interp_size
>> p[i+1,...] = s.pha_mat_data
>> ValueError: could not broadcast input array from shape (31,5,721,1,1,1,6) 
>> into shape (31,5,91,1,1,1,6)
>> Thanks
>> Best,
>> Isaac
>> ___
>> arts_users.mi mailing list
>> arts_users.mi@lists.uni-hamburg.de
>> https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi
>> ___
>> arts_users.mi mailing list
>> arts_users.mi@lists.uni-hamburg.de
>> https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi
> ___
> arts_users.mi mailing list
> arts_users.mi@lists.uni-hamburg.de
> https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi

___
arts_users.mi mailing list
arts_users.mi@lists.uni-hamburg.de
https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi

Re: [arts-users] Issues with size interpolation - ARTS SSD

2021-06-04 Thread Patrick Eriksson 

Hi,

Some quick input, for clarification.

At least for the Matlab interface there are functions for interpolating 
in frequency, angular grid and temperature, to get the data of all sizes 
on the same set of grids. That is, the idea is to use these functions 
first, before any size interpolation. This is at least indicated in the 
documentation of assp_interp_size.m.


No time for me to dig into the Python code (and anyhow don't have the 
latest version at hand), and don't remember how closely Jana mimicked my 
Matlab interface. But I should assume the same principle should be 
applied in Python. Too bad if that not clear from the documentation.


Bye,

Patrick


On 2021-06-04 16:22, Vasileios Barlakas wrote:

Hello Isaac,


I had a brief look at assp_interp_size. Indeed there are some bugs in 
this function that need to be fixed. I will allocate some time on this 
next week; sorry for any inconvenience caused. For the time being, I 
could give you some hints:



To use this function, all elements in S must have common frequency, 
temperature, and angular grid. Note here that, for some habits, e.g. 
IconSnow, the given sizes are not available in the same frequency and 
angular grids. This is one of the reasons why you get such an error message:



(31,5,721,1,1,1,6) into shape (31,5,91,1,1,1,6)


Note here that the function defines the dimensions of all the variables 
from the first element of the list, i.e., n_freq, n_temp, n_za, 1, 1, 1, 
n_phase_matrix_elements. For your case, the dimension at which python 
complains is the n_za (angular grid).



Following this rule, you could potentially fix any other issues. I hope 
this helps.



Best regards,

Vasileios



*From:* arts_users.mi-boun...@mailman.rrz.uni-hamburg.de 
 on behalf of Isaac 
Moradi 

*Sent:* Wednesday, June 2, 2021 4:36:50 PM
*To:* arts_users...@mailman.rrz.uni-hamburg.de
*Subject:* [arts-users] Issues with size interpolation - ARTS SSD
Hi all,

I am trying to do size interpolation using the assp.py included in the 
Python data interface but there seems to be multiple issues with the 
size interpolation function. There are some undefined variables that I 
fixed them but then end up with the following error.
I am trying this for IconSnow but the error should exist for any other 
habits.


Here is the message from Python:

   File 
"/data/users/imoradi/workspace/dda/DataInterfaces/Python/assp.py", line 
1154, in assp_interp_size

     p[i+1,...] = s.pha_mat_data
ValueError: could not broadcast input array from shape 
(31,5,721,1,1,1,6) into shape (31,5,91,1,1,1,6)



Thanks


Best,
Isaac
___
arts_users.mi mailing list
arts_users.mi@lists.uni-hamburg.de
https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi

___
arts_users.mi mailing list
arts_users.mi@lists.uni-hamburg.de
https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi


___
arts_users.mi mailing list
arts_users.mi@lists.uni-hamburg.de
https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi

Re: [arts-users] Issues with size interpolation - ARTS SSD

2021-06-04 Thread Isaac Moradi 
HI Vasileios,

Thanks for the hints. Meanwhile, I spent sometime and wrote a piece of code to 
read the data in and interpolate them. I will they to see if can fix it 
according to what you suggested. In any case, once you have fixed the code, I 
will compare what I have developed with the main code. 

Thanks again.


Best,
Isaac





> On Jun 4, 2021, at 10:22 AM, Vasileios Barlakas 
>  wrote:
> 
> Hello Isaac,
> 
> I had a brief look at assp_interp_size. Indeed there are some bugs in this 
> function that need to be fixed. I will allocate some time on this next week; 
> sorry for any inconvenience caused. For the time being, I could give you some 
> hints:
> 
> To use this function, all elements in S must have common frequency, 
> temperature, and angular grid. Note here that, for some habits, e.g. 
> IconSnow, the given sizes are not available in the same frequency and angular 
> grids. This is one of the reasons why you get such an error message:
> 
> (31,5,721,1,1,1,6) into shape (31,5,91,1,1,1,6)
> 
> Note here that the function defines the dimensions of all the variables from 
> the first element of the list, i.e., n_freq, n_temp, n_za, 1, 1, 1, 
> n_phase_matrix_elements. For your case, the dimension at which python 
> complains is the n_za (angular grid).
> 
> Following this rule, you could potentially fix any other issues. I hope this 
> helps. 
> 
> Best regards,
> Vasileios
> 
> From: arts_users.mi-boun...@mailman.rrz.uni-hamburg.de 
>  on behalf of Isaac Moradi 
> 
> Sent: Wednesday, June 2, 2021 4:36:50 PM
> To: arts_users...@mailman.rrz.uni-hamburg.de
> Subject: [arts-users] Issues with size interpolation - ARTS SSD
>  
> Hi all,
> 
> I am trying to do size interpolation using the assp.py included in the Python 
> data interface but there seems to be multiple issues with the size 
> interpolation function. There are some undefined variables that I fixed them 
> but then end up with the following error.
> I am trying this for IconSnow but the error should exist for any other habits.
> 
> Here is the message from Python:
> 
>   File "/data/users/imoradi/workspace/dda/DataInterfaces/Python/assp.py", 
> line 1154, in assp_interp_size
> p[i+1,...] = s.pha_mat_data
> ValueError: could not broadcast input array from shape (31,5,721,1,1,1,6) 
> into shape (31,5,91,1,1,1,6)
> 
> 
> Thanks
> 
> 
> Best,
> Isaac
> ___
> arts_users.mi mailing list
> arts_users.mi@lists.uni-hamburg.de 
> https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi 
> 
___
arts_users.mi mailing list
arts_users.mi@lists.uni-hamburg.de
https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi

Re: [arts-users] Issues with size interpolation - ARTS SSD

2021-06-04 Thread Vasileios Barlakas 
Hello Isaac,


I had a brief look at assp_interp_size. Indeed there are some bugs in this 
function that need to be fixed. I will allocate some time on this next week; 
sorry for any inconvenience caused. For the time being, I could give you some 
hints:


To use this function, all elements in S must have common frequency, 
temperature, and angular grid. Note here that, for some habits, e.g. IconSnow, 
the given sizes are not available in the same frequency and angular grids. This 
is one of the reasons why you get such an error message:


(31,5,721,1,1,1,6) into shape (31,5,91,1,1,1,6)


Note here that the function defines the dimensions of all the variables from 
the first element of the list, i.e., n_freq, n_temp, n_za, 1, 1, 1, 
n_phase_matrix_elements. For your case, the dimension at which python complains 
is the n_za (angular grid).


Following this rule, you could potentially fix any other issues. I hope this 
helps.


Best regards,

Vasileios



From: arts_users.mi-boun...@mailman.rrz.uni-hamburg.de 
 on behalf of Isaac Moradi 

Sent: Wednesday, June 2, 2021 4:36:50 PM
To: arts_users...@mailman.rrz.uni-hamburg.de
Subject: [arts-users] Issues with size interpolation - ARTS SSD

Hi all,

I am trying to do size interpolation using the assp.py included in the Python 
data interface but there seems to be multiple issues with the size 
interpolation function. There are some undefined variables that I fixed them 
but then end up with the following error.
I am trying this for IconSnow but the error should exist for any other habits.

Here is the message from Python:

  File "/data/users/imoradi/workspace/dda/DataInterfaces/Python/assp.py", line 
1154, in assp_interp_size
p[i+1,...] = s.pha_mat_data
ValueError: could not broadcast input array from shape (31,5,721,1,1,1,6) into 
shape (31,5,91,1,1,1,6)


Thanks


Best,
Isaac
___
arts_users.mi mailing list
arts_users.mi@lists.uni-hamburg.de
https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi
___
arts_users.mi mailing list
arts_users.mi@lists.uni-hamburg.de
https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi