Re: [arts-users] Issues with size interpolation - ARTS SSD
Isaac, I assume that CRTM includes temperatures up to 300K is to cover ice reaching the ground even when the surface temperature is well above 0C. In such cases the ice particles are not in equilibrium with the environment. Internally the temperature will be close to 0C. My suggestion is then to use the data for 0C also for higher temperatures. Bye, Patrick Isaac Moradi skrev: (6 juni 2021 05:25:38 CEST) >Hi Jana, > >I tried to include the za interpolation but it seems that it needs some >touch ups as there are some undefined variables. > >I did manage to do all the interpolations using the code that developed >myself and it seems to be working fine (see the attached file please). > >However, I have another issue which is unrelated to the code. In CRTM, >they have temperatures up to 300 K so I need to do extrapolation as >current SSD database doesn’t provide those temperatures. Those >temperatures may not even make sense in the real world but we don’t >intend to change them at the moment. The extrapolation creates some >negative values (please see the second attached plot). > >I wonder if any of you has any suggestion to avoid this problem? I am >not going to use those temperatures in my own work, but don’t want to >through some junks into CRTM coefficients either as someone may >eventually end up in a situation with T greater than 270 K. > > >The filenames have frequency and temperature included for reference. > >Thank you! > > >Best, >Isaac > > > > > > > > > >> On Jun 4, 2021, at 12:59 PM, Jana Mendrok >wrote: >> >> Hi Patrick et al, >> >> >> on the same set of grids. That is, the idea is to use these functions > >> first, before any size interpolation. This is at least indicated in >the >> documentation of assp_interp_size.m. >> >> >> No time for me to dig into the Python code (and anyhow don't have the > >> latest version at hand), and don't remember how closely Jana mimicked >my >> Matlab interface. >> >> Used to follow the Matlab implementation very closely. An >assp_interp_za exists. However, it's obviously nowhere applied (in the >original Zenodo published version; I did not check out newer ones). >> The header of assp_interp_size states the need for common grids. >Though I agree, that's not very prominently, hence easily missed by a >user and definitely missed when not called directly by the user. >Obviously, assp_interp_size misses a proper check that this requirement >is fulfilled... >> >> I guess, back in the days all size instances of one habit had the >same number of angles, so it wasn't necessary to apply that (what >speaks for that is that an IconSnow.rssp file exists; and that had been >created with exactly the code that was published, ie without >az-interpolation). >> >> Isaac/Vasileios - get_assp hence needs to be extended with a call of >assp_interp_za, in between the assp_import_ssdb and >assp.assp_interp_size (not deep enough into it anymore to judge whether >exact location matters...). >> >> In any case, Vasileios, thanks for caring for the code (and cleaning >up after me)! >> Best, >> >> -- >> Jana Mendrok, Ph.D. >> Deutscher Wetterdienst >> Offenbach am Main, Germany >> >> +49 (0)69 8062 3139 >> ___ >> arts_users.mi mailing list >> arts_users.mi@lists.uni-hamburg.de >> https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi ___ arts_users.mi mailing list arts_users.mi@lists.uni-hamburg.de https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi
Re: [arts-users] Issues with size interpolation - ARTS SSD
Hi Jana, I tried to include the za interpolation but it seems that it needs some touch ups as there are some undefined variables. I did manage to do all the interpolations using the code that developed myself and it seems to be working fine (see the attached file please). However, I have another issue which is unrelated to the code. In CRTM, they have temperatures up to 300 K so I need to do extrapolation as current SSD database doesn’t provide those temperatures. Those temperatures may not even make sense in the real world but we don’t intend to change them at the moment. The extrapolation creates some negative values (please see the second attached plot). I wonder if any of you has any suggestion to avoid this problem? I am not going to use those temperatures in my own work, but don’t want to through some junks into CRTM coefficients either as someone may eventually end up in a situation with T greater than 270 K. The filenames have frequency and temperature included for reference. Thank you! Best, Isaac > On Jun 4, 2021, at 12:59 PM, Jana Mendrok wrote: > > Hi Patrick et al, > > > on the same set of grids. That is, the idea is to use these functions > first, before any size interpolation. This is at least indicated in the > documentation of assp_interp_size.m. > > > No time for me to dig into the Python code (and anyhow don't have the > latest version at hand), and don't remember how closely Jana mimicked my > Matlab interface. > > Used to follow the Matlab implementation very closely. An assp_interp_za > exists. However, it's obviously nowhere applied (in the original Zenodo > published version; I did not check out newer ones). > The header of assp_interp_size states the need for common grids. Though I > agree, that's not very prominently, hence easily missed by a user and > definitely missed when not called directly by the user. Obviously, > assp_interp_size misses a proper check that this requirement is fulfilled... > > I guess, back in the days all size instances of one habit had the same number > of angles, so it wasn't necessary to apply that (what speaks for that is that > an IconSnow.rssp file exists; and that had been created with exactly the code > that was published, ie without az-interpolation). > > Isaac/Vasileios - get_assp hence needs to be extended with a call of > assp_interp_za, in between the assp_import_ssdb and assp.assp_interp_size > (not deep enough into it anymore to judge whether exact location matters...). > > In any case, Vasileios, thanks for caring for the code (and cleaning up after > me)! > Best, > > -- > Jana Mendrok, Ph.D. > Deutscher Wetterdienst > Offenbach am Main, Germany > > +49 (0)69 8062 3139 > ___ > arts_users.mi mailing list > arts_users.mi@lists.uni-hamburg.de > https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi ___ arts_users.mi mailing list arts_users.mi@lists.uni-hamburg.de https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi
Re: [arts-users] Issues with size interpolation - ARTS SSD
Hi Patrick et al, on the same set of grids. That is, the idea is to use these functions > first, before any size interpolation. This is at least indicated in the > documentation of assp_interp_size.m. > > > No time for me to dig into the Python code (and anyhow don't have the > latest version at hand), and don't remember how closely Jana mimicked my > Matlab interface. Used to follow the Matlab implementation very closely. An assp_interp_za exists. However, it's obviously nowhere applied (in the original Zenodo published version; I did not check out newer ones). The header of assp_interp_size states the need for common grids. Though I agree, that's not very prominently, hence easily missed by a user and definitely missed when not called directly by the user. Obviously, assp_interp_size misses a proper check that this requirement is fulfilled... I guess, back in the days all size instances of one habit had the same number of angles, so it wasn't necessary to apply that (what speaks for that is that an IconSnow.rssp file exists; and that had been created with exactly the code that was published, ie without az-interpolation). Isaac/Vasileios - get_assp hence needs to be extended with a call of assp_interp_za, in between the assp_import_ssdb and assp.assp_interp_size (not deep enough into it anymore to judge whether exact location matters...). In any case, Vasileios, thanks for caring for the code (and cleaning up after me)! Best, -- Jana Mendrok, Ph.D. Deutscher Wetterdienst Offenbach am Main, Germany +49 (0)69 8062 3139 ___ arts_users.mi mailing list arts_users.mi@lists.uni-hamburg.de https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi
Re: [arts-users] Issues with size interpolation - ARTS SSD
Hi Patrick, I am not actually directly calling any assp functions. I use rttov.get_assp as shown below to get what I need and send the new grids for f, T, and dveq along with the habit ID (I have been playing with allow_extrap to just allow to fill the gap for now but I know it may not be a good idea to extrapolate). The interpolations inside rttov.py are done in the order that Patrick mentioned. I didn’t try to call individual functions for interpolation but in my code, I am using the same interpolation technique (PchipInterpolator) and of course as long as the variables have the correct size then it works just fine. S,M,mD = rttov.get_assp(habID, orient, size_type='dveq', size_grid = dveq_mw, s_allow_extrap=True, t_grid=tgrid, t_allow_extrap=True, f_grid=fgrid, f_allow_extrap=True) Best, Isaac > On Jun 4, 2021, at 10:59 AM, Patrick Eriksson > wrote: > > Hi, > > Some quick input, for clarification. > > At least for the Matlab interface there are functions for interpolating in > frequency, angular grid and temperature, to get the data of all sizes on the > same set of grids. That is, the idea is to use these functions first, before > any size interpolation. This is at least indicated in the documentation of > assp_interp_size.m. > > No time for me to dig into the Python code (and anyhow don't have the latest > version at hand), and don't remember how closely Jana mimicked my Matlab > interface. But I should assume the same principle should be applied in > Python. Too bad if that not clear from the documentation. > > Bye, > > Patrick > > > On 2021-06-04 16:22, Vasileios Barlakas wrote: >> Hello Isaac, >> I had a brief look at assp_interp_size. Indeed there are some bugs in this >> function that need to be fixed. I will allocate some time on this next week; >> sorry for any inconvenience caused. For the time being, I could give you >> some hints: >> To use this function, all elements in S must have common frequency, >> temperature, and angular grid. Note here that, for some habits, e.g. >> IconSnow, the given sizes are not available in the same frequency and >> angular grids. This is one of the reasons why you get such an error message: >> (31,5,721,1,1,1,6) into shape (31,5,91,1,1,1,6) >> Note here that the function defines the dimensions of all the variables from >> the first element of the list, i.e., n_freq, n_temp, n_za, 1, 1, 1, >> n_phase_matrix_elements. For your case, the dimension at which python >> complains is the n_za (angular grid). >> Following this rule, you could potentially fix any other issues. I hope this >> helps. >> Best regards, >> Vasileios >> >> *From:* arts_users.mi-boun...@mailman.rrz.uni-hamburg.de >> on behalf of Isaac Moradi >> >> *Sent:* Wednesday, June 2, 2021 4:36:50 PM >> *To:* arts_users...@mailman.rrz.uni-hamburg.de >> *Subject:* [arts-users] Issues with size interpolation - ARTS SSD >> Hi all, >> I am trying to do size interpolation using the assp.py included in the >> Python data interface but there seems to be multiple issues with the size >> interpolation function. There are some undefined variables that I fixed them >> but then end up with the following error. >> I am trying this for IconSnow but the error should exist for any other >> habits. >> Here is the message from Python: >> File "/data/users/imoradi/workspace/dda/DataInterfaces/Python/assp.py", >> line 1154, in assp_interp_size >> p[i+1,...] = s.pha_mat_data >> ValueError: could not broadcast input array from shape (31,5,721,1,1,1,6) >> into shape (31,5,91,1,1,1,6) >> Thanks >> Best, >> Isaac >> ___ >> arts_users.mi mailing list >> arts_users.mi@lists.uni-hamburg.de >> https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi >> ___ >> arts_users.mi mailing list >> arts_users.mi@lists.uni-hamburg.de >> https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi > ___ > arts_users.mi mailing list > arts_users.mi@lists.uni-hamburg.de > https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi ___ arts_users.mi mailing list arts_users.mi@lists.uni-hamburg.de https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi
Re: [arts-users] Issues with size interpolation - ARTS SSD
Hi, Some quick input, for clarification. At least for the Matlab interface there are functions for interpolating in frequency, angular grid and temperature, to get the data of all sizes on the same set of grids. That is, the idea is to use these functions first, before any size interpolation. This is at least indicated in the documentation of assp_interp_size.m. No time for me to dig into the Python code (and anyhow don't have the latest version at hand), and don't remember how closely Jana mimicked my Matlab interface. But I should assume the same principle should be applied in Python. Too bad if that not clear from the documentation. Bye, Patrick On 2021-06-04 16:22, Vasileios Barlakas wrote: Hello Isaac, I had a brief look at assp_interp_size. Indeed there are some bugs in this function that need to be fixed. I will allocate some time on this next week; sorry for any inconvenience caused. For the time being, I could give you some hints: To use this function, all elements in S must have common frequency, temperature, and angular grid. Note here that, for some habits, e.g. IconSnow, the given sizes are not available in the same frequency and angular grids. This is one of the reasons why you get such an error message: (31,5,721,1,1,1,6) into shape (31,5,91,1,1,1,6) Note here that the function defines the dimensions of all the variables from the first element of the list, i.e., n_freq, n_temp, n_za, 1, 1, 1, n_phase_matrix_elements. For your case, the dimension at which python complains is the n_za (angular grid). Following this rule, you could potentially fix any other issues. I hope this helps. Best regards, Vasileios *From:* arts_users.mi-boun...@mailman.rrz.uni-hamburg.de on behalf of Isaac Moradi *Sent:* Wednesday, June 2, 2021 4:36:50 PM *To:* arts_users...@mailman.rrz.uni-hamburg.de *Subject:* [arts-users] Issues with size interpolation - ARTS SSD Hi all, I am trying to do size interpolation using the assp.py included in the Python data interface but there seems to be multiple issues with the size interpolation function. There are some undefined variables that I fixed them but then end up with the following error. I am trying this for IconSnow but the error should exist for any other habits. Here is the message from Python: File "/data/users/imoradi/workspace/dda/DataInterfaces/Python/assp.py", line 1154, in assp_interp_size p[i+1,...] = s.pha_mat_data ValueError: could not broadcast input array from shape (31,5,721,1,1,1,6) into shape (31,5,91,1,1,1,6) Thanks Best, Isaac ___ arts_users.mi mailing list arts_users.mi@lists.uni-hamburg.de https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi ___ arts_users.mi mailing list arts_users.mi@lists.uni-hamburg.de https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi ___ arts_users.mi mailing list arts_users.mi@lists.uni-hamburg.de https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi
Re: [arts-users] Issues with size interpolation - ARTS SSD
HI Vasileios, Thanks for the hints. Meanwhile, I spent sometime and wrote a piece of code to read the data in and interpolate them. I will they to see if can fix it according to what you suggested. In any case, once you have fixed the code, I will compare what I have developed with the main code. Thanks again. Best, Isaac > On Jun 4, 2021, at 10:22 AM, Vasileios Barlakas > wrote: > > Hello Isaac, > > I had a brief look at assp_interp_size. Indeed there are some bugs in this > function that need to be fixed. I will allocate some time on this next week; > sorry for any inconvenience caused. For the time being, I could give you some > hints: > > To use this function, all elements in S must have common frequency, > temperature, and angular grid. Note here that, for some habits, e.g. > IconSnow, the given sizes are not available in the same frequency and angular > grids. This is one of the reasons why you get such an error message: > > (31,5,721,1,1,1,6) into shape (31,5,91,1,1,1,6) > > Note here that the function defines the dimensions of all the variables from > the first element of the list, i.e., n_freq, n_temp, n_za, 1, 1, 1, > n_phase_matrix_elements. For your case, the dimension at which python > complains is the n_za (angular grid). > > Following this rule, you could potentially fix any other issues. I hope this > helps. > > Best regards, > Vasileios > > From: arts_users.mi-boun...@mailman.rrz.uni-hamburg.de > on behalf of Isaac Moradi > > Sent: Wednesday, June 2, 2021 4:36:50 PM > To: arts_users...@mailman.rrz.uni-hamburg.de > Subject: [arts-users] Issues with size interpolation - ARTS SSD > > Hi all, > > I am trying to do size interpolation using the assp.py included in the Python > data interface but there seems to be multiple issues with the size > interpolation function. There are some undefined variables that I fixed them > but then end up with the following error. > I am trying this for IconSnow but the error should exist for any other habits. > > Here is the message from Python: > > File "/data/users/imoradi/workspace/dda/DataInterfaces/Python/assp.py", > line 1154, in assp_interp_size > p[i+1,...] = s.pha_mat_data > ValueError: could not broadcast input array from shape (31,5,721,1,1,1,6) > into shape (31,5,91,1,1,1,6) > > > Thanks > > > Best, > Isaac > ___ > arts_users.mi mailing list > arts_users.mi@lists.uni-hamburg.de <mailto:arts_users.mi@lists.uni-hamburg.de> > https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi > <https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi> ___ arts_users.mi mailing list arts_users.mi@lists.uni-hamburg.de https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi
Re: [arts-users] Issues with size interpolation - ARTS SSD
Hello Isaac, I had a brief look at assp_interp_size. Indeed there are some bugs in this function that need to be fixed. I will allocate some time on this next week; sorry for any inconvenience caused. For the time being, I could give you some hints: To use this function, all elements in S must have common frequency, temperature, and angular grid. Note here that, for some habits, e.g. IconSnow, the given sizes are not available in the same frequency and angular grids. This is one of the reasons why you get such an error message: (31,5,721,1,1,1,6) into shape (31,5,91,1,1,1,6) Note here that the function defines the dimensions of all the variables from the first element of the list, i.e., n_freq, n_temp, n_za, 1, 1, 1, n_phase_matrix_elements. For your case, the dimension at which python complains is the n_za (angular grid). Following this rule, you could potentially fix any other issues. I hope this helps. Best regards, Vasileios From: arts_users.mi-boun...@mailman.rrz.uni-hamburg.de on behalf of Isaac Moradi Sent: Wednesday, June 2, 2021 4:36:50 PM To: arts_users...@mailman.rrz.uni-hamburg.de Subject: [arts-users] Issues with size interpolation - ARTS SSD Hi all, I am trying to do size interpolation using the assp.py included in the Python data interface but there seems to be multiple issues with the size interpolation function. There are some undefined variables that I fixed them but then end up with the following error. I am trying this for IconSnow but the error should exist for any other habits. Here is the message from Python: File "/data/users/imoradi/workspace/dda/DataInterfaces/Python/assp.py", line 1154, in assp_interp_size p[i+1,...] = s.pha_mat_data ValueError: could not broadcast input array from shape (31,5,721,1,1,1,6) into shape (31,5,91,1,1,1,6) Thanks Best, Isaac ___ arts_users.mi mailing list arts_users.mi@lists.uni-hamburg.de https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi ___ arts_users.mi mailing list arts_users.mi@lists.uni-hamburg.de https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi
[arts-users] Issues with size interpolation - ARTS SSD
Hi all, I am trying to do size interpolation using the assp.py included in the Python data interface but there seems to be multiple issues with the size interpolation function. There are some undefined variables that I fixed them but then end up with the following error. I am trying this for IconSnow but the error should exist for any other habits. Here is the message from Python: File "/data/users/imoradi/workspace/dda/DataInterfaces/Python/assp.py", line 1154, in assp_interp_size p[i+1,...] = s.pha_mat_data ValueError: could not broadcast input array from shape (31,5,721,1,1,1,6) into shape (31,5,91,1,1,1,6) Thanks Best, Isaac ___ arts_users.mi mailing list arts_users.mi@lists.uni-hamburg.de https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi