Re: [arts-users] Tropospheric continuum retrieval in ARTS
Dear Patrick, Thanks a lot for the help and feedback, this is really appreciated. For the second order term of the baseline, I believe one of the reason for it lies in our sideband suppression method. I'll now try to add that effect in the sensor_response variable and see if this impacts the baseline accordingly. Thanks, Eric De : Patrick Eriksson Envoyé : mardi 22 juin 2021 20:39:56 À : Sauvageat, Eric (IAP); arts_users...@mailman.rrz.uni-hamburg.de Cc : rita.edit.kaj...@ltu.se Objet : Re: [arts-users] Tropospheric continuum retrieval in ARTS Eric, > (1998)). The values of the poly_order=0 are now quite small (mostly > lower than 0.1K), however, I still only get positive values. Is that > something that could be expected from the way the function > "retrievalAddPolyfit" works maybe ? No, there is nothing that promotes positive values. But you now get so small values that nothing to worry about. You get very consistent values for poly_order 1 and 2, despite that the tropospheric opacity varies quite a bit. That should be an indication on that receiver is stable and the baseline+tropospheric part of the retrievals is working well. > Would you have any tips on how to implement the retrieval of a single > scaling parameter for a full water vapor profile ? I believe this is > more or less what the CONTABS_DO parameter was implementing according to > the first Qpack documentation ? This you do by setting the retrieval grid to have a single point (e.g. 500 hPa) and set the corresponding covariance matrix to have size 1x1. A suitable value for it is maybe 1 (meaning a 100% uncertainty at all altitudes, fully correlated). Bye, Patrick ___ arts_users.mi mailing list arts_users.mi@lists.uni-hamburg.de https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi
Re: [arts-users] Tropospheric continuum retrieval in ARTS
Eric, (1998)). The values of the poly_order=0 are now quite small (mostly lower than 0.1K), however, I still only get positive values. Is that something that could be expected from the way the function "retrievalAddPolyfit" works maybe ? No, there is nothing that promotes positive values. But you now get so small values that nothing to worry about. You get very consistent values for poly_order 1 and 2, despite that the tropospheric opacity varies quite a bit. That should be an indication on that receiver is stable and the baseline+tropospheric part of the retrievals is working well. Would you have any tips on how to implement the retrieval of a single scaling parameter for a full water vapor profile ? I believe this is more or less what the CONTABS_DO parameter was implementing according to the first Qpack documentation ? This you do by setting the retrieval grid to have a single point (e.g. 500 hPa) and set the corresponding covariance matrix to have size 1x1. A suitable value for it is maybe 1 (meaning a 100% uncertainty at all altitudes, fully correlated). Bye, Patrick ___ arts_users.mi mailing list arts_users.mi@lists.uni-hamburg.de https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi
Re: [arts-users] Tropospheric continuum retrieval in ARTS
Hi, Thanks, I'll see if I can reduce that value by lowering the apriori cov for the polyfit order 0 then. So the "single continuum fit" was really just setting a single altitude and retrieving a singe H2O VMR value at this altitude whereas the profile case was the same retrievals (species, apriori cov) but with an apriori profile instead of a single value. I'll check the rel unit then, I was not aware this could be used to retrieved a single scaling value on a profile. Best regards, Eric De : Patrick Eriksson Envoyé : vendredi 11 juin 2021 10:23:58 À : Sauvageat, Eric (IAP); arts_users...@mailman.rrz.uni-hamburg.de Objet : Re: [arts-users] Tropospheric continuum retrieval in ARTS Eric, It seems indeed that poly_order=0 is involved. I think you want this value to be small. Especially as it seems to only be positive. This behaviour would be OK if you thought that you have a calibration error that is random, but just one-sided. Sounds not reasonable. I think more realistic is to trust the calibration and let the tropospheric part take care of this part. poly_order=0 and the tropospheric correction have a very similar impact on the "baseline". The resulting tropospheric attenuation depends to some extent on the retrieved humidity profile, but that should be secondary. But you can of course take a look at how the tropospheric humidity profile look in both cases and compare. By the way, how do you do the single column case? I suggest to use the rel unit. So you scale the profile with a single relative value. Using VMR for this is less realistic (here you would change the full profile with the same VMR value, that is not realistic). Regards, Patrick On 2021-06-11 10:02, eric.sauvag...@iap.unibe.ch wrote: > Dear Patrick, > > > Thanks for you answer. > > > I did some further investigations on the value of the polyfit and as > expected the value of the constant terms is indeed quite different for > both options and very similar for 1st and 2nd degree terms (I appended > some basic representative plots to give an idea). > > > More specifically, it is smaller in the case of the "single continuum > fit" which is the one giving a +10% ozone profile. It would mean > that tropospheric attenuation is considered to be larger in the case of > the single continuum fit compared to the other option. I think this is > then consistent with the 10% higher ozone values obtained in this case, > would you agree ? > > > I remember having tried different constraint on the constant term but > only making it larger so I should redo some test reducing it more. I > believe one way to check if the tropospheric attenuation is correct is > to compare the opacity value computed from ARTS to one computed manually > on the spectra. > > I'll check that as well but thanks for your inputs already. > > > Best regards, > > Eric > > > > > > > *De :* Patrick Eriksson > *Envoyé :* mercredi 9 juin 2021 23:39:33 > *À :* Sauvageat, Eric (IAP); arts_users...@mailman.rrz.uni-hamburg.de > *Objet :* Re: [arts-users] Tropospheric continuum retrieval in ARTS > Eric, > > I don't know about a fixed setup for dealing with the troposphere in > observations of this type. As your observations at max gives one piece > of information for the troposphere, I would say that the single > retrieval point setup makes most sense. In any case it is plausible option. > > I assume that you fit the measurements in both cases. > > If I get it right, your main worry is that you get a 10% difference in > the ozone profile between the options. This should likely originate in > that the retrievals give you a troposphere having a 10% difference in > transmission. > > My suggestion is then to look at the polyfit part. Is the fitted polyfit > the same between the options? My guess is that it differs. And that > gives room for retrieving a different tropospheric transmission. > > And it could be reasons to anyhow consider the polyfit part. I assume > you have a good calibration and the uncertainty in the overall > "baseline" level is due to the troposphere. Or expressed differently, > you want to fit the overall baseline level by changing H2O in the > troposphere, not by the polyfit. Or more exactly you want the first > polyfit coefficient to be small, the polyfit should just take care of > the "wiggling". To achieve this you should set the a priori uncertainty > for the first (0-order) coefficient to be very small, to effectively > enforce a low measurement response for the coefficient. With this, the > retrieval will have to fully adjust the overall baseline level by the > H2O profile, independen
Re: [arts-users] Tropospheric continuum retrieval in ARTS
Dear Mathias, Nice, I am happy to see that other people are working on this topic as well. In terms of apriori, I am indeed doing the same kind of forcing on the ozone profile to avoid some high ozone VMR retrieved at low altitudes, problem that I got before using such a forcing. May I ask what you are using as apriori profile and covariance matrix for the H2O profile ? I tested ECMWF and Fascod as apriori for the H2O and it did not produce significant changes, the thing that matters the most in that view seems to be the apriori cov value for H2O setup with constant std deviation around 6e-4 or 6 ppm (I do all my retrievals in vmr units, both for O3 and H2O). Note that this value was mostly a guess but works (enable convergence...) for all my retrievals. This might change soon though depending on the current discussion.. Regarding the polyfit value, I am also not providing any specific apriori value, only a covariance matrix for each of the terms. Best regards, Eric De : Mathias Milz Envoyé : jeudi 10 juin 2021 08:05:53 À : Sauvageat, Eric (IAP); arts_users.mi@lists.uni-hamburg.de Cc : Rita Edit Kajtar; Uwe Raffalski Objet : Re: Tropospheric continuum retrieval in ARTS Dear Eric, We are working with a similar setup (Just at another frequency) for the radiometers in Kiruna. For water vapour we use a rather loose constraint (large S_x/S_a value) to allow the fit of the profile for the H2O background (co-fit). Of course, H2O can be unrealistic as we just use it to fit the continuum/baseline For O3 we use an altitude dependent constraint with a rather strong constraint below the tropopause (where we know the values are “negligible”) and above the sensitive area where we force the retrieval to the a priori knowledge. So we allow reasonable fitting only for the altitudes where the Jacobians show decent sensitivity. Note: We have good results with “rel” jacobians. “vmr”/absolute Jacobians caused problems. We also co-fit a baseline using “polyfit” of the lowest degree. However here we did not yet find a solution that we can start with a preselected value (e.g. an instrument-dependend baseline) so the a priori value is by default 0 and the a standard constrain constraint might cause difficulties with large large baselines due to high humidity or clouds with precipitation. Maybe this helps. Best regards, Mathias From: arts_users.mi-boun...@lists.uni-hamburg.de on behalf of eric.sauvag...@iap.unibe.ch Date: Wednesday, 9 June 2021 at 16:22 To: arts_users.mi@lists.uni-hamburg.de Subject: [arts-users] Tropospheric continuum retrieval in ARTS Dear ARTS community, I am doing stratospheric O3 retrieval with a ground-based radiometer (f=142GHz) and am trying to deal with the absorption contribution of the troposphere directly in the OEM implemented in ARTS (avoiding tropospheric correction prior to the retrievals). Up to now, I took inspiration from "qpack2_demo2.m" which suggests (if I understood it correctly) to implement a "H2O-PWR98, H2O" retrieval (main contributor of tropospheric opacity at these frequencies) on a lower atmosphere retrieval grid. This results in a water vapor profile retrieved together with my main ozone retrievals. Of course this profile has no good measurement response as my ozone radiometer is not designed to retrieve any H2O profile, but it seems that it provides the "right amount of opacity" needed to explain my spectrum. In addition, note that I am also performing a polyfit retrieval of degree 2 which is also fitting a constant term on my spectra which also probably contributes somehow to fit the global continuum absorption. I found out recently, that such a continuum retrieval was implemented in QPack1 (activated with "Q.CONTABS_DO") and from my understanding, it does not seem to retrieve any H2O profiles but only single values for the continuum (which somehow makes more sense to me). So I did try to provide a single grid retrieval point and single value for H2O cov matrix and it seems to work equally good as the retrieval including a full H2O profile (in the sense of convergence, correlation between both time series, ...) but it has a constant +10% VMR offset on my whole ozone profiles and I have no clue why. Also, I have made different tests to check the impact of the selected species (continuum vs full absorption model defined with or without H2O) but it did only produce slight changes in the results. As well, the height of the H2O grid or its altitude resolution does not seem to have significant impact on the retrievals. Sorry for this long email but I am really puzzled in what is the best way to deal with continuum absorption in ARTS and what I might be doing wrong. Therefore, any kind of feedback or help regarding this would be much appreciated. If needed, I can also provide examples plots of MR,
Re: [arts-users] Tropospheric continuum retrieval in ARTS
Dear Eric, We are working with a similar setup (Just at another frequency) for the radiometers in Kiruna. For water vapour we use a rather loose constraint (large S_x/S_a value) to allow the fit of the profile for the H2O background (co-fit). Of course, H2O can be unrealistic as we just use it to fit the continuum/baseline For O3 we use an altitude dependent constraint with a rather strong constraint below the tropopause (where we know the values are “negligible”) and above the sensitive area where we force the retrieval to the a priori knowledge. So we allow reasonable fitting only for the altitudes where the Jacobians show decent sensitivity. Note: We have good results with “rel” jacobians. “vmr”/absolute Jacobians caused problems. We also co-fit a baseline using “polyfit” of the lowest degree. However here we did not yet find a solution that we can start with a preselected value (e.g. an instrument-dependend baseline) so the a priori value is by default 0 and the a standard constrain constraint might cause difficulties with large large baselines due to high humidity or clouds with precipitation. Maybe this helps. Best regards, Mathias From: arts_users.mi-boun...@lists.uni-hamburg.de on behalf of eric.sauvag...@iap.unibe.ch Date: Wednesday, 9 June 2021 at 16:22 To: arts_users.mi@lists.uni-hamburg.de Subject: [arts-users] Tropospheric continuum retrieval in ARTS Dear ARTS community, I am doing stratospheric O3 retrieval with a ground-based radiometer (f=142GHz) and am trying to deal with the absorption contribution of the troposphere directly in the OEM implemented in ARTS (avoiding tropospheric correction prior to the retrievals). Up to now, I took inspiration from "qpack2_demo2.m" which suggests (if I understood it correctly) to implement a "H2O-PWR98, H2O" retrieval (main contributor of tropospheric opacity at these frequencies) on a lower atmosphere retrieval grid. This results in a water vapor profile retrieved together with my main ozone retrievals. Of course this profile has no good measurement response as my ozone radiometer is not designed to retrieve any H2O profile, but it seems that it provides the "right amount of opacity" needed to explain my spectrum. In addition, note that I am also performing a polyfit retrieval of degree 2 which is also fitting a constant term on my spectra which also probably contributes somehow to fit the global continuum absorption. I found out recently, that such a continuum retrieval was implemented in QPack1 (activated with "Q.CONTABS_DO") and from my understanding, it does not seem to retrieve any H2O profiles but only single values for the continuum (which somehow makes more sense to me). So I did try to provide a single grid retrieval point and single value for H2O cov matrix and it seems to work equally good as the retrieval including a full H2O profile (in the sense of convergence, correlation between both time series, ...) but it has a constant +10% VMR offset on my whole ozone profiles and I have no clue why. Also, I have made different tests to check the impact of the selected species (continuum vs full absorption model defined with or without H2O) but it did only produce slight changes in the results. As well, the height of the H2O grid or its altitude resolution does not seem to have significant impact on the retrievals. Sorry for this long email but I am really puzzled in what is the best way to deal with continuum absorption in ARTS and what I might be doing wrong. Therefore, any kind of feedback or help regarding this would be much appreciated. If needed, I can also provide examples plots of MR, AVK or profiles (not sure how it works for mailing list though). Many thanks in advance, Eric ___ arts_users.mi mailing list arts_users.mi@lists.uni-hamburg.de https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi
Re: [arts-users] Tropospheric continuum retrieval in ARTS
Eric, I don't know about a fixed setup for dealing with the troposphere in observations of this type. As your observations at max gives one piece of information for the troposphere, I would say that the single retrieval point setup makes most sense. In any case it is plausible option. I assume that you fit the measurements in both cases. If I get it right, your main worry is that you get a 10% difference in the ozone profile between the options. This should likely originate in that the retrievals give you a troposphere having a 10% difference in transmission. My suggestion is then to look at the polyfit part. Is the fitted polyfit the same between the options? My guess is that it differs. And that gives room for retrieving a different tropospheric transmission. And it could be reasons to anyhow consider the polyfit part. I assume you have a good calibration and the uncertainty in the overall "baseline" level is due to the troposphere. Or expressed differently, you want to fit the overall baseline level by changing H2O in the troposphere, not by the polyfit. Or more exactly you want the first polyfit coefficient to be small, the polyfit should just take care of the "wiggling". To achieve this you should set the a priori uncertainty for the first (0-order) coefficient to be very small, to effectively enforce a low measurement response for the coefficient. With this, the retrieval will have to fully adjust the overall baseline level by the H2O profile, independently on the grid and a priori uncertainty you use for H2O. Bye, Patrick On 2021-06-09 16:21, eric.sauvag...@iap.unibe.ch wrote: Dear ARTS community, I am doing stratospheric O3 retrieval with a ground-based radiometer (f=142GHz) and am trying to deal with the absorption contribution of the troposphere directly in the OEM implemented in ARTS (avoiding tropospheric correction prior to the retrievals). Up to now, I took inspiration from "qpack2_demo2.m" which suggests (if I understood it correctly) to implement a "H2O-PWR98, H2O" retrieval (main contributor of tropospheric opacity at these frequencies) on a lower atmosphere retrieval grid. This results in a water vapor profile retrieved together with my main ozone retrievals. Of course this profile has no good measurement response as my ozone radiometer is not designed to retrieve any H2O profile, but it seems that it provides the "right amount of opacity" needed to explain my spectrum.In addition, note that I am also performing a polyfit retrieval of degree 2 which is also fitting a constant term on my spectra which also probably contributes somehow to fit the global continuum absorption. I found out recently, that such a continuum retrieval was implemented in QPack1 (activated with "Q.CONTABS_DO") and from my understanding, it does not seem to retrieve any H2O profiles but only single values for the continuum (which somehow makes more sense to me). So I did try to provide a single grid retrieval point and single value for H2O cov matrix and it seems to work equally good as the retrieval including a full H2O profile (in the sense of convergence, correlation between both time series, ...) but it has a constant +10% VMR offset on my whole ozone profiles and I have no clue why. Also, I have made different tests to check the impact of the selected species (continuum vs full absorption model defined with or without H2O) but it did only produce slight changes in the results. As well, the height of the H2O grid or its altitude resolution does not seem to have significant impact on the retrievals. Sorry for this long email but I am really puzzled in what is the best way to deal with continuum absorption in ARTS and what I might be doing wrong. Therefore, any kind of feedback or help regarding this would be much appreciated. If needed, I can also provide examples plots of MR, AVK or profiles (not sure how it works for mailing list though). Many thanks in advance, Eric ___ arts_users.mi mailing list arts_users.mi@lists.uni-hamburg.de https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi ___ arts_users.mi mailing list arts_users.mi@lists.uni-hamburg.de https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi
[arts-users] Tropospheric continuum retrieval in ARTS
Dear ARTS community, I am doing stratospheric O3 retrieval with a ground-based radiometer (f=142GHz) and am trying to deal with the absorption contribution of the troposphere directly in the OEM implemented in ARTS (avoiding tropospheric correction prior to the retrievals). Up to now, I took inspiration from "qpack2_demo2.m" which suggests (if I understood it correctly) to implement a "H2O-PWR98, H2O" retrieval (main contributor of tropospheric opacity at these frequencies) on a lower atmosphere retrieval grid. This results in a water vapor profile retrieved together with my main ozone retrievals. Of course this profile has no good measurement response as my ozone radiometer is not designed to retrieve any H2O profile, but it seems that it provides the "right amount of opacity" needed to explain my spectrum. In addition, note that I am also performing a polyfit retrieval of degree 2 which is also fitting a constant term on my spectra which also probably contributes somehow to fit the global continuum absorption. I found out recently, that such a continuum retrieval was implemented in QPack1 (activated with "Q.CONTABS_DO") and from my understanding, it does not seem to retrieve any H2O profiles but only single values for the continuum (which somehow makes more sense to me). So I did try to provide a single grid retrieval point and single value for H2O cov matrix and it seems to work equally good as the retrieval including a full H2O profile (in the sense of convergence, correlation between both time series, ...) but it has a constant +10% VMR offset on my whole ozone profiles and I have no clue why. Also, I have made different tests to check the impact of the selected species (continuum vs full absorption model defined with or without H2O) but it did only produce slight changes in the results. As well, the height of the H2O grid or its altitude resolution does not seem to have significant impact on the retrievals. Sorry for this long email but I am really puzzled in what is the best way to deal with continuum absorption in ARTS and what I might be doing wrong. Therefore, any kind of feedback or help regarding this would be much appreciated. If needed, I can also provide examples plots of MR, AVK or profiles (not sure how it works for mailing list though). Many thanks in advance, Eric ___ arts_users.mi mailing list arts_users.mi@lists.uni-hamburg.de https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi
