Re: [arts-users] atm.scenario & cov-mat issues
Oh, right. my bad. Sx is the apriori, not the retrieval error *facepalm* Thanks, Simon! On Thu, Nov 28, 2019 at 7:55 AM Simon Pfreundschuh < simon.pfreundsc...@chalmers.se> wrote: > > The covmat_sx is output (as far as my little OEM knowledge goes). And as > a curious user, I of course like to know what is in my output/what's the > meaning of my output (and not just half of it. > particularly when it makes > me getting unsure, whether I use the right part of this > larger-than-expected output variable) ;) > > looking forward to the next release then! :) > > > This is not true though. The particular covariance matrix format is used > only for the input covariance matrices. > > The ones that can be considered output (covmat_ss, covmat_so) are > generally dense so there's nothing to be gained > > in representing them as block-diagonal matrices. > > > Cheers, > > > Simon > -- > *From:* Jana Mendrok > *Sent:* Wednesday, November 27, 2019 7:42:26 PM > *To:* Simon Pfreundschuh > *Cc:* Rita Kajtar; arts_users...@mailman.rrz.uni-hamburg.de > *Subject:* Re: [arts-users] atm.scenario & cov-mat issues > > Hi Simon, hi all, > > >> I might have expressed myself not very clearly but I remain convinced >> that the assertion >> >> is caused by the pressure retrieval grid. The smallest value in it is >> smaller than that in >> >> the pressure grid. This is similar but unrelated to the first error where >> the p_grid extended >> >> out of the range of the raw p_grid. >> > I see (I indeed misunderstood you before. Sorry.). > > I agree, this then should be caught by an error (with a proper error > message). > > >> Regarding covariance matrices: There is currently no extensive >> documentation for covariance matrices >> >> but documentation of workspace groups is something that we are working on >> for the next release. >> >> Anyhow, the user in general does not have to worry about the inverse >> blocks if she doesn't want to set >> >> them explicitly. >> > The covmat_sx is output (as far as my little OEM knowledge goes). And as a > curious user, I of course like to know what is in my output/what's the > meaning of my output (and not just half of it. particularly when it makes > me getting unsure, whether I use the right part of this > larger-than-expected output variable) ;) > looking forward to the next release then! :) > > Finally, there was another bug in the loading of the covariance matrices >> from xml files, which caused >> >> the behavior that you were describing. This one is fixed now in the >> development version of typhon. >> > Ok, thanks! > > Regards, > Jana > > > > -- > Jana Mendrok, Ph.D. > Deutscher Wetterdienst > Offenbach am Main, Germany > > +49 (0)69 8062 3139 > -- Jana Mendrok, Ph.D. Deutscher Wetterdienst Offenbach am Main, Germany +49 (0)69 8062 3139 ___ arts_users.mi mailing list arts_users.mi@lists.uni-hamburg.de https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi
Re: [arts-users] atm.scenario & cov-mat issues
> The covmat_sx is output (as far as my little OEM knowledge goes). And as a > curious user, I of course like to know what is in my output/what's the > meaning of my output (and not just half of it. > particularly when it makes > me getting unsure, whether I use the right part of this larger-than-expected > output variable) ;) > looking forward to the next release then! :) This is not true though. The particular covariance matrix format is used only for the input covariance matrices. The ones that can be considered output (covmat_ss, covmat_so) are generally dense so there's nothing to be gained in representing them as block-diagonal matrices. Cheers, Simon From: Jana Mendrok Sent: Wednesday, November 27, 2019 7:42:26 PM To: Simon Pfreundschuh Cc: Rita Kajtar; arts_users...@mailman.rrz.uni-hamburg.de Subject: Re: [arts-users] atm.scenario & cov-mat issues Hi Simon, hi all, I might have expressed myself not very clearly but I remain convinced that the assertion is caused by the pressure retrieval grid. The smallest value in it is smaller than that in the pressure grid. This is similar but unrelated to the first error where the p_grid extended out of the range of the raw p_grid. I see (I indeed misunderstood you before. Sorry.). I agree, this then should be caught by an error (with a proper error message). Regarding covariance matrices: There is currently no extensive documentation for covariance matrices but documentation of workspace groups is something that we are working on for the next release. Anyhow, the user in general does not have to worry about the inverse blocks if she doesn't want to set them explicitly. The covmat_sx is output (as far as my little OEM knowledge goes). And as a curious user, I of course like to know what is in my output/what's the meaning of my output (and not just half of it. particularly when it makes me getting unsure, whether I use the right part of this larger-than-expected output variable) ;) looking forward to the next release then! :) Finally, there was another bug in the loading of the covariance matrices from xml files, which caused the behavior that you were describing. This one is fixed now in the development version of typhon. Ok, thanks! Regards, Jana -- Jana Mendrok, Ph.D. Deutscher Wetterdienst Offenbach am Main, Germany +49 (0)69 8062 3139 ___ arts_users.mi mailing list arts_users.mi@lists.uni-hamburg.de https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi
Re: [arts-users] atm.scenario & cov-mat issues
Hi Simon, hi all, > I might have expressed myself not very clearly but I remain convinced that > the assertion > > is caused by the pressure retrieval grid. The smallest value in it is > smaller than that in > > the pressure grid. This is similar but unrelated to the first error where > the p_grid extended > > out of the range of the raw p_grid. > I see (I indeed misunderstood you before. Sorry.). I agree, this then should be caught by an error (with a proper error message). > Regarding covariance matrices: There is currently no extensive > documentation for covariance matrices > > but documentation of workspace groups is something that we are working on > for the next release. > > Anyhow, the user in general does not have to worry about the inverse > blocks if she doesn't want to set > > them explicitly. > The covmat_sx is output (as far as my little OEM knowledge goes). And as a curious user, I of course like to know what is in my output/what's the meaning of my output (and not just half of it. particularly when it makes me getting unsure, whether I use the right part of this larger-than-expected output variable) ;) looking forward to the next release then! :) Finally, there was another bug in the loading of the covariance matrices > from xml files, which caused > > the behavior that you were describing. This one is fixed now in the > development version of typhon. > Ok, thanks! Regards, Jana -- Jana Mendrok, Ph.D. Deutscher Wetterdienst Offenbach am Main, Germany +49 (0)69 8062 3139 ___ arts_users.mi mailing list arts_users.mi@lists.uni-hamburg.de https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi
Re: [arts-users] atm.scenario & cov-mat issues
Hi Jana, I might have expressed myself not very clearly but I remain convinced that the assertion is caused by the pressure retrieval grid. The smallest value in it is smaller than that in the pressure grid. This is similar but unrelated to the first error where the p_grid extended out of the range of the raw p_grid. Regarding covariance matrices: There is currently no extensive documentation for covariance matrices but documentation of workspace groups is something that we are working on for the next release. Anyhow, the user in general does not have to worry about the inverse blocks if she doesn't want to set them explicitly. Finally, there was another bug in the loading of the covariance matrices from xml files, which caused the behavior that you were describing. This one is fixed now in the development version of typhon. Cheers, Simon From: Jana Mendrok Sent: Wednesday, November 27, 2019 11:21:05 AM To: Simon Pfreundschuh Cc: Rita Kajtar; arts_users...@mailman.rrz.uni-hamburg.de Subject: Re: [arts-users] atm.scenario & cov-mat issues Hi Simon, hi all, the (p_grid) error is irrelevant (it is caught, it got fixed). The ASSERTION is the issue. It occurs once TestOEM.arts' default atmosphere (tropical standard atmo from AFGL/Fascode) is replaced with the subarctic winter AFGL/Fascode atmosphere (and the p_grid setting suitably adapted). Regarding the covariance matrix... First, I think it should occur in the documentation that covmat_sx contains the blocks and their inverse (does that also mean the inverse blocks together form the complete inverse Sx? Sorry, my instantaneous vector math knowledge is not good enough to judge that myself :( ). Did we just miss tzhat piece of information or is it not in the docs (yet)? Second, Rita reads covmat_sx in from xml file using typhon.arts.xml.load. Would that be the same as the ws.covmat_sx in your example? I had the feeling something went wrong there, too. typhon did not complain, but len(covmat_sx.blocks) was 0 (the read-in file contained the mentioned 6 blocks). Thanks & regards, Jana On Wed, Nov 27, 2019 at 9:47 AM Simon Pfreundschuh mailto:simon.pfreundsc...@chalmers.se>> wrote: Hi Rita, The error you get is because your retrieval pressure grid extends higher up than your atmospheric grid. Indeed, this error should be caught but I have to see with the others how this can be done most effectively. Regarding the covariance matrix: The covariance matrix contains 6 blocks because for each retrieval quantity, i.e. for O3, freq. shift, and Polyfit, two blocks are added. It's two because the inverses of the blocks are pre-computed and included in the covariance matrix. In Python, you can access the blocks of the covariance matrix over the 'blocks' attribute which contains one element for each retrieval quantity (their inverses are stored in the 'inverse_blocks' attribute). The matrix representation of each block can then be retrieved using its matrix attribute: sx = ws.covmat_sx.value matrix = sx.blocks[0].matrix Hope this helps! Best, Simon From: arts_users.mi-boun...@mailman.rrz.uni-hamburg.de<mailto:arts_users.mi-boun...@mailman.rrz.uni-hamburg.de> mailto:arts_users.mi-boun...@mailman.rrz.uni-hamburg.de>> on behalf of Rita Kajtar mailto:rita.kaj...@gmail.com>> Sent: Tuesday, November 26, 2019 8:17:05 PM To: arts_users...@mailman.rrz.uni-hamburg.de<mailto:arts_users...@mailman.rrz.uni-hamburg.de> Subject: [arts-users] atm.scenario & cov-mat issues Hello! I have a few questions, the first ones related to the issues I encounter when I try to use a subarctic atmospheric scenario with user-defined pressure and retrieval pressure grids (*), and the next questions related to what precisely does the covariance matrix that my ARTS file produces contain and how can one actually extract certain blocks out of it once one knows what it contains; the documentation on covmat_sx does not provide too much information on this (**) (*) ### - I am running an adapted version on TestOEM.arts and while the pressure grid and the pressure retrieval grid I set work well with the tropical atm. background that TestOEM.arts comes with, I get the following error when I try to use a subarctic-winter scenario: Run-time error in method: AtmFieldsCalc There is a problem with the grids for the following interpolation: Raw field to p_grid The minimum of the new grid must be inside the original grid. (We allow a bit of extrapolation, but not so much). Minimum of original grid: -2.3969 (0.091) Minimum allowed value for new grid: -2.7956 (0.0610782) Actual minimum of new grid: -2.81347 (0.0599964) - If I then replace the pressure grid with a 'shorter' one, so that I make sure I fall within th
Re: [arts-users] atm.scenario & cov-mat issues
Hi Simon, hi all, the (p_grid) error is irrelevant (it is caught, it got fixed). The ASSERTION is the issue. It occurs once TestOEM.arts' default atmosphere (tropical standard atmo from AFGL/Fascode) is replaced with the subarctic winter AFGL/Fascode atmosphere (and the p_grid setting suitably adapted). Regarding the covariance matrix... First, I think it should occur in the documentation that covmat_sx contains the blocks and their inverse (does that also mean the inverse blocks together form the complete inverse Sx? Sorry, my instantaneous vector math knowledge is not good enough to judge that myself :( ). Did we just miss tzhat piece of information or is it not in the docs (yet)? Second, Rita reads covmat_sx in from xml file using typhon.arts.xml.load. Would that be the same as the ws.covmat_sx in your example? I had the feeling something went wrong there, too. typhon did not complain, but len(covmat_sx.blocks) was 0 (the read-in file contained the mentioned 6 blocks). Thanks & regards, Jana On Wed, Nov 27, 2019 at 9:47 AM Simon Pfreundschuh < simon.pfreundsc...@chalmers.se> wrote: > Hi Rita, > > > The error you get is because your retrieval pressure grid extends higher > up than > > your atmospheric grid. Indeed, this error should be caught but I have to > see with > > the others how this can be done most effectively. > > > Regarding the covariance matrix: The covariance matrix contains 6 blocks > because > > for each retrieval quantity, i.e. for O3, freq. shift, and Polyfit, two > blocks are added. > > It's two because the inverses of the blocks are pre-computed and included > in the > > covariance matrix. > > > In Python, you can access the blocks of the covariance matrix over the > 'blocks' > > attribute which contains one element for each retrieval quantity (their > inverses > > are stored in the 'inverse_blocks' attribute). The matrix representation > of each block > > can then be retrieved using its matrix attribute: > > > sx = ws.covmat_sx.value > matrix = sx.blocks[0].matrix > > Hope this helps! > > > Best, > > > Simon > > > -- > *From:* arts_users.mi-boun...@mailman.rrz.uni-hamburg.de < > arts_users.mi-boun...@mailman.rrz.uni-hamburg.de> on behalf of Rita > Kajtar > *Sent:* Tuesday, November 26, 2019 8:17:05 PM > *To:* arts_users...@mailman.rrz.uni-hamburg.de > *Subject:* [arts-users] atm.scenario & cov-mat issues > > Hello! > > I have a few questions, the first ones related to the issues I encounter > when I try to use a subarctic atmospheric scenario with user-defined > pressure and retrieval pressure grids (*), and the next questions related > to what precisely does the covariance matrix that my ARTS file produces > contain and how can one actually extract certain blocks out of it once one > knows what it contains; the documentation on covmat_sx does not provide too > much information on this (**) > > (*) > ### > - I am running an adapted version on TestOEM.arts and while the pressure > grid and the pressure retrieval grid I set work well with the tropical atm. > background that TestOEM.arts comes with, I get the following error when I > try to use a subarctic-winter scenario: > > Run-time error in method: AtmFieldsCalc > There is a problem with the grids for the following interpolation: > Raw field to p_grid > The minimum of the new grid must be inside the original grid. > (We allow a bit of extrapolation, but not so much). > Minimum of original grid: -2.3969 (0.091) > Minimum allowed value for new grid: -2.7956 (0.0610782) > Actual minimum of new grid: -2.81347 (0.0599964) > > - If I then replace the pressure grid with a 'shorter' one, so that I make > sure I fall within the minimum allowed value of the grid, I hit an assertion: > > > - xaStandard > Assertion failed: (og_min <= tng), function gridpos, file > /Users/mypath/arts/src/interpolation.cc, line 345. > Abort trap: 6 > > (**) > ## > - Why for a simulation containing three retrieval species. i.e. O3, > freq.shift and polyfit (no winds considered this time), the retrieval > covariance matrix (Sx) contains six retrieval errors? and what are those > representing? The terminal output shows a 17 x 17 sized covariance matrix, > which corresponds to the 15 elem O3 array + 1 elem of freq-shift + 1 elem > of polyfit. > (covmat_sx attached) > > - I can import the generated covariance matrix in my Python code, but how > can I extract the different blocks out of it for further use, i.e > plotting? > > > With best regards, > Rita Kajtar > ___ > arts_users.mi mailing list > arts_users.mi@lists.uni-hamburg.de > https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi > -- Jana Mendrok, Ph.D. Deutscher Wetterdienst Offenbach am Main, Germany +49 (0)69 8062 3139
Re: [arts-users] atm.scenario & cov-mat issues
Hi Rita, The error you get is because your retrieval pressure grid extends higher up than your atmospheric grid. Indeed, this error should be caught but I have to see with the others how this can be done most effectively. Regarding the covariance matrix: The covariance matrix contains 6 blocks because for each retrieval quantity, i.e. for O3, freq. shift, and Polyfit, two blocks are added. It's two because the inverses of the blocks are pre-computed and included in the covariance matrix. In Python, you can access the blocks of the covariance matrix over the 'blocks' attribute which contains one element for each retrieval quantity (their inverses are stored in the 'inverse_blocks' attribute). The matrix representation of each block can then be retrieved using its matrix attribute: sx = ws.covmat_sx.value matrix = sx.blocks[0].matrix Hope this helps! Best, Simon From: arts_users.mi-boun...@mailman.rrz.uni-hamburg.de on behalf of Rita Kajtar Sent: Tuesday, November 26, 2019 8:17:05 PM To: arts_users...@mailman.rrz.uni-hamburg.de Subject: [arts-users] atm.scenario & cov-mat issues Hello! I have a few questions, the first ones related to the issues I encounter when I try to use a subarctic atmospheric scenario with user-defined pressure and retrieval pressure grids (*), and the next questions related to what precisely does the covariance matrix that my ARTS file produces contain and how can one actually extract certain blocks out of it once one knows what it contains; the documentation on covmat_sx does not provide too much information on this (**) (*) ### - I am running an adapted version on TestOEM.arts and while the pressure grid and the pressure retrieval grid I set work well with the tropical atm. background that TestOEM.arts comes with, I get the following error when I try to use a subarctic-winter scenario: Run-time error in method: AtmFieldsCalc There is a problem with the grids for the following interpolation: Raw field to p_grid The minimum of the new grid must be inside the original grid. (We allow a bit of extrapolation, but not so much). Minimum of original grid: -2.3969 (0.091) Minimum allowed value for new grid: -2.7956 (0.0610782) Actual minimum of new grid: -2.81347 (0.0599964) - If I then replace the pressure grid with a 'shorter' one, so that I make sure I fall within the minimum allowed value of the grid, I hit an assertion: - xaStandard Assertion failed: (og_min <= tng), function gridpos, file /Users/mypath/arts/src/interpolation.cc, line 345. Abort trap: 6 (**) ## - Why for a simulation containing three retrieval species. i.e. O3, freq.shift and polyfit (no winds considered this time), the retrieval covariance matrix (Sx) contains six retrieval errors? and what are those representing? The terminal output shows a 17 x 17 sized covariance matrix, which corresponds to the 15 elem O3 array + 1 elem of freq-shift + 1 elem of polyfit. (covmat_sx attached) - I can import the generated covariance matrix in my Python code, but how can I extract the different blocks out of it for further use, i.e plotting? With best regards, Rita Kajtar ___ arts_users.mi mailing list arts_users.mi@lists.uni-hamburg.de https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi
