Dear Byungsuk Lee,
I ran your controlfile several times but I could not reproduce the errors you report. For me the iteration always converges and I don't get any errors. Also the result is always the same, just as expected. Is this really exactly the code that your are running when you observe the errors? Best, Simon ________________________________ From: Richard Larsson <ric.lars...@gmail.com> Sent: Monday, September 17, 2018 11:21:57 AM To: 이병석 Cc: Simon Pfreundschuh; arts_users...@mailman.rrz.uni-hamburg.de Subject: Re: [arts-users] ARTS OEM Hi, Just as a warning, suppressing the bad partition function error is not good practice since it is known that they are, you know, bad. Standard tests in ARTS fail by several tenths of Kelvin because the partition functions in ARTS are that bad. Just run with the provided tips xml-files instead, because if your temperatures are lower than 70 K or higher than 3000 K you probably have another problem. With hope, //Richard 2018-09-17 10:31 GMT+02:00 이병석 <cb...@aracnt.com<mailto:cb...@aracnt.com>>: Hello Simon Pfreundschuh, Thank you for your reply. I have employed both of the suggestions and still get the irregular errors. From the same controlfile modified in the previous email, I have included and changed to the following lines: OEM(method="lm", max_iter=15, display_progress=1, lm_ga_settings=[100,5,2,1000,1,99]) and AgendaSet( inversion_iterate_agenda ){ xClip(ijq = 0, limit_low = 150.0, limit_high = 350) Ignore(inversion_iteration_counter) # Map x to ARTS' variables x2artsStandard # To be safe, rerun checks dealing with the atmosphere atmfields_checkedCalc atmgeom_checkedCalc # Calculate yf and Jacobian matching x. yCalc( y=yf ) # Add baseline term VectorAddVector( yf, yf, y_baseline ) # This method takes cares of some "fixes" that are needed to get the Jacobian # right for iterative solutions. No need to call this WSM for linear inversions. jacobianAdjustAndTransform } I included both of the suggestions, and from there increased the maximum number of iterations to 15 and set the temperature limits to be between 150 K and 350 K. All other lines remain the same, including the input absorption line and atmospheric fields data. Now I irregularly get the errors during the OEM computation as before, of t_field having NaN values or of bad partition function warning (which happens for any t_field value outside between 150 K and 300 K). While the bad partition function warning could be suppressed in the code, it seems that the same problem as before persists. Could you please look into this issue a little more? I am not sure how to tackle the problem. Thank you for your service. Regards, Byungsuk Lee ARA Consulting & Technology 30 Songdomirae-ro D-1510, Yeonsu-gu, Incheon 21990, South Korea -----Original Message----- From: "Simon Pfreundschuh"<simon.pfreundsc...@chalmers.se<mailto:simon.pfreundsc...@chalmers.se>> To: "이병석"<cb...@aracnt.com<mailto:cb...@aracnt.com>>; "arts_users...@mailman.rrz.uni-hamburg.de<mailto:arts_users...@mailman.rrz.uni-hamburg.de>"<arts_users...@mailman.rrz.uni-hamburg.de<mailto:arts_users...@mailman.rrz.uni-hamburg.de>>; Cc: Sent: 2018-09-14 (금) 15:13:55 Subject: Re: [arts-users] ARTS OEM Hi Byungsuk Lee, It is a bit surprising that these errors occur irregularly, since there is no randomness in the test setup. However, if you are running retrievals with external observation data it may well happen that the iteration ends up in a state that produces invalid values. For your temperature retrieval I would recommend one or both of the following solutions: 1. Use the Levenberg-Marquardt (LM) iteration scheme: Using the LM scheme with a high start value for the gamma parameter can help avoid reaching invalid values when the forward model is non-linear: OEM(method="lm", max_iter=5, display_progress=1, lm_ga_settings=[100,5,2,1000,1,99]) 2. Clip the x-vector: Using the xClip WSM inside your inversion iterate agenda you can force the x vector to contain only temperatures within a certain range. If you use this approach you should check that none of the retrieved temperatures in your final state are clipped otherwise your results may not be valid. AgendaSet( inversion_iterate_agenda ){ xClip(ijq = 0, limit_low = 100.0) Ignore(inversion_iteration_counter) # Map x to ARTS' variables x2artsStandard # To be safe, rerun checks dealing with the atmosphere atmfields_checkedCalc atmgeom_checkedCalc # Calculate yf and Jacobian matching x. yCalc( y=yf ) # Add baseline term VectorAddVector( yf, yf, y_baseline ) # This method takes cares of some "fixes" that are needed to get the Jacobian # right for iterative solutions. No need to call this WSM for linear inversions. jacobianAdjustAndTransform } Regards, Simon ________________________________ From: arts_users.mi-boun...@mailman.rrz.uni-hamburg.de<mailto:arts_users.mi-boun...@mailman.rrz.uni-hamburg.de> <arts_users.mi-boun...@mailman.rrz.uni-hamburg.de<mailto:arts_users.mi-boun...@mailman.rrz.uni-hamburg.de>> on behalf of 이병석 <cb...@aracnt.com<mailto:cb...@aracnt.com>> Sent: Friday, September 14, 2018 2:31:25 AM To: arts_users...@mailman.rrz.uni-hamburg.de<mailto:arts_users...@mailman.rrz.uni-hamburg.de> Subject: [arts-users] ARTS OEM Dear ARTS users, I am writing to address a question with the use of OEM in the ARTS development version. From the "TestOEM.arts" in "ARTS/controlfiles/artscomponents/oem", this time I changed the codes as follows. 1) From (line 115): retrievalAddAbsSpecies( species = "O3", unit = "vmr", g1 = p_ret_grid, g2 = lat_grid, g3 = lon_grid ) To: retrievalAddTemperature ( g1 = p_ret_grid, g2 = lat_grid, g3 = lon_grid, hse = "off" ) 2) From (line 125): VectorSetConstant(vars, nelem, 1e-12) DiagonalMatrix(sparse_block, vars) covmat_sxAddBlock(block = sparse_block) To: VectorSetConstant(vars, nelem, 0.1) DiagonalMatrix(sparse_block, vars) covmat_sxAddBlock(block = sparse_block) 3) From (line 37): NumericSet( f_start, 110.436e9 ) NumericSet( f_end, 111.236e9 ) IndexSet( nf, 801 ) IndexSet( np, 81 ) To: NumericSet( f_start, 110.436e9 ) NumericSet( f_end, 111.236e9 ) IndexSet( nf, 18 ) IndexSet( np, 18 ) 4) From (line 42): VectorNLinSpace( f_grid, nf, f_start, f_end ) VectorNLogSpace( p_grid, 361, 500e2, 0.1 ) VectorNLogSpace( p_ret_grid, np, 500e2, 0.1 ) To: VectorNLinSpace( f_grid, nf, f_start, f_end ) VectorNLogSpace( p_grid, 18, 500e2, 0.1 ) VectorNLogSpace( p_ret_grid, np, 500e2, 0.1 ) Overall, I changed the ozone VMR retrieval to temperature retrieval and reduced the number of frequencies and pressure layers. When I run the controlfile multiple times, I irregularly get errors. The error messages are Run-time error in oem computation: Forward Model Evaluation Error Run-time error in agenda: inversion_iterate_agenda Run-time error in method: atmfields_checkedCalc The variable *t_field* contains one or several NaNs. This is not allowed! or All temperatures in *t_field* must be > 0. or seldom the bad partition function warning. Here are my questions: 1) Is this an anticipated error, simply coming from failing to find a converging solution? Is it expected that the OEM sometimes throws an error as above? 2) Are there mechanisms (e.g. workspace methods) within the ARTS that can prevent the t_field or vmr_field from becoming either negative or NaN during the OEM iterations? Thank you for your assistance. Regards, Byungsuk Lee ARA Consulting & Technology 30 Songdomirae-ro D-1510, Yeonsu-gu, Incheon 21990, South Korea Email: cb...@aracnt.com<mailto:cb...@aracnt.com> [https://kr1-mail.worksmobile.com/readReceipt/notify/?img=SwKwKAgmKAumaxUwKotwKAushzivFourF6UwpoglKA34MxIvMr3CK434axF4pzKZFoCgMXpXp6UmKob5%2Br9C16m5W4C5bX0q%2BzkR74FTWx%2FsMBidb9e9brkZbZlR1EIRM60nW6J57NiZDSl9W4vR1zJRMNkZpZlvpt%3D%3D.gif] _______________________________________________ arts_users.mi mailing list arts_users.mi@lists.uni-hamburg.de<mailto:arts_users.mi@lists.uni-hamburg.de> https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi
_______________________________________________ arts_users.mi mailing list arts_users.mi@lists.uni-hamburg.de https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi
