Re: [arts-users] ARTS OEM multiple inversions
Hi Patrick, Thanks for your quick answer. Simon's answer did clarify it and I believe his solution is quite close to what you suggest and what is done in Qpack. I'll try it out. Thanks, Eric De : Simon Pfreundschuh Envoyé : vendredi 4 juin 2021 14:55:24 À : Sauvageat, Eric (IAP); arts_users...@mailman.rrz.uni-hamburg.de Objet : Re: ARTS OEM multiple inversions Sorry, though the response would automatically go to the list. For completeness, here's my response from earlier. Just to clarify and confirm: > same sensors, same atmosphere, etc.. just different measurements vectors using > OEM from ARTS directly ? This is what ARTS currently does: It takes multiple observations and solves for a single atmospheric state. If I understand you correctly what you want to do is same sensors but different atmospheres (although they may be the same a priori). This kind of parallelism is not supported within ARTS so you'll have to do it sequentially. It's straight forward when you use the Python interface: After you have set up your calculation, you just call the OEM WSM multiple times in a loop. What you need to pay attention to is to reset the WSVs x, yf and jacobian before calling OEM a second time because those are both in- and outputs of the method. I not sure, however, how much you will gain from this computationally since this depends on your retrieval setup. Let me know if anything remains unclear. Kind regards, Simon From: arts_users.mi-boun...@mailman.rrz.uni-hamburg.de on behalf of eric.sauvag...@iap.unibe.ch Sent: Friday, June 4, 2021 12:03:12 PM To: arts_users...@mailman.rrz.uni-hamburg.de Subject: [arts-users] ARTS OEM multiple inversions Dear ARTS community, I am performing strato-mesospheric ozone retrievals with the OEM implemented directly in ARTS (using the python interface). As my sensor is not changing much with time, I am wondering if there is a way (and if this is worth in terms of computation speed) to reuse the sensor_response matrix (FFT channel response, unbinned) for different measurements which I think was quite straightforward with Qpack. I was trying to write multiple calibrated spectra within the y vector as well as multiple meas. error cov. (Se and inv) blocks before running the OEM which I thought might be the way to go but it seems to treat them as multiple measurements of the same ozone profiles instead. Could somebody confirmed if this is expected and if yes, is there another way to do this kind of parallel OEM retrievals: same sensors, same atmosphere, etc.. just different measurements vectors using OEM from ARTS directly ? Thanks for your help, Eric ___ arts_users.mi mailing list arts_users.mi@lists.uni-hamburg.de https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi
Re: [arts-users] ARTS OEM multiple inversions
Sorry, though the response would automatically go to the list. For completeness, here's my response from earlier. Just to clarify and confirm: > same sensors, same atmosphere, etc.. just different measurements vectors using > OEM from ARTS directly ? This is what ARTS currently does: It takes multiple observations and solves for a single atmospheric state. If I understand you correctly what you want to do is same sensors but different atmospheres (although they may be the same a priori). This kind of parallelism is not supported within ARTS so you'll have to do it sequentially. It's straight forward when you use the Python interface: After you have set up your calculation, you just call the OEM WSM multiple times in a loop. What you need to pay attention to is to reset the WSVs x, yf and jacobian before calling OEM a second time because those are both in- and outputs of the method. I not sure, however, how much you will gain from this computationally since this depends on your retrieval setup. Let me know if anything remains unclear. Kind regards, Simon From: arts_users.mi-boun...@mailman.rrz.uni-hamburg.de on behalf of eric.sauvag...@iap.unibe.ch Sent: Friday, June 4, 2021 12:03:12 PM To: arts_users...@mailman.rrz.uni-hamburg.de Subject: [arts-users] ARTS OEM multiple inversions Dear ARTS community, I am performing strato-mesospheric ozone retrievals with the OEM implemented directly in ARTS (using the python interface). As my sensor is not changing much with time, I am wondering if there is a way (and if this is worth in terms of computation speed) to reuse the sensor_response matrix (FFT channel response, unbinned) for different measurements which I think was quite straightforward with Qpack. I was trying to write multiple calibrated spectra within the y vector as well as multiple meas. error cov. (Se and inv) blocks before running the OEM which I thought might be the way to go but it seems to treat them as multiple measurements of the same ozone profiles instead. Could somebody confirmed if this is expected and if yes, is there another way to do this kind of parallel OEM retrievals: same sensors, same atmosphere, etc.. just different measurements vectors using OEM from ARTS directly ? Thanks for your help, Eric ___ arts_users.mi mailing list arts_users.mi@lists.uni-hamburg.de https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi
Re: [arts-users] ARTS OEM multiple inversions
Eric, I am performing strato-mesospheric ozone retrievals with the OEM implemented directly in ARTS (using the python interface). As my sensor is not changing much with time, I am wondering if there is a way (and if this is worth in terms of computation speed) to reuse the sensor_response matrix (FFT channel response, unbinned) for different measurements which I think was quite straightforward with Qpack. In Qpack some stuff is made behind the scene. Yes, it can be avoided to recalculate the sensor response. But you need save a number of variables. At least these ones: sensor_response sensor_response_f sensor_response_pol sensor_response_dlos, sensor_response_f_grid, sensor_response_pol_grid, sensor_response_dlos_grid Maybe also good to save mblock_dlos_grid stokes_dim And then you just need to import these for a new calculation (or just keep them in memory). I was trying to write multiple calibrated spectra within the y vector as well as multiple meas. error cov. (Se and inv) blocks before running the OEM which I thought might be the way to go but it seems to treat them as multiple measurements of the same ozone profiles instead. Could somebody confirmed if this is expected and if yes, is there another way to do this kind of parallel OEM retrievals: same sensors, same atmosphere, etc.. just different measurements vectors using OEM from ARTS directly ? There is no clear cut way inside ARTS to do batch OEM inversions. I am not using Python myself, but is this part not easy to setup on the python side? Bye, Patrick ___ arts_users.mi mailing list arts_users.mi@lists.uni-hamburg.de https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi
[arts-users] ARTS OEM multiple inversions
Dear ARTS community, I am performing strato-mesospheric ozone retrievals with the OEM implemented directly in ARTS (using the python interface). As my sensor is not changing much with time, I am wondering if there is a way (and if this is worth in terms of computation speed) to reuse the sensor_response matrix (FFT channel response, unbinned) for different measurements which I think was quite straightforward with Qpack. I was trying to write multiple calibrated spectra within the y vector as well as multiple meas. error cov. (Se and inv) blocks before running the OEM which I thought might be the way to go but it seems to treat them as multiple measurements of the same ozone profiles instead. Could somebody confirmed if this is expected and if yes, is there another way to do this kind of parallel OEM retrievals: same sensors, same atmosphere, etc.. just different measurements vectors using OEM from ARTS directly ? Thanks for your help, Eric ___ arts_users.mi mailing list arts_users.mi@lists.uni-hamburg.de https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi
Re: [arts-users] ARTS OEM
Hi Byungsuk, > On 17 Oct 2018, at 10:54, Byungsuk Lee wrote: > > Hello Oliver, > > Thank you for your reply. > > I have tried with AppleClang without OpenMP, and the installation was > successfully completed. Thank you. > > With that, I tried my OEM code, but the irregular t_field errors (discussed > previously in emails with ARTS users and Simon Pfreundschuh) still occurred. > > I have tried with Homebrew Clang or GCC compilers and without the fortran and > netCDF supports as well, and the errors still persisted. > > I am starting to question that the externally installed compilers or the > packages might not be the problem. Maybe it has something more to do with > some fundamental or built-in feature of Mac that is different from Ubuntu. > What it may actually be, I could not guess. Yes, looking at the "consistency" of the results, I totally agree with you. > If anyone has tried the development version's OEM on Mac (or Macbook), with > the test file "ARTS/controlfiles/artscomponents/oem/TestOEM.arts", with the > retrieval quantity switched from O3 vmr to temperature, with the number of > frequencies and the number of pressure layers reduced to about 20 (f_grid, > p_gird, and p_ret_grid), and was successful without irregularly occurring > errors for t_field during OEM, please let me know. I'm developing on Mac myself and can take a closer look at it. But can you please mail me the controlfile which exhibits this behaviour? I want to be 100% sure I'm running the same as you and don't want to dig through the whole mail thread piecing together the bits of information about the exact setup. :-) Cheers, Oliver ___ arts_users.mi mailing list arts_users.mi@lists.uni-hamburg.de https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi
Re: [arts-users] ARTS OEM
Hello Oliver, Thank you for your reply. I have tried with AppleClang without OpenMP, and the installation was successfully completed. Thank you. With that, I tried my OEM code, but the irregular t_field errors (discussed previously in emails with ARTS users and Simon Pfreundschuh) still occurred. I have tried with Homebrew Clang or GCC compilers and without the fortran and netCDF supports as well, and the errors still persisted. I am starting to question that the externally installed compilers or the packages might not be the problem. Maybe it has something more to do with some fundamental or built-in feature of Mac that is different from Ubuntu. What it may actually be, I could not guess. If anyone has tried the development version's OEM on Mac (or Macbook), with the test file "ARTS/controlfiles/artscomponents/oem/TestOEM.arts", with the retrieval quantity switched from O3 vmr to temperature, with the number of frequencies and the number of pressure layers reduced to about 20 (f_grid, p_gird, and p_ret_grid), and was successful without irregularly occurring errors for t_field during OEM, please let me know. Regards, Byungsuk Lee On Mon, Oct 15, 2018 at 3:45 PM Oliver Lemke wrote: > Hi Byungsuk, > > > On 8 Oct 2018, at 07:26, Byungsuk Lee wrote: > > > > Hello Oliver, > > > > Thank you for your reply. > > > > 1) I don't think there is any Intel-related library in my PATH. Could > any of them below be an Intel library? Below are some of the lines from > CMakeCache.txt after cmake command using Apple Clang v10 and GNU gfortran > v8.2.0: > > [snip] > > //Details about finding OpenMP > > FIND_PACKAGE_MESSAGE_DETAILS_OpenMP:INTERNAL=[TRUE][TRUE][TRUE][c > ][v3.1()] > > //Details about finding OpenMP_C > > FIND_PACKAGE_MESSAGE_DETAILS_OpenMP_C:INTERNAL=[-Xclang > -fopenmp][/usr/local/lib/libomp.dylib][v3.1()] > > //Details about finding OpenMP_CXX > > FIND_PACKAGE_MESSAGE_DETAILS_OpenMP_CXX:INTERNAL=[-Xclang > -fopenmp][/usr/local/lib/libomp.dylib][v3.1()] > > //Details about finding OpenMP_Fortran > > > FIND_PACKAGE_MESSAGE_DETAILS_OpenMP_Fortran:INTERNAL=[-fopenmp][/usr/local/Cellar/gcc/8.2.0/lib/gcc/8/libgomp.dylib][v4.5()] > > [snip] > > Doesn't look like any Intel related libraries are present. I guess > /usr/local/lib/libomp.dylib comes from brew? You could try to do a > compilation without the Fortran, NetCDF and OpenMP features. I don't think > you need any of the Fortran modules for your calculation. You can disable > OpenMP with 'cmake -DNO_OPENMP=1 ..'. Just to see if that fixes the > compilation errors. > > > 2) After installing ARTS with Homebrew llvm clang/clang++: > > export CC=/usr/local/opt/llvm/bin/clang > > export CXX=/usr/local/opt/llvm/bin/clang++ > > export FC=/usr/local/opt/gcc/bin/gfortran > > cmake -DENABLE_C_API=1 > -DARTS_XML_DATA_PATH=/Users/BLee/Desktop/ARTS/arts-xml-data-dev > -DENABLE_FORTRAN=1 -DENABLE_NETCDF=1 .. > > make -j4 > > > > I get the following from typing "otool -L src/arts": > > src/arts: > > /usr/lib/libz.1.dylib (compatibility version 1.0.0, current > version 1.2.11) > > /usr/local/opt/netcdf/lib/libnetcdf.13.dylib (compatibility > version 13.0.0, current version 13.0.0) > > > /System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate > (compatibility version 1.0.0, current version 4.0.0) > > /usr/local/opt/gcc/lib/gcc/8/libgfortran.5.dylib (compatibility > version 6.0.0, current version 6.0.0) > > /usr/local/lib/gcc/8/libgcc_s.1.dylib (compatibility version > 1.0.0, current version 1.0.0) > > /usr/local/opt/gcc/lib/gcc/8/libquadmath.0.dylib (compatibility > version 1.0.0, current version 1.0.0) > > /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current > version 1252.50.4) > > /usr/local/opt/libomp/lib/libomp.dylib (compatibility version > 5.0.0, current version 5.0.0) > > /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current > version 400.9.0) > > > > I am guessing that the Accelerate framework is properly included? > > Yes, that looks correct. > > Cheers, > Oliver > > > > Thank you, > > > > Byungsuk Lee > > > > > > > > On Thu, Oct 4, 2018 at 5:49 PM Oliver Lemke > wrote: > > Hi Byungsuk, > > > > > On 2 Oct 2018, at 11:39, Byungsuk Lee wrote: > > > > > > Dear Jonas and Oliver, > > > > > > Thank you for your replies. I wanted to give you a sort of status > report regarding the use of ARTS on my Mac. > > > > > > 1) I checked ARTS/build/CMakeCache.txt and confirmed that no conda > package was used, other than CONDA_PROG:FILEPATH=... which I think is > irrelevant. > > > > > > 2) I tried compiling with gcc/g++ instead, both from Homebrew and from > manually installing tar.gz files from the gnu website for a few different > versions, but the same irregular invalid t_field errors in OEM happened. > Plus, at the cmake stage, they constantly produced a failure message for > "Wno-return-type-c-linkage" as follows, about which I am curious why. This > doesn't happen with
Re: [arts-users] ARTS OEM
Hi Byungsuk, > On 8 Oct 2018, at 07:26, Byungsuk Lee wrote: > > Hello Oliver, > > Thank you for your reply. > > 1) I don't think there is any Intel-related library in my PATH. Could any of > them below be an Intel library? Below are some of the lines from > CMakeCache.txt after cmake command using Apple Clang v10 and GNU gfortran > v8.2.0: > [snip] > //Details about finding OpenMP > FIND_PACKAGE_MESSAGE_DETAILS_OpenMP:INTERNAL=[TRUE][TRUE][TRUE][c ][v3.1()] > //Details about finding OpenMP_C > FIND_PACKAGE_MESSAGE_DETAILS_OpenMP_C:INTERNAL=[-Xclang > -fopenmp][/usr/local/lib/libomp.dylib][v3.1()] > //Details about finding OpenMP_CXX > FIND_PACKAGE_MESSAGE_DETAILS_OpenMP_CXX:INTERNAL=[-Xclang > -fopenmp][/usr/local/lib/libomp.dylib][v3.1()] > //Details about finding OpenMP_Fortran > FIND_PACKAGE_MESSAGE_DETAILS_OpenMP_Fortran:INTERNAL=[-fopenmp][/usr/local/Cellar/gcc/8.2.0/lib/gcc/8/libgomp.dylib][v4.5()] > [snip] Doesn't look like any Intel related libraries are present. I guess /usr/local/lib/libomp.dylib comes from brew? You could try to do a compilation without the Fortran, NetCDF and OpenMP features. I don't think you need any of the Fortran modules for your calculation. You can disable OpenMP with 'cmake -DNO_OPENMP=1 ..'. Just to see if that fixes the compilation errors. > 2) After installing ARTS with Homebrew llvm clang/clang++: > export CC=/usr/local/opt/llvm/bin/clang > export CXX=/usr/local/opt/llvm/bin/clang++ > export FC=/usr/local/opt/gcc/bin/gfortran > cmake -DENABLE_C_API=1 > -DARTS_XML_DATA_PATH=/Users/BLee/Desktop/ARTS/arts-xml-data-dev > -DENABLE_FORTRAN=1 -DENABLE_NETCDF=1 .. > make -j4 > > I get the following from typing "otool -L src/arts": > src/arts: > /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version > 1.2.11) > /usr/local/opt/netcdf/lib/libnetcdf.13.dylib (compatibility version > 13.0.0, current version 13.0.0) > /System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate > (compatibility version 1.0.0, current version 4.0.0) > /usr/local/opt/gcc/lib/gcc/8/libgfortran.5.dylib (compatibility version > 6.0.0, current version 6.0.0) > /usr/local/lib/gcc/8/libgcc_s.1.dylib (compatibility version 1.0.0, > current version 1.0.0) > /usr/local/opt/gcc/lib/gcc/8/libquadmath.0.dylib (compatibility version > 1.0.0, current version 1.0.0) > /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current > version 1252.50.4) > /usr/local/opt/libomp/lib/libomp.dylib (compatibility version 5.0.0, > current version 5.0.0) > /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version > 400.9.0) > > I am guessing that the Accelerate framework is properly included? Yes, that looks correct. Cheers, Oliver > Thank you, > > Byungsuk Lee > > > > On Thu, Oct 4, 2018 at 5:49 PM Oliver Lemke > wrote: > Hi Byungsuk, > > > On 2 Oct 2018, at 11:39, Byungsuk Lee wrote: > > > > Dear Jonas and Oliver, > > > > Thank you for your replies. I wanted to give you a sort of status report > > regarding the use of ARTS on my Mac. > > > > 1) I checked ARTS/build/CMakeCache.txt and confirmed that no conda package > > was used, other than CONDA_PROG:FILEPATH=... which I think is irrelevant. > > > > 2) I tried compiling with gcc/g++ instead, both from Homebrew and from > > manually installing tar.gz files from the gnu website for a few different > > versions, but the same irregular invalid t_field errors in OEM happened. > > Plus, at the cmake stage, they constantly produced a failure message for > > "Wno-return-type-c-linkage" as follows, about which I am curious why. This > > doesn't happen with clang/clang++. > > > > ... > > -- Performing Test CCFLAG_Wno-sign-conversion > > -- Performing Test CCFLAG_Wno-sign-conversion - Success > > -- Performing Test CXXFLAG_Wno-sign-conversion > > -- Performing Test CXXFLAG_Wno-sign-conversion - Success > > -- Performing Test CCFLAG_Wno-unknown-pragmas > > -- Performing Test CCFLAG_Wno-unknown-pragmas - Success > > -- Performing Test CXXFLAG_Wno-unknown-pragmas > > -- Performing Test CXXFLAG_Wno-unknown-pragmas - Success > > -- Performing Test CCFLAG_Wno-return-type-c-linkage > > -- Performing Test CCFLAG_Wno-return-type-c-linkage - Failed > > -- Performing Test CXXFLAG_Wno-return-type-c-linkage > > -- Performing Test CXXFLAG_Wno-return-type-c-linkage - Failed > > -- Performing Test CCFLAG_Wno-strict-overflow > > -- Performing Test CCFLAG_Wno-strict-overflow - Success > > -- Performing Test CXXFLAG_Wno-strict-overflow > > -- Performing Test CXXFLAG_Wno-strict-overflow - Success > > ... > > Some compilers don't support the flag no-return-type-c-linkage. That is > totally fine and the reason that we test its existence with cmake before > enabling it. > > > 3) Instead of the clang/clang++ from Homebrew llvm with which I initially > > installed ARTS, I thought I could instead try using Mac's command line > > tools clang/clang++
Re: [arts-users] ARTS OEM
Hello Oliver, Thank you for your reply. 1) I don't think there is any Intel-related library in my PATH. Could any of them below be an Intel library? Below are some of the lines from CMakeCache.txt after cmake command using Apple Clang v10 and GNU gfortran v8.2.0: //Path to a library. LAPACK_Accelerate_LIBRARY:FILEPATH=/System/Library/Frameworks/Accelerate.framework //Path to a library. BLAS_Accelerate_LIBRARY:FILEPATH=/System/Library/Frameworks/Accelerate.framework //CXX compiler CMAKE_CXX_COMPILER:FILEPATH=/Library/Developer/CommandLineTools/usr/bin/clang++ //C compiler CMAKE_C_COMPILER:FILEPATH=/Library/Developer/CommandLineTools/usr/bin/clang //Path to the gomp library for OpenMP OpenMP_gomp_LIBRARY:FILEPATH=/usr/local/Cellar/gcc/8.2.0/lib/gcc/8/libgomp.dylib //Path to a library. OpenMP_libomp_LIBRARY:FILEPATH=/usr/local/lib/libomp.dylib //Details about finding Doxygen FIND_PACKAGE_MESSAGE_DETAILS_Doxygen:INTERNAL=[/usr/local/bin/doxygen][cfound components: doxygen dot ][v1.8.14()] //Details about finding FFTW FIND_PACKAGE_MESSAGE_DETAILS_FFTW:INTERNAL=[/usr/local/lib/libfftw3.a][/usr/local/include][v()] //Details about finding LATEX FIND_PACKAGE_MESSAGE_DETAILS_LATEX:INTERNAL=[/Library/TeX/texbin/latex][c ][v()] //Details about finding NetCDF FIND_PACKAGE_MESSAGE_DETAILS_NetCDF:INTERNAL=[/usr/local/lib/libnetcdf.dylib][/usr/local/include][v()] //Details about finding OpenMP FIND_PACKAGE_MESSAGE_DETAILS_OpenMP:INTERNAL=[TRUE][TRUE][TRUE][c ][v3.1()] //Details about finding OpenMP_C FIND_PACKAGE_MESSAGE_DETAILS_OpenMP_C:INTERNAL=[-Xclang -fopenmp][/usr/local/lib/libomp.dylib][v3.1()] //Details about finding OpenMP_CXX FIND_PACKAGE_MESSAGE_DETAILS_OpenMP_CXX:INTERNAL=[-Xclang -fopenmp][/usr/local/lib/libomp.dylib][v3.1()] //Details about finding OpenMP_Fortran FIND_PACKAGE_MESSAGE_DETAILS_OpenMP_Fortran:INTERNAL=[-fopenmp][/usr/local/Cellar/gcc/8.2.0/lib/gcc/8/libgomp.dylib][v4.5()] //Details about finding Threads FIND_PACKAGE_MESSAGE_DETAILS_Threads:INTERNAL=[TRUE][v()] //Details about finding ZLIB FIND_PACKAGE_MESSAGE_DETAILS_ZLIB:INTERNAL=[/usr/lib/libz.dylib][/usr/include][v1.2.11()] 2) After installing ARTS with Homebrew llvm clang/clang++: export CC=/usr/local/opt/llvm/bin/clang export CXX=/usr/local/opt/llvm/bin/clang++ export FC=/usr/local/opt/gcc/bin/gfortran cmake -DENABLE_C_API=1 -DARTS_XML_DATA_PATH=/Users/BLee/Desktop/ARTS/arts-xml-data-dev -DENABLE_FORTRAN=1 -DENABLE_NETCDF=1 .. make -j4 I get the following from typing "otool -L src/arts": src/arts: /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.11) /usr/local/opt/netcdf/lib/libnetcdf.13.dylib (compatibility version 13.0.0, current version 13.0.0) /System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate (compatibility version 1.0.0, current version 4.0.0) /usr/local/opt/gcc/lib/gcc/8/libgfortran.5.dylib (compatibility version 6.0.0, current version 6.0.0) /usr/local/lib/gcc/8/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0) /usr/local/opt/gcc/lib/gcc/8/libquadmath.0.dylib (compatibility version 1.0.0, current version 1.0.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.50.4) /usr/local/opt/libomp/lib/libomp.dylib (compatibility version 5.0.0, current version 5.0.0) /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 400.9.0) I am guessing that the Accelerate framework is properly included? Thank you, Byungsuk Lee On Thu, Oct 4, 2018 at 5:49 PM Oliver Lemke wrote: > Hi Byungsuk, > > > On 2 Oct 2018, at 11:39, Byungsuk Lee wrote: > > > > Dear Jonas and Oliver, > > > > Thank you for your replies. I wanted to give you a sort of status report > regarding the use of ARTS on my Mac. > > > > 1) I checked ARTS/build/CMakeCache.txt and confirmed that no conda > package was used, other than CONDA_PROG:FILEPATH=... which I think is > irrelevant. > > > > 2) I tried compiling with gcc/g++ instead, both from Homebrew and from > manually installing tar.gz files from the gnu website for a few different > versions, but the same irregular invalid t_field errors in OEM happened. > Plus, at the cmake stage, they constantly produced a failure message for > "Wno-return-type-c-linkage" as follows, about which I am curious why. This > doesn't happen with clang/clang++. > > > > ... > > -- Performing Test CCFLAG_Wno-sign-conversion > > -- Performing Test CCFLAG_Wno-sign-conversion - Success > > -- Performing Test CXXFLAG_Wno-sign-conversion > > -- Performing Test CXXFLAG_Wno-sign-conversion - Success > > -- Performing Test CCFLAG_Wno-unknown-pragmas > > -- Performing Test CCFLAG_Wno-unknown-pragmas - Success > > -- Performing Test CXXFLAG_Wno-unknown-pragmas > > -- Performing Test CXXFLAG_Wno-unknown-pragmas - Success > > -- Performing Test CCFLAG_Wno-return-type-c-linkage > > -- Performing Test CCFLAG_Wno-return-type-c-linkage - Failed > > -- Performing Test CXXFLAG_Wno-return-type-c-linkage > > -- Performing Test
Re: [arts-users] ARTS OEM
Hi Byungsuk, > On 2 Oct 2018, at 11:39, Byungsuk Lee wrote: > > Dear Jonas and Oliver, > > Thank you for your replies. I wanted to give you a sort of status report > regarding the use of ARTS on my Mac. > > 1) I checked ARTS/build/CMakeCache.txt and confirmed that no conda package > was used, other than CONDA_PROG:FILEPATH=... which I think is irrelevant. > > 2) I tried compiling with gcc/g++ instead, both from Homebrew and from > manually installing tar.gz files from the gnu website for a few different > versions, but the same irregular invalid t_field errors in OEM happened. > Plus, at the cmake stage, they constantly produced a failure message for > "Wno-return-type-c-linkage" as follows, about which I am curious why. This > doesn't happen with clang/clang++. > > ... > -- Performing Test CCFLAG_Wno-sign-conversion > -- Performing Test CCFLAG_Wno-sign-conversion - Success > -- Performing Test CXXFLAG_Wno-sign-conversion > -- Performing Test CXXFLAG_Wno-sign-conversion - Success > -- Performing Test CCFLAG_Wno-unknown-pragmas > -- Performing Test CCFLAG_Wno-unknown-pragmas - Success > -- Performing Test CXXFLAG_Wno-unknown-pragmas > -- Performing Test CXXFLAG_Wno-unknown-pragmas - Success > -- Performing Test CCFLAG_Wno-return-type-c-linkage > -- Performing Test CCFLAG_Wno-return-type-c-linkage - Failed > -- Performing Test CXXFLAG_Wno-return-type-c-linkage > -- Performing Test CXXFLAG_Wno-return-type-c-linkage - Failed > -- Performing Test CCFLAG_Wno-strict-overflow > -- Performing Test CCFLAG_Wno-strict-overflow - Success > -- Performing Test CXXFLAG_Wno-strict-overflow > -- Performing Test CXXFLAG_Wno-strict-overflow - Success > ... Some compilers don't support the flag no-return-type-c-linkage. That is totally fine and the reason that we test its existence with cmake before enabling it. > 3) Instead of the clang/clang++ from Homebrew llvm with which I initially > installed ARTS, I thought I could instead try using Mac's command line tools > clang/clang++ (AppleClang v10), but they failed at the make stage with the > following error which I am not sure how to tackle. > > ... > [ 43%] Linking CXX executable make_auto_workspace_h > Undefined symbols for architecture x86_64: > "___kmpc_critical", referenced from: > ArtsOut& operator<< std::__1::char_traits, std::__1::allocator > >(ArtsOut&, > std::__1::basic_string, > std::__1::allocator > const&) in arts.cc.o > ArtsOut& operator<<(ArtsOut&, char const (&) [35]) in > file.cc.o > ArtsOut& operator<<(ArtsOut&, char const (&) [75]) in > file.cc.o > ArtsOut& operator<<(ArtsOut&, char const (&) [2]) in file.cc.o > ArtsOut& operator<< >(ArtsOut&, > my_basic_string const&) in file.cc.o > ArtsOut& operator<<(ArtsOut&, char const (&) [3]) in file.cc.o > ArtsOut& operator<<(ArtsOut&, long const&) in file.cc.o > ... > "___kmpc_end_critical", referenced from: > ArtsOut& operator<< std::__1::char_traits, std::__1::allocator > >(ArtsOut&, > std::__1::basic_string, > std::__1::allocator > const&) in arts.cc.o > ArtsOut& operator<<(ArtsOut&, char const (&) [35]) in > file.cc.o > ArtsOut& operator<<(ArtsOut&, char const (&) [75]) in > file.cc.o > ArtsOut& operator<<(ArtsOut&, char const (&) [2]) in file.cc.o > ArtsOut& operator<< >(ArtsOut&, > my_basic_string const&) in file.cc.o > ArtsOut& operator<<(ArtsOut&, char const (&) [3]) in file.cc.o > ArtsOut& operator<<(ArtsOut&, long const&) in file.cc.o > ... > "___kmpc_global_thread_num", referenced from: > ArtsOut& operator<< std::__1::char_traits, std::__1::allocator > >(ArtsOut&, > std::__1::basic_string, > std::__1::allocator > const&) in arts.cc.o > ArtsOut& operator<<(ArtsOut&, char const (&) [35]) in > file.cc.o > ArtsOut& operator<<(ArtsOut&, char const (&) [75]) in > file.cc.o > ArtsOut& operator<<(ArtsOut&, char const (&) [2]) in file.cc.o > ArtsOut& operator<< >(ArtsOut&, > my_basic_string const&) in file.cc.o > ArtsOut& operator<<(ArtsOut&, char const (&) [3]) in file.cc.o > ArtsOut& operator<<(ArtsOut&, long const&) in file.cc.o > ... > ld: symbol(s) not found for architecture x86_64 > clang: error: linker command failed with exit code 1 (use -v to see > invocation) > make[2]: *** [src/make_auto_workspace_h] Error 1 > make[1]: *** [src/CMakeFiles/make_auto_workspace_h.dir/all] Error 2 > [ 43%] Building CXX object > src/CMakeFiles/test_gridded_fields.dir/test_gridded_fields.cc.o > [ 43%] Building CXX object src/CMakeFiles/test_legendre.dir/test_legendre.cc.o > [ 43%] Linking CXX executable test_legendre > [ 43%] Built target test_legendre > [ 43%] Linking CXX executable test_gridded_fields > [ 43%] Built target test_gridded_fields > make: *** [all] Error 2 > ... This is interesting. The __kmpc_* Symbols come from the Intel Math Kernel Library. These errors usually indicate that the Intel Threading library was enabled,
Re: [arts-users] ARTS OEM
_t - done >> -- Check size of float >> -- Check size of float - done >> -- Check size of double >> -- Check size of double - done >> -- Looking for unlink >> -- Looking for unlink - found >> -- Found ZLIB: /usr/lib/x86_64-linux-gnu/libz.so (found version "1.2.11") >> -- Looking for Fortran sgemm >> -- Looking for Fortran sgemm - found >> -- A library with BLAS API found. >> -- Looking for Fortran cheev >> -- Looking for Fortran cheev - found >> -- A library with LAPACK API found. >> -- Found NetCDF: /usr/lib/x86_64-linux-gnu/libnetcdf.so >> -- Found FFTW: /usr/lib/x86_64-linux-gnu/libfftw3.so >> -- Found OpenMP_C: -fopenmp=libomp (found version "3.1") >> -- Found OpenMP_CXX: -fopenmp=libomp (found version "3.1") >> -- Found OpenMP_Fortran: -fopenmp (found version "4.5") >> -- Found OpenMP: TRUE (found version "3.1") >> -- Performing Test CCFLAG_W >> -- Performing Test CCFLAG_W - Success >> -- Performing Test CXXFLAG_W >> -- Performing Test CXXFLAG_W - Success >> -- Performing Test CCFLAG_Wall >> -- Performing Test CCFLAG_Wall - Success >> -- Performing Test CXXFLAG_Wall >> -- Performing Test CXXFLAG_Wall - Success >> -- Performing Test CCFLAG_Wshadow >> -- Performing Test CCFLAG_Wshadow - Success >> -- Performing Test CXXFLAG_Wshadow >> -- Performing Test CXXFLAG_Wshadow - Success >> -- Performing Test CCFLAG_Wconversion >> -- Performing Test CCFLAG_Wconversion - Success >> -- Performing Test CXXFLAG_Wconversion >> -- Performing Test CXXFLAG_Wconversion - Success >> -- Performing Test CCFLAG_Wno-sign-conversion >> -- Performing Test CCFLAG_Wno-sign-conversion - Success >> -- Performing Test CXXFLAG_Wno-sign-conversion >> -- Performing Test CXXFLAG_Wno-sign-conversion - Success >> -- Performing Test CCFLAG_Wno-unknown-pragmas >> -- Performing Test CCFLAG_Wno-unknown-pragmas - Success >> -- Performing Test CXXFLAG_Wno-unknown-pragmas >> -- Performing Test CXXFLAG_Wno-unknown-pragmas - Success >> -- Performing Test CCFLAG_Wno-return-type-c-linkage >> -- Performing Test CCFLAG_Wno-return-type-c-linkage - Success >> -- Performing Test CXXFLAG_Wno-return-type-c-linkage >> -- Performing Test CXXFLAG_Wno-return-type-c-linkage - Success >> -- Performing Test CCFLAG_Wno-strict-overflow >> -- Performing Test CCFLAG_Wno-strict-overflow - Success >> -- Performing Test CXXFLAG_Wno-strict-overflow >> -- Performing Test CXXFLAG_Wno-strict-overflow - Success >> -- Found arts-xml-data in /path/to/ARTS/xml-data >> -- Disort enabled (use -DNO_DISORT=1 to disable) >> -- RT4 enabled (use -DNO_RT4=1 to disable) >> -- FASTEM enabled (use -DNO_FASTEM=1 to disable) >> -- Refice enabled (use -DNO_REFICE=1 to disable) >> -- Tmatrix enabled (use -DNO_TMATRIX=1 to disable) >> -- RELMAT enabled (use -DNO_RELMAT=1 to disable) >> -- HITRAN XSEC enabled (experimental) >> -- OEM enabled >> -- C API enabled >> -- Found Doxygen: /home/user/anaconda3/bin/doxygen (found version "1.8.14") >> found components: doxygen dot >> -- Found LATEX: /usr/bin/latex >> -- Configuring done >> -- Generating done >> -- Build files have been written to: /path/to/ARTS/build >> >> >> >> I guess I could just use my Ubuntu which does not produce the errors, but if >> you have time to look into this issue on Mac and pinpoint which difference >> may be the cause, that would be great. >> >> Thank you, >> >> Lee >> >> >> >> On Wed, Sep 19, 2018 at 8:09 AM Simon Pfreundschuh >> wrote: >> Dear Byungsuk Lee, >> >> >> I ran your controlfile several times but I could not reproduce the errors >> you report. >> >> For me the iteration always converges and I don't get any errors. Also the >> result >> >> is always the same, just as expected. Is this really exactly the code that >> your are >> >> running when you observe the errors? >> >> >> Best, >> >> >> Simon >> From: Richard Larsson >> Sent: Monday, September 17, 2018 11:21:57 AM >> To: 이병석 >> Cc: Simon Pfreundschuh; arts_users...@mailman.rrz.uni-hamburg.de >> Subject: Re: [arts-users] ARTS OEM >> >> Hi, >> >> Just as a warning, suppressing the bad partition function error is not good >> practice since it is known that they are, you know, bad. Standard tests in >> ARTS fail by several tenths of Kelvin because the partition functions in >> AR
Re: [arts-users] ARTS OEM
Dear Byungsuk Lee,
I ran your controlfile several times but I could not reproduce the errors you
report.
For me the iteration always converges and I don't get any errors. Also the
result
is always the same, just as expected. Is this really exactly the code that your
are
running when you observe the errors?
Best,
Simon
From: Richard Larsson
Sent: Monday, September 17, 2018 11:21:57 AM
To: 이병석
Cc: Simon Pfreundschuh; arts_users...@mailman.rrz.uni-hamburg.de
Subject: Re: [arts-users] ARTS OEM
Hi,
Just as a warning, suppressing the bad partition function error is not good
practice since it is known that they are, you know, bad. Standard tests in
ARTS fail by several tenths of Kelvin because the partition functions in ARTS
are that bad. Just run with the provided tips xml-files instead, because if
your temperatures are lower than 70 K or higher than 3000 K you probably have
another problem.
With hope,
//Richard
2018-09-17 10:31 GMT+02:00 이병석 mailto:cb...@aracnt.com>>:
Hello Simon Pfreundschuh,
Thank you for your reply.
I have employed both of the suggestions and still get the irregular errors.
From the same controlfile modified in the previous email, I have included and
changed to the following lines:
OEM(method="lm",
max_iter=15,
display_progress=1,
lm_ga_settings=[100,5,2,1000,1,99])
and
AgendaSet( inversion_iterate_agenda ){
xClip(ijq = 0, limit_low = 150.0, limit_high = 350)
Ignore(inversion_iteration_counter)
# Map x to ARTS' variables
x2artsStandard
# To be safe, rerun checks dealing with the atmosphere
atmfields_checkedCalc
atmgeom_checkedCalc
# Calculate yf and Jacobian matching x.
yCalc( y=yf )
# Add baseline term
VectorAddVector( yf, yf, y_baseline )
# This method takes cares of some "fixes" that are needed to get the Jacobian
# right for iterative solutions. No need to call this WSM for linear
inversions.
jacobianAdjustAndTransform
}
I included both of the suggestions, and from there increased the maximum number
of iterations to 15 and set the temperature limits to be between 150 K and 350
K. All other lines remain the same, including the input absorption line and
atmospheric fields data.
Now I irregularly get the errors during the OEM computation as before, of
t_field having NaN values or of bad partition function warning (which happens
for any t_field value outside between 150 K and 300 K). While the bad partition
function warning could be suppressed in the code, it seems that the same
problem as before persists.
Could you please look into this issue a little more? I am not sure how to
tackle the problem.
Thank you for your service.
Regards,
Byungsuk Lee
ARA Consulting & Technology
30 Songdomirae-ro D-1510, Yeonsu-gu, Incheon 21990, South Korea
-Original Message-
From: "Simon
Pfreundschuh"mailto:simon.pfreundsc...@chalmers.se>>
To: "이병석"mailto:cb...@aracnt.com>>;
"arts_users...@mailman.rrz.uni-hamburg.de<mailto:arts_users...@mailman.rrz.uni-hamburg.de>"mailto:arts_users...@mailman.rrz.uni-hamburg.de>>;
Cc:
Sent: 2018-09-14 (금) 15:13:55
Subject: Re: [arts-users] ARTS OEM
Hi Byungsuk Lee,
It is a bit surprising that these errors occur irregularly, since there is no
randomness
in the test setup. However, if you are running retrievals with external
observation data
it may well happen that the iteration ends up in a state that produces invalid
values.
For your temperature retrieval I would recommend one or both of the following
solutions:
1. Use the Levenberg-Marquardt (LM) iteration scheme: Using the LM scheme with
a high start
value for the gamma parameter can help avoid reaching invalid values when
the forward
model is non-linear:
OEM(method="lm",
max_iter=5,
display_progress=1,
lm_ga_settings=[100,5,2,1000,1,99])
2. Clip the x-vector: Using the xClip WSM inside your inversion iterate agenda
you can force the
x vector to contain only temperatures within a certain range. If you use
this approach you
should check that none of the retrieved temperatures in your final state are
clipped otherwise your
results may not be valid.
AgendaSet( inversion_iterate_agenda ){
xClip(ijq = 0, limit_low = 100.0)
Ignore(inversion_iteration_counter)
# Map x to ARTS' variables
x2artsStandard
# To be safe, rerun checks dealing with the atmosphere
atmfields_checkedCalc
atmgeom_checkedCalc
# Calculate yf and Jacobian matching x.
yCalc( y=yf )
# Add baseline term
VectorAddVector( yf, yf, y_baseline )
# This method takes cares of some "fixes" that are needed to get the
Jacobian
# right for iterative solutions. No need to call this WSM for linear
inversions.
jacobianAdjustAndTransform
}
Regards,
Simon
From:
Re: [arts-users] ARTS OEM
Hi Byungsuk,
We provide the TIPS as published by HITRAN via an XML file in:
arts-xml-data/spectroscopy/PartitionSums/TIPS/tips.xml. The README
contains some more information.
About the 300 K limit today, I do not know where it is from and the
differences between it and the TIPS data is not strongly defined. The
builtin partition_functions are historical, and tests show that it should
be mostly reliable up to 300 K but not much more than that.
If you need a comparison for the temperature ranges you are interested in,
you can run "nlte_fieldSetLteInternalPartitionFunction" and
"nlte_fieldSetLteExternalPartitionFunction"
after having set partition_functions to tips.xml. The NLTE field is a the
size of your atmosphere times the size of the energy levels you are
interested in.
With hope,
//Richard
2018-09-18 2:33 GMT+02:00 이병석 :
> Hello Richard,
>
> Thank you for your reply.
>
> May I ask which tips xml files you are referring to?
>
> Also a question: it seems that the bad partition function warning occurs
> for any input t_field temperature beyond the range between 150 K and 300 K.
> In reality the surface temperature of the Earth frequently exceeds 300 K
> (26.85 deg C). For input values such as 30 deg C or a little higher (e.g.
> 30-40 deg C, early 300's K), would the results still be bad if the warning
> is suppressed?
>
> Regards,
>
>
> Byungsuk Lee
>
> ARA Consulting & Technology
>
> 30 Songdomirae-ro D-1510, Yeonsu-gu, Incheon 21990, South Korea
>
> -Original Message-
> *From:* "Richard Larsson"
> *To:* "이병석";
> *Cc:* "Simon Pfreundschuh"; <
> arts_users...@mailman.rrz.uni-hamburg.de>;
> *Sent:* 2018-09-17 (월) 18:21:57
> *Subject:* Re: [arts-users] ARTS OEM
>
> Hi,
>
> Just as a warning, suppressing the bad partition function error is not
> good practice since it is known that they are, you know, bad. Standard
> tests in ARTS fail by several tenths of Kelvin because the partition
> functions in ARTS are that bad. Just run with the provided tips xml-files
> instead, because if your temperatures are lower than 70 K or higher than
> 3000 K you probably have another problem.
>
> With hope,
> //Richard
>
> 2018-09-17 10:31 GMT+02:00 이병석 :
>
> Hello Simon Pfreundschuh,
>
> Thank you for your reply.
>
> I have employed both of the suggestions and still get the irregular
> errors. From the same controlfile modified in the previous email, I have
> included and changed to the following lines:
>
> OEM(method="lm",
>max_iter=15,
>display_progress=1,
>lm_ga_settings=[100,5,2,1000,1,99])
>
> and
>
> AgendaSet( inversion_iterate_agenda ){
>
> xClip(ijq = 0, limit_low = 150.0, limit_high = 350)
>
> Ignore(inversion_iteration_counter)
> # Map x to ARTS' variables
> x2artsStandard
>
> # To be safe, rerun checks dealing with the atmosphere
> atmfields_checkedCalc
> atmgeom_checkedCalc
>
> # Calculate yf and Jacobian matching x.
> yCalc( y=yf )
>
> # Add baseline term
> VectorAddVector( yf, yf, y_baseline )
>
> # This method takes cares of some "fixes" that are needed to get the
> Jacobian
> # right for iterative solutions. No need to call this WSM for linear
> inversions.
> jacobianAdjustAndTransform
> }
>
> I included both of the suggestions, and from there
> increased the maximum number of iterations to 15
> and set the temperature limits to be between 150 K and 350 K.
> All other lines remain the same, including the input absorption line and
> atmospheric fields data.
>
> Now I irregularly get the errors during the OEM computation as before, of
> t_field having NaN values or of bad partition function warning (which
> happens for any t_field value outside between 150 K and 300 K). While the
> bad partition function warning could be suppressed in the code, it seems
> that the same problem as before persists.
>
> Could you please look into this issue a little more? I am not sure how to
> tackle the problem.
>
> Thank you for your service.
>
> Regards,
>
>
> Byungsuk Lee
>
> ARA Consulting & Technology
>
> 30 Songdomirae-ro D-1510, Yeonsu-gu, Incheon 21990, South Korea
>
>
>
> -Original Message-
> *From:* "Simon Pfreundschuh"
> *To:* "이병석"; "arts_users...@mailman.rrz.uni-hamburg.de"<
> arts_users...@mailman.rrz.uni-hamburg.de>;
> *Cc:*
> *Sent:* 2018-09-14 (금) 15:13:55
> *Subject:* Re: [arts-users] ARTS OEM
>
> Hi Byungsuk Lee,
>
> It is a bit surprising that these errors occur irregularly, since there is
> no randomness
> in the test
Re: [arts-users] ARTS OEM
Hello Richard,
Thank you for your reply.
May I ask which tips xml files you are referring to?
Also a question: it seems that the bad partition function warning occurs for
any input t_field temperature beyond the range between 150 K and 300 K. In
reality the surface temperature of the Earth frequently exceeds 300 K (26.85
deg C). For input values such as 30 deg C or a little higher (e.g. 30-40 deg C,
early 300's K), would the results still be bad if the warning is suppressed?
Regards,
Byungsuk Lee
ARA Consulting & Technology
30 Songdomirae-ro D-1510, Yeonsu-gu, Incheon 21990, South Korea
-Original Message-
From: "Richard Larsson"
To: "이병석";
Cc: "Simon Pfreundschuh";
;
Sent: 2018-09-17 (월) 18:21:57
Subject: Re: [arts-users] ARTS OEM
Hi,
Just as a warning, suppressing the bad partition function error is not good
practice since it is known that they are, you know, bad. Standard tests in
ARTS fail by several tenths of Kelvin because the partition functions in ARTS
are that bad. Just run with the provided tips xml-files instead, because if
your temperatures are lower than 70 K or higher than 3000 K you probably have
another problem.
With hope,
//Richard
2018-09-17 10:31 GMT+02:00 이병석 :
Hello Simon Pfreundschuh,
Thank you for your reply.
I have employed both of the suggestions and still get the irregular errors.
From the same controlfile modified in the previous email, I have included and
changed to the following lines:
OEM(method="lm",
max_iter=15,
display_progress=1,
lm_ga_settings=[100,5,2,1000,1,99])
and
AgendaSet( inversion_iterate_agenda ){
xClip(ijq = 0, limit_low = 150.0, limit_high = 350)
Ignore(inversion_iteration_counter)
# Map x to ARTS' variables
x2artsStandard
# To be safe, rerun checks dealing with the atmosphere
atmfields_checkedCalc
atmgeom_checkedCalc
# Calculate yf and Jacobian matching x.
yCalc( y=yf )
# Add baseline term
VectorAddVector( yf, yf, y_baseline )
# This method takes cares of some "fixes" that are needed to get the Jacobian
# right for iterative solutions. No need to call this WSM for linear
inversions.
jacobianAdjustAndTransform
}
I included both of the suggestions, and from there increased the maximum number
of iterations to 15 and set the temperature limits to be between 150 K and 350
K. All other lines remain the same, including the input absorption line and
atmospheric fields data.
Now I irregularly get the errors during the OEM computation as before, of
t_field having NaN values or of bad partition function warning (which happens
for any t_field value outside between 150 K and 300 K). While the bad partition
function warning could be suppressed in the code, it seems that the same
problem as before persists.
Could you please look into this issue a little more? I am not sure how to
tackle the problem.
Thank you for your service.
Regards,
Byungsuk Lee
ARA Consulting & Technology
30 Songdomirae-ro D-1510, Yeonsu-gu, Incheon 21990, South Korea
-Original Message-
From: "Simon Pfreundschuh"
To: "이병석";
"arts_users...@mailman.rrz.uni-hamburg.de";
Cc:
Sent: 2018-09-14 (금) 15:13:55
Subject: Re: [arts-users] ARTS OEM
Hi Byungsuk Lee,
It is a bit surprising that these errors occur irregularly, since there is no
randomness
in the test setup. However, if you are running retrievals with external
observation data
it may well happen that the iteration ends up in a state that produces invalid
values.
For your temperature retrieval I would recommend one or both of the following
solutions:
1. Use the Levenberg-Marquardt (LM) iteration scheme: Using the LM scheme with
a high start
value for the gamma parameter can help avoid reaching invalid values when
the forward
model is non-linear:
OEM(method="lm",
max_iter=5,
display_progress=1,
lm_ga_settings=[100,5,2,1000,1,99])
2. Clip the x-vector: Using the xClip WSM inside your inversion iterate agenda
you can force the
x vector to contain only temperatures within a certain range. If you use
this approach you
should check that none of the retrieved temperatures in your final state are
clipped otherwise your
results may not be valid.
AgendaSet( inversion_iterate_agenda ){
xClip(ijq = 0, limit_low = 100.0)
Ignore(inversion_iteration_counter)
# Map x to ARTS' variables
x2artsStandard
# To be safe, rerun checks dealing with the atmosphere
atmfields_checkedCalc
atmgeom_checkedCalc
# Calculate yf and Jacobian matching x.
yCalc( y=yf )
# Add baseline term
VectorAddVector( yf, yf, y_baseline )
# This method takes cares of some "fixes" that are needed to get the
Jacobian
# right for iterative solutions. No need to call this WSM for linear
inversions.
jacobianAdjustAnd
Re: [arts-users] ARTS OEM
Hi,
Just as a warning, suppressing the bad partition function error is not good
practice since it is known that they are, you know, bad. Standard tests in
ARTS fail by several tenths of Kelvin because the partition functions in
ARTS are that bad. Just run with the provided tips xml-files instead,
because if your temperatures are lower than 70 K or higher than 3000 K you
probably have another problem.
With hope,
//Richard
2018-09-17 10:31 GMT+02:00 이병석 :
> Hello Simon Pfreundschuh,
>
> Thank you for your reply.
>
> I have employed both of the suggestions and still get the irregular
> errors. From the same controlfile modified in the previous email, I have
> included and changed to the following lines:
>
> OEM(method="lm",
>max_iter=15,
>display_progress=1,
>lm_ga_settings=[100,5,2,1000,1,99])
>
> and
>
> AgendaSet( inversion_iterate_agenda ){
>
> xClip(ijq = 0, limit_low = 150.0, limit_high = 350)
>
> Ignore(inversion_iteration_counter)
> # Map x to ARTS' variables
> x2artsStandard
>
> # To be safe, rerun checks dealing with the atmosphere
> atmfields_checkedCalc
> atmgeom_checkedCalc
>
> # Calculate yf and Jacobian matching x.
> yCalc( y=yf )
>
> # Add baseline term
> VectorAddVector( yf, yf, y_baseline )
>
> # This method takes cares of some "fixes" that are needed to get the
> Jacobian
> # right for iterative solutions. No need to call this WSM for linear
> inversions.
> jacobianAdjustAndTransform
> }
>
> I included both of the suggestions, and from there
> increased the maximum number of iterations to 15 and set the temperature
> limits to be between 150
> K and 350 K. All other lines remain the same, including the input
> absorption line and atmospheric fields data.
>
> Now I irregularly get the errors during the OEM computation as before, of
> t_field having NaN values or of bad partition function warning (which
> happens for any t_field value outside between 150 K and 300 K). While the
> bad partition function warning could be suppressed in the code, it seems
> that the same problem as before persists.
>
> Could you please look into this issue a little more? I am not sure how to
> tackle the problem.
>
> Thank you for your service.
>
> Regards,
>
>
> Byungsuk Lee
>
> ARA Consulting & Technology
>
> 30 Songdomirae-ro D-1510, Yeonsu-gu, Incheon 21990, South Korea
>
>
>
> -----Original Message-
> *From:* "Simon Pfreundschuh"
> *To:* "이병석"; "arts_users...@mailman.rrz.uni-hamburg.de"<
> arts_users...@mailman.rrz.uni-hamburg.de>;
> *Cc:*
> *Sent:* 2018-09-14 (금) 15:13:55
> *Subject:* Re: [arts-users] ARTS OEM
>
> Hi Byungsuk Lee,
>
> It is a bit surprising that these errors occur irregularly, since there is
> no randomness
> in the test setup. However, if you are running retrievals with external
> observation data
> it may well happen that the iteration ends up in a state that produces
> invalid values.
>
> For your temperature retrieval I would recommend one or both of the
> following solutions:
>
> 1. Use the Levenberg-Marquardt (LM) iteration scheme: Using the LM scheme
> with a high start
>value for the gamma parameter can help avoid reaching invalid values
> when the forward
>model is non-linear:
>
> OEM(method="lm",
> max_iter=5,
> display_progress=1,
> lm_ga_settings=[100,5,2,1000,1,99])
>
> 2. Clip the x-vector: Using the xClip WSM inside your inversion iterate
> agenda you can force the
>x vector to contain only temperatures within a certain range. If you
> use this approach you
>should check that none of the retrieved temperatures in your final
> state are clipped otherwise your
>results may not be valid.
>
> AgendaSet( inversion_iterate_agenda ){
>
> xClip(ijq = 0, limit_low = 100.0)
> Ignore(inversion_iteration_counter)
> # Map x to ARTS' variables
> x2artsStandard
>
> # To be safe, rerun checks dealing with the atmosphere
> atmfields_checkedCalc
> atmgeom_checkedCalc
>
> # Calculate yf and Jacobian matching x.
> yCalc( y=yf )
>
> # Add baseline term
> VectorAddVector( yf, yf, y_baseline )
>
> # This method takes cares of some "fixes" that are needed to get the
> Jacobian
> # right for iterative solutions. No need to call this WSM for linear
> inversions.
> jacobianAdjustAndTransform
> }
>
>
> Regards,
>
> Simon
>
> --
> *From:* arts_users.mi-boun...@mailman.rrz.uni
Re: [arts-users] ARTS OEM
Hello Simon Pfreundschuh,
Thank you for your reply.
I have employed both of the suggestions and still get the irregular errors.
From the same controlfile modified in the previous email, I have included and
changed to the following lines:
OEM(method="lm",
max_iter=15,
display_progress=1,
lm_ga_settings=[100,5,2,1000,1,99])
and
AgendaSet( inversion_iterate_agenda ){
xClip(ijq = 0, limit_low = 150.0, limit_high = 350)
Ignore(inversion_iteration_counter)
# Map x to ARTS' variables
x2artsStandard
# To be safe, rerun checks dealing with the atmosphere
atmfields_checkedCalc
atmgeom_checkedCalc
# Calculate yf and Jacobian matching x.
yCalc( y=yf )
# Add baseline term
VectorAddVector( yf, yf, y_baseline )
# This method takes cares of some "fixes" that are needed to get the Jacobian
# right for iterative solutions. No need to call this WSM for linear
inversions.
jacobianAdjustAndTransform
}
I included both of the suggestions, and from there increased the maximum number
of iterations to 15 and set the temperature limits to be between 150 K and 350
K. All other lines remain the same, including the input absorption line and
atmospheric fields data.
Now I irregularly get the errors during the OEM computation as before, of
t_field having NaN values or of bad partition function warning (which happens
for any t_field value outside between 150 K and 300 K). While the bad partition
function warning could be suppressed in the code, it seems that the same
problem as before persists.
Could you please look into this issue a little more? I am not sure how to
tackle the problem.
Thank you for your service.
Regards,
Byungsuk Lee
ARA Consulting & Technology
30 Songdomirae-ro D-1510, Yeonsu-gu, Incheon 21990, South Korea
-Original Message-
From: "Simon Pfreundschuh"
To: "이병석";
"arts_users...@mailman.rrz.uni-hamburg.de";
Cc:
Sent: 2018-09-14 (금) 15:13:55
Subject: Re: [arts-users] ARTS OEM
Hi Byungsuk Lee,
It is a bit surprising that these errors occur irregularly, since there is no
randomness
in the test setup. However, if you are running retrievals with external
observation data
it may well happen that the iteration ends up in a state that produces invalid
values.
For your temperature retrieval I would recommend one or both of the following
solutions:
1. Use the Levenberg-Marquardt (LM) iteration scheme: Using the LM scheme with
a high start
value for the gamma parameter can help avoid reaching invalid values when
the forward
model is non-linear:
OEM(method="lm",
max_iter=5,
display_progress=1,
lm_ga_settings=[100,5,2,1000,1,99])
2. Clip the x-vector: Using the xClip WSM inside your inversion iterate agenda
you can force the
x vector to contain only temperatures within a certain range. If you use
this approach you
should check that none of the retrieved temperatures in your final state are
clipped otherwise your
results may not be valid.
AgendaSet( inversion_iterate_agenda ){
xClip(ijq = 0, limit_low = 100.0)
Ignore(inversion_iteration_counter)
# Map x to ARTS' variables
x2artsStandard
# To be safe, rerun checks dealing with the atmosphere
atmfields_checkedCalc
atmgeom_checkedCalc
# Calculate yf and Jacobian matching x.
yCalc( y=yf )
# Add baseline term
VectorAddVector( yf, yf, y_baseline )
# This method takes cares of some "fixes" that are needed to get the
Jacobian
# right for iterative solutions. No need to call this WSM for linear
inversions.
jacobianAdjustAndTransform
}
Regards,
Simon
From: arts_users.mi-boun...@mailman.rrz.uni-hamburg.de
on behalf of 이병석
Sent: Friday, September 14, 2018 2:31:25 AM
To: arts_users...@mailman.rrz.uni-hamburg.de
Subject: [arts-users] ARTS OEM
Dear ARTS users,
I am writing to address a question with the use of OEM in the ARTS development
version.
From the "TestOEM.arts" in "ARTS/controlfiles/artscomponents/oem", this time I
changed the codes as follows.
1) From (line 115):
retrievalAddAbsSpecies(
species = "O3",
unit = "vmr",
g1 = p_ret_grid,
g2 = lat_grid,
g3 = lon_grid
)
To:
retrievalAddTemperature (
g1 = p_ret_grid,
g2 = lat_grid,
g3 = lon_grid,
hse = "off" )
2) From (line 125):
VectorSetConstant(vars, nelem, 1e-12)
DiagonalMatrix(sparse_block, vars)
covmat_sxAddBlock(block = sparse_block)
To:
VectorSetConstant(vars, nelem, 0.1)
DiagonalMatrix(sparse_block, vars)
covmat_sxAddBlock(block = sparse_block)
3) From (line 37):
NumericSet( f_start, 110.436e9 )
NumericSet( f_end, 111.236e9 )
IndexSet( nf, 801 )
IndexSet( np, 81 )
To:
NumericSet( f_start, 110.436e9 )
NumericSet( f_end, 111.236e9 )
IndexSet( nf, 18 )
IndexSet( np, 18 )
4) From (line 42):
VectorNLinS
Re: [arts-users] ARTS OEM
Hi Byungsuk Lee,
It is a bit surprising that these errors occur irregularly, since there is no
randomness
in the test setup. However, if you are running retrievals with external
observation data
it may well happen that the iteration ends up in a state that produces invalid
values.
For your temperature retrieval I would recommend one or both of the following
solutions:
1. Use the Levenberg-Marquardt (LM) iteration scheme: Using the LM scheme with
a high start
value for the gamma parameter can help avoid reaching invalid values when
the forward
model is non-linear:
OEM(method="lm",
max_iter=5,
display_progress=1,
lm_ga_settings=[100,5,2,1000,1,99])
2. Clip the x-vector: Using the xClip WSM inside your inversion iterate agenda
you can force the
x vector to contain only temperatures within a certain range. If you use
this approach you
should check that none of the retrieved temperatures in your final state are
clipped otherwise your
results may not be valid.
AgendaSet( inversion_iterate_agenda ){
xClip(ijq = 0, limit_low = 100.0)
Ignore(inversion_iteration_counter)
# Map x to ARTS' variables
x2artsStandard
# To be safe, rerun checks dealing with the atmosphere
atmfields_checkedCalc
atmgeom_checkedCalc
# Calculate yf and Jacobian matching x.
yCalc( y=yf )
# Add baseline term
VectorAddVector( yf, yf, y_baseline )
# This method takes cares of some "fixes" that are needed to get the
Jacobian
# right for iterative solutions. No need to call this WSM for linear
inversions.
jacobianAdjustAndTransform
}
Regards,
Simon
From: arts_users.mi-boun...@mailman.rrz.uni-hamburg.de
on behalf of 이병석
Sent: Friday, September 14, 2018 2:31:25 AM
To: arts_users...@mailman.rrz.uni-hamburg.de
Subject: [arts-users] ARTS OEM
Dear ARTS users,
I am writing to address a question with the use of OEM in the ARTS development
version.
From the "TestOEM.arts" in "ARTS/controlfiles/artscomponents/oem", this time I
changed the codes as follows.
1) From (line 115):
retrievalAddAbsSpecies(
species = "O3",
unit = "vmr",
g1 = p_ret_grid,
g2 = lat_grid,
g3 = lon_grid
)
To:
retrievalAddTemperature (
g1 = p_ret_grid,
g2 = lat_grid,
g3 = lon_grid,
hse = "off" )
2) From (line 125):
VectorSetConstant(vars, nelem, 1e-12)
DiagonalMatrix(sparse_block, vars)
covmat_sxAddBlock(block = sparse_block)
To:
VectorSetConstant(vars, nelem, 0.1)
DiagonalMatrix(sparse_block, vars)
covmat_sxAddBlock(block = sparse_block)
3) From (line 37):
NumericSet( f_start, 110.436e9 )
NumericSet( f_end, 111.236e9 )
IndexSet( nf, 801 )
IndexSet( np, 81 )
To:
NumericSet( f_start, 110.436e9 )
NumericSet( f_end, 111.236e9 )
IndexSet( nf, 18 )
IndexSet( np, 18 )
4) From (line 42):
VectorNLinSpace( f_grid, nf, f_start, f_end )
VectorNLogSpace( p_grid,361, 500e2, 0.1 )
VectorNLogSpace( p_ret_grid, np, 500e2, 0.1 )
To:
VectorNLinSpace( f_grid, nf, f_start, f_end )
VectorNLogSpace( p_grid,18, 500e2, 0.1 )
VectorNLogSpace( p_ret_grid, np, 500e2, 0.1 )
Overall, I changed the ozone VMR retrieval to temperature retrieval and reduced
the number of frequencies and pressure layers.
When I run the controlfile multiple times, I irregularly get errors. The error
messages are
Run-time error in oem computation: Forward Model Evaluation Error
Run-time error in agenda: inversion_iterate_agenda
Run-time error in method: atmfields_checkedCalc
The variable *t_field* contains one or several NaNs. This is not allowed!
or
All temperatures in *t_field* must be > 0.
or seldom the bad partition function warning.
Here are my questions:
1) Is this an anticipated error, simply coming from failing to find a
converging solution? Is it expected that the OEM sometimes throws an error as
above?
2) Are there mechanisms (e.g. workspace methods) within the ARTS that can
prevent the t_field or vmr_field from becoming either negative or NaN during
the OEM iterations?
Thank you for your assistance.
Regards,
Byungsuk Lee
ARA Consulting & Technology
30 Songdomirae-ro D-1510, Yeonsu-gu, Incheon 21990, South Korea
Email: cb...@aracnt.com
[https://kr1-mail.worksmobile.com/readReceipt/notify/?img=SlbwKAgmKAumaxUwKotwKAushzKXF4UZFAJCF6MXMr3CF4k0pog9aA2%2Fpzu%2FKqu%2FM43gMXpXp6UmKxE5%2Br9C16m5W4C5bX0q%2BzkR74FTWx%2FsMBidb9e9brkZbZlR1EIn1BFdbZl9W4vR1zJRMNkZpZlvpt%3D%3D.gif]
___
arts_users.mi mailing list
arts_users.mi@lists.uni-hamburg.de
https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi
[arts-users] ARTS OEM
Dear ARTS users, I am writing to address a question with the use of OEM in the ARTS development version. From the "TestOEM.arts" in "ARTS/controlfiles/artscomponents/oem", this time I changed the codes as follows. 1) From (line 115): retrievalAddAbsSpecies( species = "O3", unit = "vmr", g1 = p_ret_grid, g2 = lat_grid, g3 = lon_grid ) To: retrievalAddTemperature ( g1 = p_ret_grid, g2 = lat_grid, g3 = lon_grid, hse = "off" ) 2) From (line 125): VectorSetConstant(vars, nelem, 1e-12) DiagonalMatrix(sparse_block, vars) covmat_sxAddBlock(block = sparse_block) To: VectorSetConstant(vars, nelem, 0.1) DiagonalMatrix(sparse_block, vars) covmat_sxAddBlock(block = sparse_block) 3) From (line 37): NumericSet( f_start, 110.436e9 ) NumericSet( f_end, 111.236e9 ) IndexSet( nf, 801 ) IndexSet( np, 81 ) To: NumericSet( f_start, 110.436e9 ) NumericSet( f_end, 111.236e9 ) IndexSet( nf, 18 ) IndexSet( np, 18 ) 4) From (line 42): VectorNLinSpace( f_grid, nf, f_start, f_end ) VectorNLogSpace( p_grid,361, 500e2, 0.1 ) VectorNLogSpace( p_ret_grid, np, 500e2, 0.1 ) To: VectorNLinSpace( f_grid, nf, f_start, f_end ) VectorNLogSpace( p_grid,18, 500e2, 0.1 ) VectorNLogSpace( p_ret_grid, np, 500e2, 0.1 ) Overall, I changed the ozone VMR retrieval to temperature retrieval and reduced the number of frequencies and pressure layers. When I run the controlfile multiple times, I irregularly get errors. The error messages are Run-time error in oem computation: Forward Model Evaluation Error Run-time error in agenda: inversion_iterate_agenda Run-time error in method: atmfields_checkedCalc The variable *t_field* contains one or several NaNs. This is not allowed! or All temperatures in *t_field* must be > 0. or seldom the bad partition function warning. Here are my questions: 1) Is this an anticipated error, simply coming from failing to find a converging solution? Is it expected that the OEM sometimes throws an error as above? 2) Are there mechanisms (e.g. workspace methods) within the ARTS that can prevent the t_field or vmr_field from becoming either negative or NaN during the OEM iterations? Thank you for your assistance. Regards, Byungsuk Lee ARA Consulting & Technology 30 Songdomirae-ro D-1510, Yeonsu-gu, Incheon 21990, South Korea Email: cb...@aracnt.com ___ arts_users.mi mailing list arts_users.mi@lists.uni-hamburg.de https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi
