[hugin-ptx] Re: possible memory leak in enblend enfuse? (was: hugin-mac-2009.4.0-Beta1 for download)
Hi Stefan, 2009/10/8 Stefan Peter s_pe...@swissonline.ch: Hi List, Sorry if the following is all known by the old hands. Please feel free to correct me at your leisure, otherwise I may die dumb ;) I have downloaded the project in question, and here are some results. I used Hugin 2009.2.0.4461 on linux X86_64 with 4GB Memory and 8 GB swap for the first test. After creating a makefile from the pto, all tests have been conducted on the command line. o The original project worked fine in relatively short time (~ 30 minutes, IIRC). o After adding an additional target (blended and fused equirect), the system started to thrash after about 1 hour and I had to cancel the job after a runtime of 9 hours without result. These tests where done using enblend 4.0-da9bc1a1ed87 Afterwards, I switched to Linux X86_32 with 3GB of memory and 1 GB of swap: o The original project did not finish. An strace of the final enblend call (the one that creates the final equirect) showed that an mmap operation on one of the source files (400 MBytes!) failed. o Subsequent tests with various differently compiled enblend versions had some influence, but did not provide a remedy. I was not able to stitch the final pano, no matter what options i used for compiling enblend. But the point of failure shifted from Image 2 to Image 17 when disabling openmp and enabling image cache. As a preliminary statement, I'd say that o Enblending/enfusing large panos with 16bit tiffs is very memory intensive. If you don't have the RAM required, the final enblend step will fail with an out of memory message. o Having plenty of swap space is no cure: the system starts thrashing and will get unresponsive up to the point where you cancel the job voluntarily. o Although I did some valgrind and dmalloc tests, I could not find any leeks or problems beside the fact that all tiff files where accessed using mmap. This results in a memory footprint that mostly depends on the size of the input files. As a temporary fix, I would recommend to use --disable--openmp and --enable-image-cache in order to limit the memory consumption. However, this has a negative influence on the runtime of enblend / enfuse. Another option is to reduce the size of the input images. It may be sufficient to use a compression for tiffs (I have not tested this), but reducing the color depth from 16 to 8 bit will have, most probably, the largest influence. The final solution for the memory problems in enblend / enfuse can only be an access method that does not rely on being able to access the _whole_ source files over memory maps. Without having looked at the code responsible for this (and not being a C++ programmer at all), I think that it should be possible to use a window (let's say 128 MBytes) for accessing at least the input files. With kind regards Stefan Peter Thank you very much for you in depth analysis. Here are my thoughts. Could anyone verify them? 1. the problem occurs when image cache is disabled or the limit exceeds (or almost fits almost exatly) memory size 2. the problem occurs when image cache is enabled but there is not enough memory to write temporary to disk. Both of them makes me think that it can be avoided by carefully setting image cache to let's say available memory minus 100MB. So the simplest solution may be setting apropriate image cache value at runtime. But it would have it's drawbacks on systems with hot pluggable memory. Lukas --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups hugin and other free panoramic software group. A list of frequently asked questions is available at: http://wiki.panotools.org/Hugin_FAQ To post to this group, send email to hugin-ptx@googlegroups.com To unsubscribe from this group, send email to hugin-ptx+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/hugin-ptx -~--~~~~--~~--~--~---
[hugin-ptx] Re: possible memory leak in enblend enfuse? (was: hugin-mac-2009.4.0-Beta1 for download)
2009/10/7 Harry van der Wolf hvdw...@gmail.com: I did another run from the Gui (took a few minutes to find how to compile that one). That gave more results: I didn't know that there is a GUI for it. [assemble.h:308]: (possible style) Pre-Incrementing variable 'i' is preferred to Post-Incrementing [assemble.h:308]: (possible style) Pre-Incrementing variable 'i' is preferred to Post-Incrementing [vigra_impex/exr.cxx:310]: (style) Redundant condition. It is safe to deallocate a NULL pointer [vigra_impex/hdr.cxx:116]: (style) Member variable not initialized in the constructor 'HDRCodecImpl::height' [vigra_impex/hdr.cxx:116]: (style) Member variable not initialized in the constructor 'HDRCodecImpl::width' [vigra_impex/jpeg.cxx:132]: (error) Class JPEGCodecImpl which is inherited by class JPEGDecoderImplBase does not have a virtual destructor [vigra_impex/jpeg.cxx:132]: (error) Class JPEGCodecImpl which is inherited by class JPEGEncoderImplBase does not have a virtual destructor [vigra_impex/rgbe.c:90]: (style) The scope of the variable f can be limited [vigra_impex/tiff.cxx:199]: (style) Member variable not initialized in the constructor 'TIFFCodecImpl::layers' [vigra_impex/viff.cxx:245]: (style) Function parameter 'index' is passed by value. It could be passed by reference instead. I've got these: [enblend/include/vigra/multi_array.hxx:318]: (error) Class MultiArrayView which is inherited by class MultiArray does not have a virtual destructor [enblend/src/vigra_impex/jpeg.cxx:132]: (error) Class JPEGCodecImpl which is inherited by class JPEGDecoderImplBase does not have a virtual destructor [enblend/src/vigra_impex/jpeg.cxx:132]: (error) Class JPEGCodecImpl which is inherited by class JPEGEncoderImplBase does not have a virtual destructor This doesn't seem to be memory leaks of some kind. Still I report it here. Thanks Harry 2009/10/7 Harry van der Wolf hvdw...@gmail.com To answer my own mail. :-) I just compiled cppcheck on OSX and did a standard run on the enblend trunk. It displays the following [./vigra_impex/jpeg.cxx:132]: (error) Class JPEGCodecImpl which is inherited by class JPEGDecoderImplBase does not have a virtual destructor [./vigra_impex/jpeg.cxx:132]: (error) Class JPEGCodecImpl which is inherited by class JPEGEncoderImplBase does not have a virtual destructor Then I did a full check. That resulted in exactly the same result as mentioned above. This is like giving a monkey an encyclopedia. He has no idea what to do with it. I don't even know if this is usefull. I hope some clever progammers (super monkeys?) know what to do with this. Harry 2009/10/7 Harry van der Wolf hvdw...@gmail.com 2009/10/7 Lukáš Jirkovský l.jirkov...@gmail.com I'm not Mac user (although I find it really cool but very expensive) but I may found solution for out of memory problem. I had a discussion about memory and I mentioned these fragmentation problems on OS. And get interesting advice – use TLSF [1]. It is said that it doesn't suffer from fragmentation much. Maybe some of you want to take a look at it. [1] http://rtportal.upv.es/rtmalloc/ Lukas Hi Lukáš, (You are the one person who I had hoped to react. I started to do tests myself yesterday evening but they take long). Both on 32bit windows, 32bit linux and 32bit OSX, or with the 32bit versions of enblend, we regularly see the error message of enblend crashing due to a memory allocation error. We see this both at hugin-ptx as well as in the bugtracker for large projects. We sometimes see it for enfuse in case of large projects which need to be fused as well. Untill now we blamed it on memory fragmentation, but maybe it's something else. George Row is one of the persons who encounters these errors on OSX. I have received a large set (2.5GB) from George in June, some time before my mac crashed. At that time I did some tests. These results are gone (no backup of test results).Yesterday I took Georges set from my big disk backup server and did some tests myself trying to stich a 12000X6000 (slightly bigger) panorama in hugin-2009.4.0-beta1. My question to you now is: You recently did some memory leak patching on the hugin trunk, using cppcheck, thereby finding some things in celeste. You reported this via http://groups.google.com/group/hugin-ptx/browse_thread/thread/e2b5b09e4706fb80. Can you do the same for the enblend trunk? If you want to do this and you find the time for it in the near furure, be so kind to publish this on the hugin-ptx list. But please: don't feel obliged. If you don't have time or just do not want to do this: just say so. = If you don't want to do this, you can now stop reading. = If you want to do this or are at least interested: please continue reading. Below you will find my tests on OSX. It's done on the 2.5GB bracked village hotel project of George Row. Below the tail of the enblend error for a very recent 32bit
[hugin-ptx] Re: possible memory leak in enblend enfuse? (was: hugin-mac-2009.4.0-Beta1 for download)
2009/10/7 Stefan Peter s_pe...@swissonline.ch: Hi Lukáš instead of changing the RAM on your PC, you could use a virtual machine like vmware or virtualbox. There, you can limit the resources at your will. Regards Stefan Peter Hi Stefan, I know about this option but I thing changing RAM is faster. I'm already using vmware but I don't have necessary development tools installed on any hosted OS. Lukas --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups hugin and other free panoramic software group. A list of frequently asked questions is available at: http://wiki.panotools.org/Hugin_FAQ To post to this group, send email to hugin-ptx@googlegroups.com To unsubscribe from this group, send email to hugin-ptx-unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/hugin-ptx -~--~~~~--~~--~--~---
[hugin-ptx] Re: possible memory leak in enblend enfuse? (was: hugin-mac-2009.4.0-Beta1 for download)
2009/10/8 Lukáš Jirkovský l.jirkov...@gmail.com 2009/10/7 Harry van der Wolf hvdw...@gmail.com: I did another run from the Gui (took a few minutes to find how to compile that one). That gave more results: I didn't know that there is a GUI for it. cd into the gui directory and run qmake; make; make install. This off course implies that the QT development stuff and (runtime) libraries are available on your system. Harry --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups hugin and other free panoramic software group. A list of frequently asked questions is available at: http://wiki.panotools.org/Hugin_FAQ To post to this group, send email to hugin-ptx@googlegroups.com To unsubscribe from this group, send email to hugin-ptx-unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/hugin-ptx -~--~~~~--~~--~--~---
[hugin-ptx] Re: possible memory leak in enblend enfuse? (was: hugin-mac-2009.4.0-Beta1 for download)
2009/10/8 Harry van der Wolf hvdw...@gmail.com: 2009/10/8 Lukáš Jirkovský l.jirkov...@gmail.com 2009/10/7 Harry van der Wolf hvdw...@gmail.com: I did another run from the Gui (took a few minutes to find how to compile that one). That gave more results: I didn't know that there is a GUI for it. cd into the gui directory and run qmake; make; make install. This off course implies that the QT development stuff and (runtime) libraries are available on your system. Harry It seems that the package I'm using doesn't contain GUI. Maybe I'll try it some day but the command line is quite sufficient for me ;-) Lukas --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups hugin and other free panoramic software group. A list of frequently asked questions is available at: http://wiki.panotools.org/Hugin_FAQ To post to this group, send email to hugin-ptx@googlegroups.com To unsubscribe from this group, send email to hugin-ptx-unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/hugin-ptx -~--~~~~--~~--~--~---
[hugin-ptx] Re: possible memory leak in enblend enfuse? (was: hugin-mac-2009.4.0-Beta1 for download)
I downloaded the git from danmar_cppcheck 2009/10/8 Lukáš Jirkovský l.jirkov...@gmail.com 2009/10/8 Harry van der Wolf hvdw...@gmail.com: 2009/10/8 Lukáš Jirkovský l.jirkov...@gmail.com 2009/10/7 Harry van der Wolf hvdw...@gmail.com: I did another run from the Gui (took a few minutes to find how to compile that one). That gave more results: I didn't know that there is a GUI for it. cd into the gui directory and run qmake; make; make install. This off course implies that the QT development stuff and (runtime) libraries are available on your system. Harry It seems that the package I'm using doesn't contain GUI. Maybe I'll try it some day but the command line is quite sufficient for me ;-) Lukas --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups hugin and other free panoramic software group. A list of frequently asked questions is available at: http://wiki.panotools.org/Hugin_FAQ To post to this group, send email to hugin-ptx@googlegroups.com To unsubscribe from this group, send email to hugin-ptx-unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/hugin-ptx -~--~~~~--~~--~--~---
[hugin-ptx] Re: possible memory leak in enblend enfuse? (was: hugin-mac-2009.4.0-Beta1 for download)
I've tried to reproduce the bug but without success. I've remapped images. Enblend usually takes about 2-3 GB of memory with this pano. I lowered RAM to 256MB and swap from 512MB to 195MB. (KDE takes about 200MB itself). I've run enblend with: -m 200 -b 8196 to expose the bug even earlier. After almost 4 hours when the free memory was always about 50MB and my PC almost didn't respond (after key press in terminal it took about 2 minutes to draw anything) I've not exposed the bug. Lukas --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups hugin and other free panoramic software group. A list of frequently asked questions is available at: http://wiki.panotools.org/Hugin_FAQ To post to this group, send email to hugin-ptx@googlegroups.com To unsubscribe from this group, send email to hugin-ptx-unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/hugin-ptx -~--~~~~--~~--~--~---
[hugin-ptx] Re: possible memory leak in enblend enfuse? (was: hugin-mac-2009.4.0-Beta1 for download)
Am Thursday 08 October 2009 schrieb Lukáš Jirkovský: I've tried to reproduce the bug but without success. I've remapped images. Enblend usually takes about 2-3 GB of memory with this pano. I lowered RAM to 256MB and swap from 512MB to 195MB. (KDE takes about 200MB itself). I've run enblend with: -m 200 -b 8196 to expose the bug even earlier. After almost 4 hours when the free memory was always about 50MB and my PC almost didn't respond (after key press in terminal it took about 2 minutes to draw anything) I've not exposed the bug. Lukas Maybe the '-m' option _hides_ the bug. At least I observed this behaviour earlier. Now I use enblend without image cache, here the '-m' option does not make sense. Kornel -- Kornel Benko kornel.be...@berlin.de signature.asc Description: This is a digitally signed message part.
[hugin-ptx] Re: possible memory leak in enblend enfuse? (was: hugin-mac-2009.4.0-Beta1 for download)
Hi List, Sorry if the following is all known by the old hands. Please feel free to correct me at your leisure, otherwise I may die dumb ;) I have downloaded the project in question, and here are some results. I used Hugin 2009.2.0.4461 on linux X86_64 with 4GB Memory and 8 GB swap for the first test. After creating a makefile from the pto, all tests have been conducted on the command line. o The original project worked fine in relatively short time (~ 30 minutes, IIRC). o After adding an additional target (blended and fused equirect), the system started to thrash after about 1 hour and I had to cancel the job after a runtime of 9 hours without result. These tests where done using enblend 4.0-da9bc1a1ed87 Afterwards, I switched to Linux X86_32 with 3GB of memory and 1 GB of swap: o The original project did not finish. An strace of the final enblend call (the one that creates the final equirect) showed that an mmap operation on one of the source files (400 MBytes!) failed. o Subsequent tests with various differently compiled enblend versions had some influence, but did not provide a remedy. I was not able to stitch the final pano, no matter what options i used for compiling enblend. But the point of failure shifted from Image 2 to Image 17 when disabling openmp and enabling image cache. As a preliminary statement, I'd say that o Enblending/enfusing large panos with 16bit tiffs is very memory intensive. If you don't have the RAM required, the final enblend step will fail with an out of memory message. o Having plenty of swap space is no cure: the system starts thrashing and will get unresponsive up to the point where you cancel the job voluntarily. o Although I did some valgrind and dmalloc tests, I could not find any leeks or problems beside the fact that all tiff files where accessed using mmap. This results in a memory footprint that mostly depends on the size of the input files. As a temporary fix, I would recommend to use --disable--openmp and --enable-image-cache in order to limit the memory consumption. However, this has a negative influence on the runtime of enblend / enfuse. Another option is to reduce the size of the input images. It may be sufficient to use a compression for tiffs (I have not tested this), but reducing the color depth from 16 to 8 bit will have, most probably, the largest influence. The final solution for the memory problems in enblend / enfuse can only be an access method that does not rely on being able to access the _whole_ source files over memory maps. Without having looked at the code responsible for this (and not being a C++ programmer at all), I think that it should be possible to use a window (let's say 128 MBytes) for accessing at least the input files. With kind regards Stefan Peter --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups hugin and other free panoramic software group. A list of frequently asked questions is available at: http://wiki.panotools.org/Hugin_FAQ To post to this group, send email to hugin-ptx@googlegroups.com To unsubscribe from this group, send email to hugin-ptx-unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/hugin-ptx -~--~~~~--~~--~--~---
[hugin-ptx] Re: possible memory leak in enblend enfuse? (was: hugin-mac-2009.4.0-Beta1 for download)
Hi I'd be interested in doing some tests, too. I remember having had memory issues as well, but I was never able to reproduce them reliably here on Linux / Linux_64 and Windows. Is there someplace one could get the project in question? Cheers Stefan Peter --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups hugin and other free panoramic software group. A list of frequently asked questions is available at: http://wiki.panotools.org/Hugin_FAQ To post to this group, send email to hugin-ptx@googlegroups.com To unsubscribe from this group, send email to hugin-ptx-unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/hugin-ptx -~--~~~~--~~--~--~---
[hugin-ptx] Re: possible memory leak in enblend enfuse? (was: hugin-mac-2009.4.0-Beta1 for download)
To answer my own mail. :-) I just compiled cppcheck on OSX and did a standard run on the enblend trunk. It displays the following [./vigra_impex/jpeg.cxx:132]: (error) Class JPEGCodecImpl which is inherited by class JPEGDecoderImplBase does not have a virtual destructor [./vigra_impex/jpeg.cxx:132]: (error) Class JPEGCodecImpl which is inherited by class JPEGEncoderImplBase does not have a virtual destructor Then I did a full check. That resulted in exactly the same result as mentioned above. This is like giving a monkey an encyclopedia. He has no idea what to do with it. I don't even know if this is usefull. I hope some clever progammers (super monkeys?) know what to do with this. Harry 2009/10/7 Harry van der Wolf hvdw...@gmail.com 2009/10/7 Lukáš Jirkovský l.jirkov...@gmail.com I'm not Mac user (although I find it really cool but very expensive) but I may found solution for out of memory problem. I had a discussion about memory and I mentioned these fragmentation problems on OS. And get interesting advice – use TLSF [1]. It is said that it doesn't suffer from fragmentation much. Maybe some of you want to take a look at it. [1] http://rtportal.upv.es/rtmalloc/ Lukas Hi Lukáš, (You are the one person who I had hoped to react. I started to do tests myself yesterday evening but they take long). Both on 32bit windows, 32bit linux and 32bit OSX, or with the 32bit versions of enblend, we regularly see the error message of enblend crashing due to a memory allocation error. We see this both at hugin-ptx as well as in the bugtracker for large projects. We sometimes see it for enfuse in case of large projects which need to be fused as well. Untill now we blamed it on memory fragmentation, but maybe it's something else. George Row is one of the persons who encounters these errors on OSX. I have received a large set (2.5GB) from George in June, some time before my mac crashed. At that time I did some tests. These results are gone (no backup of test results).Yesterday I took Georges set from my big disk backup server and did some tests myself trying to stich a 12000X6000 (slightly bigger) panorama in hugin-2009.4.0-beta1. My question to you now is: You recently did some memory leak patching on the hugin trunk, using cppcheck, thereby finding some things in celeste. You reported this via http://groups.google.com/group/hugin-ptx/browse_thread/thread/e2b5b09e4706fb80 . Can you do the same for the enblend trunk? If you want to do this and you find the time for it in the near furure, be so kind to publish this on the hugin-ptx list. But please: don't feel obliged. If you don't have time or just do not want to do this: just say so. = If you don't want to do this, you can now stop reading. = If you want to do this or are at least interested: please continue reading. Below you will find my tests on OSX. It's done on the 2.5GB bracked village hotel project of George Row. Below the tail of the enblend error for a very recent 32bit Christoph Spiel build: enblend: info: loading next image: FoyleDays_M2_040007.tif 1/1 enblend: info: creating blend mask: 1/3enblend(11221) malloc: *** vm_allocate(size=580620288) failed (error code=3) enblend(11221) malloc: *** error: can't allocate region enblend(11221) malloc: *** set a breakpoint in szone_error to debug enblend: out of memory enblend: St9bad_alloc gnumake: *** [FoyleDays_M2_04.tif] Error 1 Below the tail of the error for the stable 32bit enblend 3.2 build (This to prove it's not a recent problem. It's already there in the 3.2 stable build). enblend(50447) malloc: *** mmap(size=2097152) failed (error code=12) *** error: can't allocate region *** set a breakpoint in malloc_error_break to debug enblend(50447) malloc: *** mmap(size=2097152) failed (error code=12) *** error: can't allocate region *** set a breakpoint in malloc_error_break to debug enblend(50447) malloc: *** mmap(size=2097152) failed (error code=12) *** error: can't allocate region *** set a breakpoint in malloc_error_break to debug enblend(50447) malloc: *** mmap(size=2097152) failed (error code=12) *** error: can't allocate region *** set a breakpoint in malloc_error_break to debug enblend: out of memory St9bad_alloc gnumake: *** [mamaloe_exposure_00.tif] Error 1 Test report for a 64bit enblend: After 6 hours and much further in the process my system crashed. I will rerun tonight. (I can start and monitor remote. I can't start a crashed mac from remote.) Note: I build 32bit binaries by default as they run on every mac. A 64 bit version only runs on Leopard and Snow Leopard on 64bit hardware. 64bit brings hardly performance gain or other benefits, only when making gigapixel pano's.) To me this does NOT mean that the 64 bit behaves better. It only proves IMO that due to the huge 64bit address space, enblend can (might) just continue leaving it 's memory junk without filling the address space that
[hugin-ptx] Re: possible memory leak in enblend enfuse? (was: hugin-mac-2009.4.0-Beta1 for download)
Hi Harry, 2009/10/7 Harry van der Wolf hvdw...@gmail.com: 2009/10/7 Lukáš Jirkovský l.jirkov...@gmail.com I'm not Mac user (although I find it really cool but very expensive) but I may found solution for out of memory problem. I had a discussion about memory and I mentioned these fragmentation problems on OS. And get interesting advice – use TLSF [1]. It is said that it doesn't suffer from fragmentation much. Maybe some of you want to take a look at it. [1] http://rtportal.upv.es/rtmalloc/ Lukas Hi Lukáš, (You are the one person who I had hoped to react. I started to do tests myself yesterday evening but they take long). Thanks for being confident in me. Both on 32bit windows, 32bit linux and 32bit OSX, or with the 32bit versions of enblend, we regularly see the error message of enblend crashing due to a memory allocation error. We see this both at hugin-ptx as well as in the bugtracker for large projects. We sometimes see it for enfuse in case of large projects which need to be fused as well. Untill now we blamed it on memory fragmentation, but maybe it's something else. It is possible that it's something else. The question is: how to find it? George Row is one of the persons who encounters these errors on OSX. I have received a large set (2.5GB) from George in June, some time before my mac crashed. At that time I did some tests. These results are gone (no backup of test results).Yesterday I took Georges set from my big disk backup server and did some tests myself trying to stich a 12000X6000 (slightly bigger) panorama in hugin-2009.4.0-beta1. My question to you now is: You recently did some memory leak patching on the hugin trunk, using cppcheck, thereby finding some things in celeste. You reported this via http://groups.google.com/group/hugin-ptx/browse_thread/thread/e2b5b09e4706fb80. Can you do the same for the enblend trunk? If you want to do this and you find the time for it in the near furure, be so kind to publish this on the hugin-ptx list. But please: don't feel obliged. If you don't have time or just do not want to do this: just say so. The cppcheck is still running but enblend.cc has already been checked. It has not detected any (interesting) problems yet. The major leak of these static analysis tools is that they are not perfect. Even the ones like Coverity uses. It may be interesting to try valgrind. Note: I've just read your post that it doesn't detect anything interesting. = If you don't want to do this, you can now stop reading. = If you want to do this or are at least interested: please continue reading. Below you will find my tests on OSX. It's done on the 2.5GB bracked village hotel project of George Row. Below the tail of the enblend error for a very recent 32bit Christoph Spiel build: enblend: info: loading next image: FoyleDays_M2_040007.tif 1/1 enblend: info: creating blend mask: 1/3enblend(11221) malloc: *** vm_allocate(size=580620288) failed (error code=3) enblend(11221) malloc: *** error: can't allocate region enblend(11221) malloc: *** set a breakpoint in szone_error to debug enblend: out of memory enblend: St9bad_alloc gnumake: *** [FoyleDays_M2_04.tif] Error 1 Below the tail of the error for the stable 32bit enblend 3.2 build (This to prove it's not a recent problem. It's already there in the 3.2 stable build). enblend(50447) malloc: *** mmap(size=2097152) failed (error code=12) *** error: can't allocate region *** set a breakpoint in malloc_error_break to debug enblend(50447) malloc: *** mmap(size=2097152) failed (error code=12) *** error: can't allocate region *** set a breakpoint in malloc_error_break to debug enblend(50447) malloc: *** mmap(size=2097152) failed (error code=12) *** error: can't allocate region *** set a breakpoint in malloc_error_break to debug enblend(50447) malloc: *** mmap(size=2097152) failed (error code=12) *** error: can't allocate region *** set a breakpoint in malloc_error_break to debug enblend: out of memory St9bad_alloc gnumake: *** [mamaloe_exposure_00.tif] Error 1 It would be nice to find out which malloc() fails. According to the older discussion it seems to be a problem in CachedFileImage. IMO the best thing how to try it would be do build enblend without image cache and create HUGE swap space so it won't run out of memory. If stitch works then we should look for the problem in image cache. I'd try it here but it would take ages since I don't have very powerful PC. I'll try to limit RAM on my PC (there is some switch to Linux kernel but IIRC it's almost undocumented) If it doesn't work I can try to replace all my RAM modules with an old 256MB RAM module and disable/reduce swap space. Then it may occur earlier and with smaller projects. I'm a bit afraid that it doesn't depend on how much RAM (or better virtual memory) is there but on the stitch size. ie. that when the stitch output is big enough it exposes some weird bug when it
[hugin-ptx] Re: possible memory leak in enblend enfuse? (was: hugin-mac-2009.4.0-Beta1 for download)
I did another run from the Gui (took a few minutes to find how to compile that one). That gave more results: [assemble.h:308]: (possible style) Pre-Incrementing variable 'i' is preferred to Post-Incrementing [assemble.h:308]: (possible style) Pre-Incrementing variable 'i' is preferred to Post-Incrementing [vigra_impex/exr.cxx:310]: (style) Redundant condition. It is safe to deallocate a NULL pointer [vigra_impex/hdr.cxx:116]: (style) Member variable not initialized in the constructor 'HDRCodecImpl::height' [vigra_impex/hdr.cxx:116]: (style) Member variable not initialized in the constructor 'HDRCodecImpl::width' [vigra_impex/jpeg.cxx:132]: (error) Class JPEGCodecImpl which is inherited by class JPEGDecoderImplBase does not have a virtual destructor [vigra_impex/jpeg.cxx:132]: (error) Class JPEGCodecImpl which is inherited by class JPEGEncoderImplBase does not have a virtual destructor [vigra_impex/rgbe.c:90]: (style) The scope of the variable f can be limited [vigra_impex/tiff.cxx:199]: (style) Member variable not initialized in the constructor 'TIFFCodecImpl::layers' [vigra_impex/viff.cxx:245]: (style) Function parameter 'index' is passed by value. It could be passed by reference instead. This doesn't seem to be memory leaks of some kind. Still I report it here. Harry 2009/10/7 Harry van der Wolf hvdw...@gmail.com To answer my own mail. :-) I just compiled cppcheck on OSX and did a standard run on the enblend trunk. It displays the following [./vigra_impex/jpeg.cxx:132]: (error) Class JPEGCodecImpl which is inherited by class JPEGDecoderImplBase does not have a virtual destructor [./vigra_impex/jpeg.cxx:132]: (error) Class JPEGCodecImpl which is inherited by class JPEGEncoderImplBase does not have a virtual destructor Then I did a full check. That resulted in exactly the same result as mentioned above. This is like giving a monkey an encyclopedia. He has no idea what to do with it. I don't even know if this is usefull. I hope some clever progammers (super monkeys?) know what to do with this. Harry 2009/10/7 Harry van der Wolf hvdw...@gmail.com 2009/10/7 Lukáš Jirkovský l.jirkov...@gmail.com I'm not Mac user (although I find it really cool but very expensive) but I may found solution for out of memory problem. I had a discussion about memory and I mentioned these fragmentation problems on OS. And get interesting advice – use TLSF [1]. It is said that it doesn't suffer from fragmentation much. Maybe some of you want to take a look at it. [1] http://rtportal.upv.es/rtmalloc/ Lukas Hi Lukáš, (You are the one person who I had hoped to react. I started to do tests myself yesterday evening but they take long). Both on 32bit windows, 32bit linux and 32bit OSX, or with the 32bit versions of enblend, we regularly see the error message of enblend crashing due to a memory allocation error. We see this both at hugin-ptx as well as in the bugtracker for large projects. We sometimes see it for enfuse in case of large projects which need to be fused as well. Untill now we blamed it on memory fragmentation, but maybe it's something else. George Row is one of the persons who encounters these errors on OSX. I have received a large set (2.5GB) from George in June, some time before my mac crashed. At that time I did some tests. These results are gone (no backup of test results).Yesterday I took Georges set from my big disk backup server and did some tests myself trying to stich a 12000X6000 (slightly bigger) panorama in hugin-2009.4.0-beta1. My question to you now is: You recently did some memory leak patching on the hugin trunk, using cppcheck, thereby finding some things in celeste. You reported this via http://groups.google.com/group/hugin-ptx/browse_thread/thread/e2b5b09e4706fb80 . Can you do the same for the enblend trunk? If you want to do this and you find the time for it in the near furure, be so kind to publish this on the hugin-ptx list. But please: don't feel obliged. If you don't have time or just do not want to do this: just say so. = If you don't want to do this, you can now stop reading. = If you want to do this or are at least interested: please continue reading. Below you will find my tests on OSX. It's done on the 2.5GB bracked village hotel project of George Row. Below the tail of the enblend error for a very recent 32bit Christoph Spiel build: enblend: info: loading next image: FoyleDays_M2_040007.tif 1/1 enblend: info: creating blend mask: 1/3enblend(11221) malloc: *** vm_allocate(size=580620288) failed (error code=3) enblend(11221) malloc: *** error: can't allocate region enblend(11221) malloc: *** set a breakpoint in szone_error to debug enblend: out of memory enblend: St9bad_alloc gnumake: *** [FoyleDays_M2_04.tif] Error 1 Below the tail of the error for the stable 32bit enblend 3.2 build (This to prove it's not a recent problem. It's already there in the 3.2 stable build).
[hugin-ptx] Re: possible memory leak in enblend enfuse? (was: hugin-mac-2009.4.0-Beta1 for download)
Hi Lukáš instead of changing the RAM on your PC, you could use a virtual machine like vmware or virtualbox. There, you can limit the resources at your will. Regards Stefan Peter --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups hugin and other free panoramic software group. A list of frequently asked questions is available at: http://wiki.panotools.org/Hugin_FAQ To post to this group, send email to hugin-ptx@googlegroups.com To unsubscribe from this group, send email to hugin-ptx-unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/hugin-ptx -~--~~~~--~~--~--~---
