Hi again, here you can download a minimal example:
https://drive.google.com/open?id=0BzmlODsuIIz0elpIcU1kdmpNTlE (download button is the arrow on the top right) In order to run it and get the memory_profiler output you have to install memory_profiler e.g. with pip3 install memory_profiler and run the file with python3 -m memory_profiler minimal_test.py If you just want to run the example without memory profiling and installing memory_profiler, you have to comment out or remove line 8 "@profile" Cheers, Martin On 07/13/2017 01:21 PM, Martin Fleck wrote: > > Hi Juan, hi Greg, > > quoting Greg: > > I think the main reason for the increased memory usage is that the > output type of the label function is int64 while your input is most > likely uint8. > > Indeed, this could be the complete problem already! For the analysis I > use a binary image - so only one bit per pixel. > > Greg: Regarding your PR and my analysis: My analysis using a 1.2GB > file stops due to memory problems already in > skimage.morphology.remove_small_objects() even if the major memory > blowup happens with skimage.morphology.label(). > So there are problems at multiple steps that hopefully can be improved. > > Quoting Juan: > > For example, what are the data types of the outputs in Matlab? > > the first steps of my analysis are to convert the 8 bit input image to > a meaningful binary image. The whole analysis is done on binary > images. So all inputs and outputs in Matlab are of Matlab Class "logical". > > I will provide you with a minimal example script and data for the > skimage case. > I will try to create equivalent memory inofrmation in Matlab. > > I'll both post it here as soon as I'm done with that. > > Thanks so far! > > Martin > > On 07/13/2017 03:05 AM, Juan Nunez-Iglesias wrote: >> Hi Martin, >> >> No one on this list wants to push you to more Matlab usage, believe >> me. ;) >> >> Do you think you could provide a script and sample data that we can >> use for troubleshooting? As Greg pointed out, the optimization >> approach *might* have to be data-type dependent. We could, for >> example, provide a dtype= keyword argument that would force the >> output to be of a particular, more memory-efficient type, if you know >> in advance how many objects you expect. >> >> If you can provide something similar to a memory profile, and >> diagnostic information, for your equivalent Matlab script, that would >> be really useful, so we know what we are aiming for. For example, >> what are the data types of the outputs in Matlab? >> >> Juan. >> >> On 13 Jul 2017, 9:59 AM +1000, Gregory Lee <grle...@gmail.com>, wrote: >>> Hi Martin, >>> >>> My problem my analysis uses much more memory than I expect. >>> I attached output from the memory_profiler package, with which I >>> tried >>> to keep track of the memory consumption of my analysis. >>> You can see that for an ~8MiB file that I used for testing, >>> skimage.measure.label needs to use 56MiB of memory, which >>> surprised me. >>> >>> >>> I haven't looked at it in much detail, but I did find what appear to >>> be some unnecessary copies in the top-level Cython routine called by >>> skimage.morphology.label. I opened a PR to try and avoid this here: >>> https://github.com/scikit-image/scikit-image/pull/2701 >>> <https://github.com/scikit-image/scikit-image/pull/2701> >>> >>> However, I think that PR is going to give a minor performance >>> improvement, but not help with memory use much if at all. I think >>> the main reason for the increased memory usage is that the output >>> type of the label function is int64 while your input is most likely >>> uint8. This means that the labels array requires 8 times the memory >>> usage of the uint8 input. I don't think there is much way around >>> that without making a version of the routines that allows specifying >>> a smaller integer dtype. >>> >>> - Greg >>> _______________________________________________ >>> scikit-image mailing list >>> scikit-image@python.org >>> https://mail.python.org/mailman/listinfo/scikit-image >> >> >> _______________________________________________ >> scikit-image mailing list >> scikit-image@python.org >> https://mail.python.org/mailman/listinfo/scikit-image > > > > _______________________________________________ > scikit-image mailing list > scikit-image@python.org > https://mail.python.org/mailman/listinfo/scikit-image
_______________________________________________ scikit-image mailing list scikit-image@python.org https://mail.python.org/mailman/listinfo/scikit-image
