> > Is the problematic case you cite rather common or the exceptional? > Perhaps the -wm option is a good place to set a lower memory limit in > this case, with the default being higher (instead of the other way > around)?
Unless we install a spy in gdalwarp that sends reports to a server, nobody will know I'm afraid ;-) > > Which is more common, 1) many small images being merged into a larger > one, 2) warping of large images or 3) merging of large+small images? > It would be nice as you suggest to have some benchmarks, would the 3 > cases I just mentioned be representative enough? What would be typical > sizes for each case? I can imagine, in simple warping scenarios, images of 1000x1000 and 10000x10000 pixels. And for mosaicing : merging images of size 1000x1000 in a 50000x50000 RGB mosaic. But this is really arbitrary... > Perhaps a compromise could be found, that > minimizes performance penalties on average. > The mentioned ticket has a proposed patch (by you) - perhaps it could > be used in some benchmarks? Actually, I only saved the patch that was posted in a volatile paste by Adam. The rationale behind it is looks reasonable. I think I tested it a bit at that time, but was probably too shy to apply it without more testing. > On a side note, I *think* that most users use default value and do not > use the -wm option unless they really look into the docs - therefore > it would be nice to have a better default. I agree that users will unlikely play with -wm option whose value might need some expertise to set it to an appropriate value. So I think that increasing the default value for -wm is reasonable if it helps in common scenarios. But I think benchmarks will show that we will need to fix the issue raised in the ticket first, probably with Adam's patch or something close to that idea. There are a few cases where choosing 1/4 of the RAM by default might cause issues : 1) If several gdalwarp processes are launched in parallel, they might exhaust the RAM and suffer from excessive swapping. But I think it is OK in that advanced scenario to tell them : "just set -wm to a lower value so that the sum of the -wm values is lower than XX% of the RAM". 2) 1/4 of the RAM isn't appropriate for 32 bit process. You can have a machine with 64 GB RAM, but a 32 bit process can only address 2 GB (generally, sometimes 3 GB). So we should probably not try to use more than 1 GB (or less) to avoid exhausting virtual memory space in that case. > > >> 2) Add a config option so that the default can be set for all > >> operations by the user (which overrides 1, but can be overridden for > >> each operation) > > > > Not sure to understand how option 2) if different from the -wm option and > > how it would be implemented ... > > Simply a config option, like there is for raster i/o cache (GDAL_CACHEMAX). > This would be a simple fix that allows the user to set the default for > his/her typical usage (and select another value with -wm when needed). > There might not be a need to change the default value then. I'm not convinced that adding a new config option that does the same thing as - wm will really help. Possibly, it would make things more confusing. > I have a small patch that implements that, should I create a ticket to > centralize this discussion and post the patch? > > cheers, > Etienne > > >> Any comments? > >> > >> regards, > >> Etienne > >> _______________________________________________ > >> gdal-dev mailing list > >> [email protected] > >> http://lists.osgeo.org/mailman/listinfo/gdal-dev _______________________________________________ gdal-dev mailing list [email protected] http://lists.osgeo.org/mailman/listinfo/gdal-dev
