When you say that "Meaning that, for each input image to mosaic, the distance map will theoretically require the full image in memory to be processed." do you mean that, for every single input image, I need to load an array with the full size of the output mosaic?
On Wednesday, September 12, 2018 at 5:04:05 PM UTC+2, Rémi wrote: > > Your GeoTiff is compressed (PACKBITS) and has rectangular blocks of > 416x320. This problem could come from here has you have 2.5k images. > I would give a try to unflate all your images. To do it, use > gdal_translate -of "GTiff" -ot "Byte" input.tif output.tif > then re-run Mosaic ! > > Le mercredi 12 septembre 2018 16:56:42 UTC+2, Gus a écrit : >> >> Driver: GTiff/GeoTIFF >> Files: p_731_a.tif >> Size is 4145, 3189 >> Coordinate System is: >> PROJCS["WGS 84 / UTM zone 54S", >> GEOGCS["WGS 84", >> DATUM["WGS_1984", >> SPHEROID["WGS 84",6378137,298.257223563, >> AUTHORITY["EPSG","7030"]], >> AUTHORITY["EPSG","6326"]], >> PRIMEM["Greenwich",0, >> AUTHORITY["EPSG","8901"]], >> UNIT["degree",0.0174532925199433, >> AUTHORITY["EPSG","9122"]], >> AUTHORITY["EPSG","4326"]], >> PROJECTION["Transverse_Mercator"], >> PARAMETER["latitude_of_origin",0], >> PARAMETER["central_meridian",141], >> PARAMETER["scale_factor",0.9996], >> PARAMETER["false_easting",500000], >> PARAMETER["false_northing",10000000], >> UNIT["metre",1, >> AUTHORITY["EPSG","9001"]], >> AXIS["Easting",EAST], >> AXIS["Northing",NORTH], >> AUTHORITY["EPSG","32754"]] >> Origin = (271914.159999999974389,6123214.030000000260770) >> Pixel Size = (1.000000000000000,-1.000000000000000) >> Metadata: >> AREA_OR_POINT=Area >> Image Structure Metadata: >> COMPRESSION=PACKBITS >> INTERLEAVE=PIXEL >> Corner Coordinates: >> Upper Left ( 271914.160, 6123214.030) (138d30' 1.82"E, 35d 0'28.82"S) >> Lower Left ( 271914.160, 6120025.030) (138d29'58.67"E, 35d 2'12.24"S) >> Upper Right ( 276059.160, 6123214.030) (138d32'45.20"E, 35d 0'32.15"S) >> Lower Right ( 276059.160, 6120025.030) (138d32'42.11"E, 35d 2'15.58"S) >> Center ( 273986.660, 6121619.530) (138d31'21.95"E, 35d 1'22.21"S) >> Band 1 Block=416x320 Type=Byte, ColorInterp=Cyan >> Band 2 Block=416x320 Type=Byte, ColorInterp=Magenta >> Band 3 Block=416x320 Type=Byte, ColorInterp=Yellow >> >> >> On Wednesday, September 12, 2018 at 4:48:14 PM UTC+2, Rémi wrote: >>> >>> The color space is l-alpha-beta but I don't think it's related to this >>> strange behavior. >>> >>> I suspect that your images are in a non-streamable format. Meaning, you >>> can't read them block by block, instead you are forced to read the entire >>> file. Can you paste a gdalinfo of 1 of your file (feel free to modify >>> coordinates and stuff) it's just to check the image file format. >>> If I'm right, you can try to convert all your files in GeoTiff format >>> (using otbcli_Convert with an output file extension equal to ".tif", then >>> run the Mosaic application over the fresh .tif files and it will work fine >>> >>> Rémi >>> >>> Le mercredi 12 septembre 2018 16:32:43 UTC+2, Gus a écrit : >>>> >>>> Trying to reduce the RAM value seems to have no effect on memory >>>> footprint. If I set it to 1, it grows a lot at first, then the process >>>> seems to progress very slowly. If I set it to 10, it gows a lot at first, >>>> then it does progress increasingly ocuppying memory. It seems to be >>>> independent of your colour space transform, whatever method you choose, >>>> band per band or (I guess you use cieLab?). I'm testing the no >>>> harmonization options that you told (that I had already tried in windows). >>>> It's writing the tiff and seems to be ok but, although it probably will >>>> finish, it's taking a (while slower) progressively increasing amount of >>>> ram. >>>> I tested ram parameter in otbcli_HaralickTextureExtraction and it seems >>>> to correctly make use of chunks within parameter. >>>> Just in case, I'm using imagery of this kind, 8bit unsigned integer >>>> where I use 0 as no value (it worked ok for small sets). >>>> >>>> >>>> On Wednesday, September 12, 2018 at 2:07:45 PM UTC+2, Rémi wrote: >>>>> >>>>> From this log, it seems like the problem occurs during statistics >>>>> computation. >>>>> >>>>> To be sure of this, can you please check that the following commands >>>>> work fine: >>>>> >>>>> - otbcli_Mosaic -il /media/gus/Gus/seagrass/3_band/* -out >>>>> /media/gus/Gus/seagrass/processed_data/r2_simple_noharmo.tif uint8 >>>>> -ram 128 >>>>> - otbcli_Mosaic -il /media/gus/Gus/seagrass/3_band/* -comp.feather >>>>> large -out >>>>> /media/gus/Gus/seagrass/processed_data/r2_feather_noharmo.tif >>>>> uint8 -tmpdir /media/gus/Gus/seagrass/processed_data/temp -ram 128 >>>>> >>>>> And can you please tell me if these commands are successful or not: >>>>> >>>>> - otbcli_Mosaic -il /media/gus/Gus/seagrass/3_band/* -harmo.method >>>>> band -harmo.cost musig -out >>>>> /media/gus/Gus/seagrass/processed_data/r2_harmo-band.tif uint8 -ram 128 >>>>> - otbcli_Mosaic -il /media/gus/Gus/seagrass/3_band/* -harmo.method >>>>> rgb -harmo.cost musig -out >>>>> /media/gus/Gus/seagrass/processed_data/r2_harmo-rgb.tif uint8 -ram 128 >>>>> >>>>> Now, you can try to set a very small RAM value and tell me what's >>>>> happening. Set -ram 1 for instance... >>>>> >>>>> Le mercredi 12 septembre 2018 11:42:19 UTC+2, Gus a écrit : >>>>>> >>>>>> I tried again last night. OTB 6.6.0 packaged linux version. >>>>>> Command line: ./otbcli_Mosaic -il /media/gus/Gus/seagrass/3_band/* >>>>>> -comp.feather large -harmo.method rgb -harmo.cost musig -out >>>>>> /media/gus/Gus/seagrass/processed_data/r2.tif uint8 -tmpdir >>>>>> /media/gus/Gus/seagrass/processed_data/temp -ram 128 -progress 1 >>>>>> >>>>>> Last lines of output: >>>>>> 2018-09-12 03:29:30 (INFO): Computing statistics in a decorrelated >>>>>> colors space suitable for true-colors >>>>>> 2018-09-12 03:29:44 (INFO): Estimated memory for full processing: >>>>>> 3.15661e+06MB (avail.: 128 MB), optimal image partitioning: 24662 blocks >>>>>> 2018-09-12 03:29:44 (INFO): Estimation will be performed in 18639 >>>>>> blocks of 832x320 pixels >>>>>> >>>>>> Anyway, after start, used RAM begins to slowly increase until it >>>>>> occupied my swap space and froze the computer during the night. >>>>>> >>>>>> [image: Screenshot at 2018-09-12 03-30-10.png] >>>>>> >>>>>> >>>>>> >>>>>> On Sunday, September 9, 2018 at 5:23:19 PM UTC+2, Rémi wrote: >>>>>>> >>>>>>> Hello Gus, >>>>>>> >>>>>>> To investigate a bit I would like to know if you can copy/paste: >>>>>>> -your command line >>>>>>> -the full log >>>>>>> (In a text file, or on pastebin) >>>>>>> >>>>>>> What version of OTB, and what OS do you have? >>>>>>> Thanks, >>>>>>> >>>>>>> Rémi >>>>>>> >>>>>>> Le samedi 8 septembre 2018 18:07:56 UTC+2, Gus a écrit : >>>>>>>> >>>>>>>> Thanks, Rémi >>>>>>>> >>>>>>>> Your method works fairly well, the results are really good and >>>>>>>> visually pleasant. In fact, I was able to use you app (from gui) for >>>>>>>> small >>>>>>>> subsets of my data, which is a set of aerial pictures of the sea >>>>>>>> surface >>>>>>>> from which the sunglint has been removed. >>>>>>>> >>>>>>>> [image: 2dc4d9d7-25da-4254-9cb5-31536801f461.jpg][image: >>>>>>>> 54ed0802-e492-438f-9f4d-8dea3593a2b6.jpg] >>>>>>>> >>>>>>>> But, when I feed your program with the complete set (about 2500 >>>>>>>> pictures, 30 Mb each approximately, 8 bit colour) it fails. Last thing >>>>>>>> I >>>>>>>> can check is that the temporary distance maps have been correctly >>>>>>>> written, >>>>>>>> and that's the last console output. while if Iskip the large >>>>>>>> feathering >>>>>>>> step, it seems to work ok, as in next image using slim blending mode, >>>>>>>> but >>>>>>>> as for large blending, it fails... >>>>>>>> >>>>>>>> [image: sea_1PNG.PNG] >>>>>>>> >>>>>>>> On Saturday, September 8, 2018 at 10:57:53 AM UTC+2, Rémi wrote: >>>>>>>>> >>>>>>>>> Hello Gus, >>>>>>>>> >>>>>>>>> The RAM parameter deals only with the memory footprint during >>>>>>>>> mosaic compositing and statistics computation, both streamable >>>>>>>>> pipelines >>>>>>>>> (no limitation on images size). >>>>>>>>> However, the distance map calculation (which is a temporary file, >>>>>>>>> then used as an input of the streamed mosaicing compositing pipeline, >>>>>>>>> just >>>>>>>>> for feathering) is not a streamable process. Meaning that, for each >>>>>>>>> input >>>>>>>>> image to mosaic, the distance map will theoretically require the full >>>>>>>>> image >>>>>>>>> in memory to be processed. That's why there is the distancemap.sr >>>>>>>>> (sampling ratio) parameter. This enables the subsampling of the input >>>>>>>>> image, to compute the distance map with a smaller footprint. See >>>>>>>>> README.md, >>>>>>>>> section "Performance tuning: Distance map images sampling ratio". >>>>>>>>> This is a current limitation of the application, I will add this >>>>>>>>> explicitly in the application description. >>>>>>>>> >>>>>>>>> Your problem could also come from the fact that your input images >>>>>>>>> have not the same projection reference (and leading to a near >>>>>>>>> infinite >>>>>>>>> image, e.g. if you mix degrees and meters units, or with wrong >>>>>>>>> coordinate >>>>>>>>> reference system), check this. >>>>>>>>> >>>>>>>>> Hope that helps, >>>>>>>>> >>>>>>>>> Rémi >>>>>>>>> >>>>>>>>> Le vendredi 7 septembre 2018 01:44:23 UTC+2, Gus a écrit : >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Hello everyone. >>>>>>>>>> >>>>>>>>>> I don't clearly understand the process followed by the remote >>>>>>>>>> module "mosaic". The memory occupied during some phase after >>>>>>>>>> distance maps >>>>>>>>>> calculation for large blending with lab color space harmonization >>>>>>>>>> grows a >>>>>>>>>> lot, so I obtain a huge page file and my computer reboots. I checked >>>>>>>>>> the >>>>>>>>>> paper and peeked into the code a bit, but I'm not familiar with the >>>>>>>>>> libraries. Is there any way of estimating the ram needed for the >>>>>>>>>> process? >>>>>>>>>> Is there a way of overcoming this issue? (probably I will process it >>>>>>>>>> in >>>>>>>>>> chunks anyway but that would simplify things quite a lot). >>>>>>>>>> >>>>>>>>>> Cheers >>>>>>>>>> >>>>>>>>>> Gus >>>>>>>>>> >>>>>>>>> -- -- Check the OTB FAQ at http://www.orfeo-toolbox.org/FAQ.html You received this message because you are subscribed to the Google Groups "otb-users" group. To post to this group, send email to otb-users@googlegroups.com To unsubscribe from this group, send email to otb-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/otb-users?hl=en --- You received this message because you are subscribed to the Google Groups "otb-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to otb-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
