Your GeoTiff is compressed (PACKBITS) and has rectangular blocks of 416x320. This problem could come from here has you have 2.5k images. I would give a try to unflate all your images. To do it, use gdal_translate -of "GTiff" -ot "Byte" input.tif output.tif then re-run Mosaic !
Le mercredi 12 septembre 2018 16:56:42 UTC+2, Gus a écrit : > > Driver: GTiff/GeoTIFF > Files: p_731_a.tif > Size is 4145, 3189 > Coordinate System is: > PROJCS["WGS 84 / UTM zone 54S", > GEOGCS["WGS 84", > DATUM["WGS_1984", > SPHEROID["WGS 84",6378137,298.257223563, > AUTHORITY["EPSG","7030"]], > AUTHORITY["EPSG","6326"]], > PRIMEM["Greenwich",0, > AUTHORITY["EPSG","8901"]], > UNIT["degree",0.0174532925199433, > AUTHORITY["EPSG","9122"]], > AUTHORITY["EPSG","4326"]], > PROJECTION["Transverse_Mercator"], > PARAMETER["latitude_of_origin",0], > PARAMETER["central_meridian",141], > PARAMETER["scale_factor",0.9996], > PARAMETER["false_easting",500000], > PARAMETER["false_northing",10000000], > UNIT["metre",1, > AUTHORITY["EPSG","9001"]], > AXIS["Easting",EAST], > AXIS["Northing",NORTH], > AUTHORITY["EPSG","32754"]] > Origin = (271914.159999999974389,6123214.030000000260770) > Pixel Size = (1.000000000000000,-1.000000000000000) > Metadata: > AREA_OR_POINT=Area > Image Structure Metadata: > COMPRESSION=PACKBITS > INTERLEAVE=PIXEL > Corner Coordinates: > Upper Left ( 271914.160, 6123214.030) (138d30' 1.82"E, 35d 0'28.82"S) > Lower Left ( 271914.160, 6120025.030) (138d29'58.67"E, 35d 2'12.24"S) > Upper Right ( 276059.160, 6123214.030) (138d32'45.20"E, 35d 0'32.15"S) > Lower Right ( 276059.160, 6120025.030) (138d32'42.11"E, 35d 2'15.58"S) > Center ( 273986.660, 6121619.530) (138d31'21.95"E, 35d 1'22.21"S) > Band 1 Block=416x320 Type=Byte, ColorInterp=Cyan > Band 2 Block=416x320 Type=Byte, ColorInterp=Magenta > Band 3 Block=416x320 Type=Byte, ColorInterp=Yellow > > > On Wednesday, September 12, 2018 at 4:48:14 PM UTC+2, Rémi wrote: >> >> The color space is l-alpha-beta but I don't think it's related to this >> strange behavior. >> >> I suspect that your images are in a non-streamable format. Meaning, you >> can't read them block by block, instead you are forced to read the entire >> file. Can you paste a gdalinfo of 1 of your file (feel free to modify >> coordinates and stuff) it's just to check the image file format. >> If I'm right, you can try to convert all your files in GeoTiff format >> (using otbcli_Convert with an output file extension equal to ".tif", then >> run the Mosaic application over the fresh .tif files and it will work fine >> >> Rémi >> >> Le mercredi 12 septembre 2018 16:32:43 UTC+2, Gus a écrit : >>> >>> Trying to reduce the RAM value seems to have no effect on memory >>> footprint. If I set it to 1, it grows a lot at first, then the process >>> seems to progress very slowly. If I set it to 10, it gows a lot at first, >>> then it does progress increasingly ocuppying memory. It seems to be >>> independent of your colour space transform, whatever method you choose, >>> band per band or (I guess you use cieLab?). I'm testing the no >>> harmonization options that you told (that I had already tried in windows). >>> It's writing the tiff and seems to be ok but, although it probably will >>> finish, it's taking a (while slower) progressively increasing amount of ram. >>> I tested ram parameter in otbcli_HaralickTextureExtraction and it seems >>> to correctly make use of chunks within parameter. >>> Just in case, I'm using imagery of this kind, 8bit unsigned integer >>> where I use 0 as no value (it worked ok for small sets). >>> >>> >>> On Wednesday, September 12, 2018 at 2:07:45 PM UTC+2, Rémi wrote: >>>> >>>> From this log, it seems like the problem occurs during statistics >>>> computation. >>>> >>>> To be sure of this, can you please check that the following commands >>>> work fine: >>>> >>>> - otbcli_Mosaic -il /media/gus/Gus/seagrass/3_band/* -out >>>> /media/gus/Gus/seagrass/processed_data/r2_simple_noharmo.tif uint8 -ram >>>> 128 >>>> - otbcli_Mosaic -il /media/gus/Gus/seagrass/3_band/* -comp.feather >>>> large -out >>>> /media/gus/Gus/seagrass/processed_data/r2_feather_noharmo.tif >>>> uint8 -tmpdir /media/gus/Gus/seagrass/processed_data/temp -ram 128 >>>> >>>> And can you please tell me if these commands are successful or not: >>>> >>>> - otbcli_Mosaic -il /media/gus/Gus/seagrass/3_band/* -harmo.method >>>> band -harmo.cost musig -out >>>> /media/gus/Gus/seagrass/processed_data/r2_harmo-band.tif uint8 -ram 128 >>>> - otbcli_Mosaic -il /media/gus/Gus/seagrass/3_band/* -harmo.method >>>> rgb -harmo.cost musig -out >>>> /media/gus/Gus/seagrass/processed_data/r2_harmo-rgb.tif uint8 -ram 128 >>>> >>>> Now, you can try to set a very small RAM value and tell me what's >>>> happening. Set -ram 1 for instance... >>>> >>>> Le mercredi 12 septembre 2018 11:42:19 UTC+2, Gus a écrit : >>>>> >>>>> I tried again last night. OTB 6.6.0 packaged linux version. >>>>> Command line: ./otbcli_Mosaic -il /media/gus/Gus/seagrass/3_band/* >>>>> -comp.feather large -harmo.method rgb -harmo.cost musig -out >>>>> /media/gus/Gus/seagrass/processed_data/r2.tif uint8 -tmpdir >>>>> /media/gus/Gus/seagrass/processed_data/temp -ram 128 -progress 1 >>>>> >>>>> Last lines of output: >>>>> 2018-09-12 03:29:30 (INFO): Computing statistics in a decorrelated >>>>> colors space suitable for true-colors >>>>> 2018-09-12 03:29:44 (INFO): Estimated memory for full processing: >>>>> 3.15661e+06MB (avail.: 128 MB), optimal image partitioning: 24662 blocks >>>>> 2018-09-12 03:29:44 (INFO): Estimation will be performed in 18639 >>>>> blocks of 832x320 pixels >>>>> >>>>> Anyway, after start, used RAM begins to slowly increase until it >>>>> occupied my swap space and froze the computer during the night. >>>>> >>>>> [image: Screenshot at 2018-09-12 03-30-10.png] >>>>> >>>>> >>>>> >>>>> On Sunday, September 9, 2018 at 5:23:19 PM UTC+2, Rémi wrote: >>>>>> >>>>>> Hello Gus, >>>>>> >>>>>> To investigate a bit I would like to know if you can copy/paste: >>>>>> -your command line >>>>>> -the full log >>>>>> (In a text file, or on pastebin) >>>>>> >>>>>> What version of OTB, and what OS do you have? >>>>>> Thanks, >>>>>> >>>>>> Rémi >>>>>> >>>>>> Le samedi 8 septembre 2018 18:07:56 UTC+2, Gus a écrit : >>>>>>> >>>>>>> Thanks, Rémi >>>>>>> >>>>>>> Your method works fairly well, the results are really good and >>>>>>> visually pleasant. In fact, I was able to use you app (from gui) for >>>>>>> small >>>>>>> subsets of my data, which is a set of aerial pictures of the sea >>>>>>> surface >>>>>>> from which the sunglint has been removed. >>>>>>> >>>>>>> [image: 2dc4d9d7-25da-4254-9cb5-31536801f461.jpg][image: >>>>>>> 54ed0802-e492-438f-9f4d-8dea3593a2b6.jpg] >>>>>>> >>>>>>> But, when I feed your program with the complete set (about 2500 >>>>>>> pictures, 30 Mb each approximately, 8 bit colour) it fails. Last thing >>>>>>> I >>>>>>> can check is that the temporary distance maps have been correctly >>>>>>> written, >>>>>>> and that's the last console output. while if Iskip the large feathering >>>>>>> step, it seems to work ok, as in next image using slim blending mode, >>>>>>> but >>>>>>> as for large blending, it fails... >>>>>>> >>>>>>> [image: sea_1PNG.PNG] >>>>>>> >>>>>>> On Saturday, September 8, 2018 at 10:57:53 AM UTC+2, Rémi wrote: >>>>>>>> >>>>>>>> Hello Gus, >>>>>>>> >>>>>>>> The RAM parameter deals only with the memory footprint during >>>>>>>> mosaic compositing and statistics computation, both streamable >>>>>>>> pipelines >>>>>>>> (no limitation on images size). >>>>>>>> However, the distance map calculation (which is a temporary file, >>>>>>>> then used as an input of the streamed mosaicing compositing pipeline, >>>>>>>> just >>>>>>>> for feathering) is not a streamable process. Meaning that, for each >>>>>>>> input >>>>>>>> image to mosaic, the distance map will theoretically require the full >>>>>>>> image >>>>>>>> in memory to be processed. That's why there is the distancemap.sr >>>>>>>> (sampling ratio) parameter. This enables the subsampling of the input >>>>>>>> image, to compute the distance map with a smaller footprint. See >>>>>>>> README.md, >>>>>>>> section "Performance tuning: Distance map images sampling ratio". >>>>>>>> This is a current limitation of the application, I will add this >>>>>>>> explicitly in the application description. >>>>>>>> >>>>>>>> Your problem could also come from the fact that your input images >>>>>>>> have not the same projection reference (and leading to a near infinite >>>>>>>> image, e.g. if you mix degrees and meters units, or with wrong >>>>>>>> coordinate >>>>>>>> reference system), check this. >>>>>>>> >>>>>>>> Hope that helps, >>>>>>>> >>>>>>>> Rémi >>>>>>>> >>>>>>>> Le vendredi 7 septembre 2018 01:44:23 UTC+2, Gus a écrit : >>>>>>>>> >>>>>>>>> >>>>>>>>> Hello everyone. >>>>>>>>> >>>>>>>>> I don't clearly understand the process followed by the remote >>>>>>>>> module "mosaic". The memory occupied during some phase after distance >>>>>>>>> maps >>>>>>>>> calculation for large blending with lab color space harmonization >>>>>>>>> grows a >>>>>>>>> lot, so I obtain a huge page file and my computer reboots. I checked >>>>>>>>> the >>>>>>>>> paper and peeked into the code a bit, but I'm not familiar with the >>>>>>>>> libraries. Is there any way of estimating the ram needed for the >>>>>>>>> process? >>>>>>>>> Is there a way of overcoming this issue? (probably I will process it >>>>>>>>> in >>>>>>>>> chunks anyway but that would simplify things quite a lot). >>>>>>>>> >>>>>>>>> Cheers >>>>>>>>> >>>>>>>>> Gus >>>>>>>>> >>>>>>>> -- -- Check the OTB FAQ at http://www.orfeo-toolbox.org/FAQ.html You received this message because you are subscribed to the Google Groups "otb-users" group. To post to this group, send email to otb-users@googlegroups.com To unsubscribe from this group, send email to otb-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/otb-users?hl=en --- You received this message because you are subscribed to the Google Groups "otb-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to otb-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
