Re: [otb-users] How to change default RAM limit?

[Leonardo Hardtke]

Sure it helps!
Thank you very much.

Cheers.

On Friday, 18 May 2018 02:32:27 UTC+10, Julien Michel wrote:
>
> GDAL_CACHEMAX is the size of gdal cache to store blocks it already reads 
> for further use. Depending on you processing chain, it may or may not be 
> relevant.
>
>  
>
> You can try setting the chunk size yourself :
>
>  
>
> “/tmp/example1.tif?:sizemode=nbsplits:sizevalue=1”
>
>  
>
> Regards,
>
>  
>
> Julien
>
>  
>
>  
>
>  
>
>  
>
>  
>
> *De :* otb-...@googlegroups.com  [mailto:
> otb-...@googlegroups.com ] *De la part de* Leonardo Hardtke
> *Envoyé :* mercredi 16 mai 2018 12:42
> *À :* otb-users 
> *Objet :* Re: [otb-users] How to change default RAM limit?
>
>  
>
> Thank you very much for the answer. I have a lot to learn!
>
>  
>
> Yes, my question was under the assumption that small chunks would make the 
> program run slower. :). I tried setting the OTB_MAX_RAM_HINT to 4000, and 
> could see an improvement (~50 %)!
>
>  
>
> leo@leo-ThinkPad-E480:~/devel/otbapps/image_segmentation/build$ export 
> OTB_MAX_RAM_HINT=128
> leo@leo-ThinkPad-E480:~/devel/otbapps/image_segmentation/build$ time 
> ./ConnectedThresholdImageFilter_NC ../test_img/fire_test.tif 
> /tmp/example1.tif 2073 2372 200 1500 2072 2371
> 2018-05-16 20:28:59 (INFO): No kwl metadata found in file 
> ../test_img/fire_test.tif
> 2018-05-16 20:28:59 (INFO): Default RAM limit for OTB is 128 MB
> 2018-05-16 20:28:59 (INFO): GDAL maximum cache size is 393 MB
> 2018-05-16 20:28:59 (INFO): OTB will use at most 8 threads
> 2018-05-16 20:28:59 (INFO): Estimated memory for full processing: 
> 3.94151e+06MB (avail.: 128 MB), optimal image partitioning: 30794 blocks
> 2018-05-16 20:28:59 (INFO): File /tmp/example1.tif will be written in 
> 32396 blocks of 43x43 pixels
>
> real0m20.951s
> user1m3.094s
> sys 0m9.747s
> leo@leo-ThinkPad-E480:~/devel/otbapps/image_segmentation/build$ export 
> OTB_MAX_RAM_HINT=4000
> leo@leo-ThinkPad-E480:~/devel/otbapps/image_segmentation/build$ time 
> ./ConnectedThresholdImageFilter_NC ../test_img/fire_test.tif 
> /tmp/example1.tif 2073 2372 200 1500 2072 2371
> 2018-05-16 20:29:35 (INFO): No kwl metadata found in file 
> ../test_img/fire_test.tif
> 2018-05-16 20:29:35 (INFO): Default RAM limit for OTB is 4000 MB
> 2018-05-16 20:29:35 (INFO): GDAL maximum cache size is 393 MB
> 2018-05-16 20:29:35 (INFO): OTB will use at most 8 threads
> 2018-05-16 20:29:35 (INFO): Estimated memory for full processing: 
> 3.94151e+06MB (avail.: 4000 MB), optimal image partitioning: 986 blocks
> 2018-05-16 20:29:35 (INFO): File /tmp/example1.tif will be written in 1830 
> blocks of 256x128 pixels
>
> real0m13.211s
> user0m59.665s
> sys 0m1.873s
>
>  
>
> I was checking htop and the process did not take 4000 mb of ram but only 
> 1200 mb. Also, what does GDAL cache size do?
>
>  
>
> The logs are very useful! Great addition! I am actually using the master 
> version (self compiled) on ubuntu 18.04, because I tried the superbuild and 
> it failed. I can try to reproduce the failure and let you know what it was.
>
>  
>
> As I mentioned, the code I am using is 
> https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/blob/develop/Examples/Segmentation/ConnectedThresholdImageFilter.cxx
>
>  
>
> Do you think it could run faster by tweaking other parameters?
>
>  
>
> The plan is to replace an old IDL script that is used in my workplace to 
> aid in the mapping of bushfires. Ideally I would like to implement it as a 
> QGIS plugin, but the to convince my supervisor it needs to run as fast as 
> the IDL code, which takes about 4 second to run.
>
>  
>
> Thanks again!
>
>  
>
>  
>
>  
>
>  
>
>  
>
>
> On Wednesday, May 16, 2018 at 4:18:53 PM UTC+10, Julien Michel wrote:
>
> Hi,
>
>  
>
> You can add a RAM parameter to your application that will allow you to 
> control this (AddRamParameter()). You can also set the OTB_MAX_RAM_HINT 
> environment variable (this will be documented in cookbook for next release 
> [1].
>
>  
>
> Looking at your logs (btw those logs are pretty new in OTB, congrats for 
> using a bleeding edge version ;)), OTB is splitting the data into very 
> small chunks (one line at a time). This is likely to be slower than larger 
> chunks. To get larger chunks, you can increase the RAM parameter, but you 
> can even set the chunk size yourself using extended filenames [1].
>
>  
>
> How long is the inner pipeline of your application ? If OTB decides that 
> it can only process one line at a time with 128 Mb, I expect the inner 
> pipeline to be quite long (or include non-streamed filters).
>
>  
>
> Hope that helps,
>
>  
>
> Julien
>
>  
>
> [1] 
> https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/blob/develop/Documentation/Cookbook/rst/AdvancedUse.rst
>
>  
>
> *De :* otb-...@googlegroups.com [mailto:otb-...@googlegroups.com] *De la 
> part de* Leonardo Hardtke
> *Envoyé :* mercredi 16 mai 2018 06:47
> *À :* otb-users 
> *Objet :* 

RE: [otb-users] How to change default RAM limit?

[Michel Julien]

GDAL_CACHEMAX is the size of gdal cache to store blocks it already reads for 
further use. Depending on you processing chain, it may or may not be relevant.

You can try setting the chunk size yourself :

“/tmp/example1.tif?:sizemode=nbsplits:sizevalue=1”

Regards,

Julien





De : otb-users@googlegroups.com [mailto:otb-users@googlegroups.com] De la part 
de Leonardo Hardtke
Envoyé : mercredi 16 mai 2018 12:42
À : otb-users 
Objet : Re: [otb-users] How to change default RAM limit?

Thank you very much for the answer. I have a lot to learn!

Yes, my question was under the assumption that small chunks would make the 
program run slower. :). I tried setting the OTB_MAX_RAM_HINT to 4000, and could 
see an improvement (~50 %)!

leo@leo-ThinkPad-E480:~/devel/otbapps/image_segmentation/build$ export 
OTB_MAX_RAM_HINT=128
leo@leo-ThinkPad-E480:~/devel/otbapps/image_segmentation/build$ time 
./ConnectedThresholdImageFilter_NC ../test_img/fire_test.tif /tmp/example1.tif 
2073 2372 200 1500 2072 2371
2018-05-16 20:28:59 (INFO): No kwl metadata found in file 
../test_img/fire_test.tif
2018-05-16 20:28:59 (INFO): Default RAM limit for OTB is 128 MB
2018-05-16 20:28:59 (INFO): GDAL maximum cache size is 393 MB
2018-05-16 20:28:59 (INFO): OTB will use at most 8 threads
2018-05-16 20:28:59 (INFO): Estimated memory for full processing: 3.94151e+06MB 
(avail.: 128 MB), optimal image partitioning: 30794 blocks
2018-05-16 20:28:59 (INFO): File /tmp/example1.tif will be written in 32396 
blocks of 43x43 pixels

real0m20.951s
user1m3.094s
sys 0m9.747s
leo@leo-ThinkPad-E480:~/devel/otbapps/image_segmentation/build$ export 
OTB_MAX_RAM_HINT=4000
leo@leo-ThinkPad-E480:~/devel/otbapps/image_segmentation/build$ time 
./ConnectedThresholdImageFilter_NC ../test_img/fire_test.tif /tmp/example1.tif 
2073 2372 200 1500 2072 2371
2018-05-16 20:29:35 (INFO): No kwl metadata found in file 
../test_img/fire_test.tif
2018-05-16 20:29:35 (INFO): Default RAM limit for OTB is 4000 MB
2018-05-16 20:29:35 (INFO): GDAL maximum cache size is 393 MB
2018-05-16 20:29:35 (INFO): OTB will use at most 8 threads
2018-05-16 20:29:35 (INFO): Estimated memory for full processing: 3.94151e+06MB 
(avail.: 4000 MB), optimal image partitioning: 986 blocks
2018-05-16 20:29:35 (INFO): File /tmp/example1.tif will be written in 1830 
blocks of 256x128 pixels

real0m13.211s
user0m59.665s
sys 0m1.873s

I was checking htop and the process did not take 4000 mb of ram but only 1200 
mb. Also, what does GDAL cache size do?

The logs are very useful! Great addition! I am actually using the master 
version (self compiled) on ubuntu 18.04, because I tried the superbuild and it 
failed. I can try to reproduce the failure and let you know what it was.

As I mentioned, the code I am using is 
https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/blob/develop/Examples/Segmentation/ConnectedThresholdImageFilter.cxx

Do you think it could run faster by tweaking other parameters?

The plan is to replace an old IDL script that is used in my workplace to aid in 
the mapping of bushfires. Ideally I would like to implement it as a QGIS 
plugin, but the to convince my supervisor it needs to run as fast as the IDL 
code, which takes about 4 second to run.

Thanks again!






On Wednesday, May 16, 2018 at 4:18:53 PM UTC+10, Julien Michel wrote:
Hi,

You can add a RAM parameter to your application that will allow you to control 
this (AddRamParameter()). You can also set the OTB_MAX_RAM_HINT environment 
variable (this will be documented in cookbook for next release [1].

Looking at your logs (btw those logs are pretty new in OTB, congrats for using 
a bleeding edge version ;)), OTB is splitting the data into very small chunks 
(one line at a time). This is likely to be slower than larger chunks. To get 
larger chunks, you can increase the RAM parameter, but you can even set the 
chunk size yourself using extended filenames [1].

How long is the inner pipeline of your application ? If OTB decides that it can 
only process one line at a time with 128 Mb, I expect the inner pipeline to be 
quite long (or include non-streamed filters).

Hope that helps,

Julien

[1] 
https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/blob/develop/Documentation/Cookbook/rst/AdvancedUse.rst

De : otb-...@googlegroups.com 
[mailto:otb-...@googlegroups.com] De la part de Leonardo Hardtke
Envoyé : mercredi 16 mai 2018 06:47
À : otb-users 
Objet : [otb-users] How to change default RAM limit?

Dear all,
I am starting a new project and was investigating the likelihood of doing it in 
OTB. I used otb apps from python many times before but never tried to write my 
own apps. Part of my application will be based on the code 
/Examples/Segmentation/ConnectedThresholdImageFilter.cxx

It compiles and runs OK, but it is a little bit slow (8 secs) for what I need 
(interactive application with Landsat images [7591, 7731]). The info shows it 
uses only 

[otb-users] Apply geometric transformation based on pithc, roll and yaw

[Agustin Lobo]

Hi!
Is it possible to use OTB to apply a geometric transformation based on
pithc, roll and yaw?
Thanks

-- 
-- 
Check the OTB FAQ at
http://www.orfeo-toolbox.org/FAQ.html

You received this message because you are subscribed to the Google
Groups "otb-users" group.
To post to this group, send email to otb-users@googlegroups.com
To unsubscribe from this group, send email to
otb-users+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/otb-users?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"otb-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to otb-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.