I started watching this with htop and all the cpus are getting action.
There is a pattern where the number of threads spikes from about 162 up to
215 and the number of running threads spkies to about 50ish for a few
secounds, then the running threads drops to 2 for 5-10 seconds and repeats
this pattern. I thinking that the parent thread is spinning up a bunch of
workers, the finish, then the parent thread cycles through each of the
finished workers collecting the results and presumably write it to disk or
something. If it is writing to disk, there could be a huge potential
performance improvement by writing the output to memory if enough memory is
available which is clearly the case on this machine, then flushing the
memory to disk. The current process is only using 3 GB or memory when it
has 100 GB available to it and the system has 120GB.
On Wednesday, May 17, 2017 at 12:13:04 PM UTC-4, Stephen Woodbridge wrote:
>
> Hi, first I want to say the LSMS Segmentation is very cool and works
> nicely. I recently got access to a sever with 56 cores and 128GB of memory
> but I can't seem to get it to use more than 10-15 cores. I'm running the
> smoothing on an image approx 20000x20000 in size. The image is a gdal VRT
> file that combines 8 DOQQ images into a mosaic. It has 4 bands R, G, B, IR
> with each having Mask Flags: PER_DATASET (see below). I'm running this from
> a Python script like:
>
> def smoothing(fin, fout, foutpos, spatialr, ranger, rangeramp, thres,
> maxiter, ram):
> app = otbApplication.Registry.CreateApplication('MeanShiftSmoothing')
> app.SetParameterString('in', fin)
> app.SetParameterString('fout', fout)
> app.SetParameterString('foutpos', foutpos)
> app.SetParameterInt('spatialr', spatialr)
> app.SetParameterFloat('ranger', ranger)
> app.SetParameterFloat('rangeramp', rangeramp)
> app.SetParameterFloat('thres', thres)
> app.SetParameterInt('maxiter', maxiter)
> app.SetParameterInt('ram', ram)
> app.SetParameterInt('modesearch', 0)
> app.ExecuteAndWriteOutput()
>
> Where:
> spatialr: 24
> ranger: 36
> rangeramp: 0
> thres: 0.1
> maxiter: 100
> ram: 102400
>
> Any thoughts on how I can get this to utilize more of the processing power
> of this machine?
>
> -Steve
>
> woodbri@optane28:/u/ror/buildings/tmp$ otbcli_ReadImageInfo -in tmp-23081-
> areaofinterest.vrt
> 2017 May 17 15:36:04 : Application.logger (INFO)
> Image general information:
> Number of bands : 4
> No data flags : Not found
> Start index : [0,0]
> Size : [19933,19763]
> Origin : [-118.442,34.0035]
> Spacing : [9.83578e-06,-9.83578e-06]
> Estimated ground spacing (in meters): [0.90856,1.09369]
>
> Image acquisition information:
> Sensor :
> Image identification number:
> Image projection : GEOGCS["WGS 84",
> DATUM["WGS_1984",
> SPHEROID["WGS 84",6378137,298.257223563,
> AUTHORITY["EPSG","7030"]],
> AUTHORITY["EPSG","6326"]],
> PRIMEM["Greenwich",0],
> UNIT["degree",0.0174532925199433],
> AUTHORITY["EPSG","4326"]]
>
> Image default RGB composition:
> [R, G, B] = [0,1,2]
>
> Ground control points information:
> Number of GCPs = 0
> GCPs projection =
>
> Output parameters value:
> indexx: 0
> indexy: 0
> sizex: 19933
> sizey: 19763
> spacingx: 9.835776837e-06
> spacingy: -9.835776837e-06
> originx: -118.4418488
> originy: 34.00345612
> estimatedgroundspacingx: 0.9085595012
> estimatedgroundspacingy: 1.093693733
> numberbands: 4
> sensor:
> id:
> time:
> ullat: 0
> ullon: 0
> urlat: 0
> urlon: 0
> lrlat: 0
> lrlon: 0
> lllat: 0
> lllon: 0
> town:
> country:
> rgb.r: 0
> rgb.g: 1
> rgb.b: 2
> projectionref: GEOGCS["WGS 84",
> DATUM["WGS_1984",
> SPHEROID["WGS 84",6378137,298.257223563,
> AUTHORITY["EPSG","7030"]],
> AUTHORITY["EPSG","6326"]],
> PRIMEM["Greenwich",0],
> UNIT["degree",0.0174532925199433],
> AUTHORITY["EPSG","4326"]]
> keyword:
> gcp.count: 0
> gcp.proj:
> gcp.ids:
> gcp.info:
> gcp.imcoord:
> gcp.geocoord:
>
> woodbri@optane28:/u/ror/buildings/tmp$ gdalinfo tmp-23081-areaofinterest.
> vrt
> Driver: VRT/Virtual Raster
> Files: tmp-23081-areaofinterest.vrt
> /u/ror/buildings/tmp/tmp-23081-areaofinterest.vrt.vrt
> Size is 19933, 19763
> Coordinate System is:
> GEOGCS["WGS 84",
> DATUM["WGS_1984",
> SPHEROID["WGS 84",6378137,298.257223563,
> AUTHORITY["EPSG","7030"]],
> AUTHORITY["EPSG","6326"]],
> PRIMEM["Greenwich",0],
> UNIT["degree",0.0174532925199433],
> AUTHORITY["EPSG","4326"]]
> Origin = (-118.441851318576212,34.003461706049677)
> Pixel Size = (0.000009835776490,-0.000009835776490)
> Corner Coordinates:
> Upper Left (-118.4418513, 34.0034617) (118d26'30.66"W, 34d 0'12.46"N)
> Lower Left (-118.4418513, 33.8090773) (118d26'30.66"W, 33d48'32.68"N)
> Upper Right (-118.2457948, 34.0034617) (118d14'44.86"W, 34d 0'12.46"N)
> Lower Right (-118.2457948, 33.8090773) (118d14'44.86"W, 33d48'32.68"N)
> Center (-118.3438231, 33.9062695) (118d20'37.76"W, 33d54'22.57"N)
> Band 1 Block=128x128 Type=Byte, ColorInterp=Red
> Mask Flags: PER_DATASET
> Band 2 Block=128x128 Type=Byte, ColorInterp=Green
> Mask Flags: PER_DATASET
> Band 3 Block=128x128 Type=Byte, ColorInterp=Blue
> Mask Flags: PER_DATASET
> Band 4 Block=128x128 Type=Byte, ColorInterp=Gray
> Mask Flags: PER_DATASET
>
>
>
>
--
--
Check the OTB FAQ at
http://www.orfeo-toolbox.org/FAQ.html
You received this message because you are subscribed to the Google
Groups "otb-users" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/otb-users?hl=en
---
You received this message because you are subscribed to the Google Groups
"otb-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
