Hi Luc,
Luc Claustres wrote:
Hi,
After different positive results using VPB to generate world
databases (at 450m/150m imagery resolution and 300m/200m DEM
resolution), we have tried something more ambitious. Our input data
are now :
- a world imagery cover at 0.5s (~15m) : 72 ECW files ~ 94 GB - a
world DEM cover at 3s (~90m) : 871 GeoTiff files ~ 60 GB
Here is the command line script used under Windows :
Code:
SET PATH=%PATH%;.\VPB SET GDAL_DATA=.\VPB\data
vpbmaster^ -t H:\World_0s5_ECW^ -d H:\World_3s_TIF^ --splits 6 10^
--geocentric^ --no-mip-mapping^ --terrain^ --RGB-24^ -O
"compressImageData JPEG_QUALITY 75"^ -o World15m\world15m.ive
We read data from external USB 2 drives but write output to the local
HD. Depending on the machine and whether we use DEM or not, the
approximated time to completion is huge (10-60 days), but we can go
along with it.
However, after a few days of computation the approximated time stops
decreasing or even worse it starts to increase. We think this might
be due to the large amount of files generated (after 70h more than
one million files are there), getting into disk/system overload as a
result. We have also tried onto a Linux system but without a really
more effective processing.
I am not so familiar with VPB source code, so I prefer to look for
advices here before going into a more detailed understanding. First
of all, how is the approximated time to completion computed and how
is it accurate ?
As far as I remember I ignored this time, because it did not seem
accurate at all for our setup.
Then, as far as I can see, using a cluster should
not help us to manage the file problem even if it can stretch timing
a little bit. At last, I do not know if someone has already tried to
generate such a big database but what will be your advices to manage
this more efficiently ?
We use a small cluster and quoting from
http://www.openscenegraph.org/projects/VirtualPlanetBuilder/wiki/ExampleClusterJPD
"The vpbmaster command created 473 tasks and after 50 hours of
processing created 1.5 million files with a total size of 487GB."
The input data is around 800GB if I remember OK.
So the 60 days does sound a bit extreme. Are you getting any messages
that vpb is transforming the coordinate system of your files? If so
you'd probably be better off doing this as a preprocess step using the
gdal tools. I would e.g. convert the ECW's to Geotiff's in the final
projection that you use.
Also, you could try adding overviews to your input data, e.g.
gdaladdo my.tiff 2 4 8 16
It could speed up vpb's loading of the input data.
You could also try add --compressed --npot to the vpb command line.
Hope this helps
rgds
jp
If not possible, we will probably try the online way to go
(osgEarth)...
Thank you!
Cheers, Luc
------------------ Read this topic online here:
http://forum.openscenegraph.org/viewtopic.php?p=22451#22451
_______________________________________________ osg-users mailing
list [email protected]
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
--
This message is subject to the CSIR's copyright terms and conditions, e-mail legal notice, and implemented Open Document Format (ODF) standard.
The full disclaimer details can be found at http://www.csir.co.za/disclaimer.html.
This message has been scanned for viruses and dangerous content by MailScanner,
and is believed to be clean. MailScanner thanks Transtec Computers for their support.
_______________________________________________
osg-users mailing list
[email protected]
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
