-------- Original Message -------- From: "Simone Giannecchini" <[EMAIL PROTECTED]> To: "Vincent Schut" <[EMAIL PROTECTED]> Subject: Re:[Geotools-devel] [Geoserver-devel] questions about pyramids or overviews for very large image mosaics Date: 10/20/2006 04:33 PM > On 10/20/06, Vincent Schut <[EMAIL PROTECTED]> wrote: > >> -------- Original Message -------- >> From: "Simone Giannecchini" <[EMAIL PROTECTED]> >> To: "Vincent Schut" <[EMAIL PROTECTED]> >> Subject: Re:[Geoserver-devel] questions about pyramids or overviews for >> very large image mosaics >> Date: 10/20/2006 03:57 PM >> >>> On 10/20/06, Vincent Schut <[EMAIL PROTECTED]> wrote: >>> >>>> Continuing the WCS testing saga... >>>> >>>> We have a bunch of srtm raster data here and I thought that might be >>>> nice for some coverage mosaic testing... Creating the mosaic went OK >>>> using the mosaicIndexBuilder from geotools/ext/coverage-development. >>>> >>> >>> >>>> However, when trying to create pyramids for this mosaics, the >>>> pyramidbuilder gives an error: >>>> >>>> Oct 20, 2006 3:12:43 PM >>>> it.geosolutions.utils.imagepyramid.PyramidBuilder getNotification >>>> INFO: Progress is at 0.0 >>>> attached message is: Requested scale factor is 2 >>>> Oct 20, 2006 3:12:43 PM >>>> it.geosolutions.utils.imagepyramid.PyramidBuilder getNotification >>>> INFO: Progress is at 0.0 >>>> attached message is: Acquiring a mosaic reader to mosaic >>>> /mnt/amalia/srtm3/SRTM_Data_GeoTiff/srtm3_mosaic.shp >>>> Error while parsing JAI registry file >>>> "/usr/local/src/geotools/branch-2.3.x/module/coverage/target/classes/META-INF/registryFile.jai" >>>> >>>> : >>>> Error in registry file at line number #31 >>>> A descriptor is already registered against the name >>>> "org.geotools.Combine" under registry mode "rendered" >>>> Error in registry file at line number #32 >>>> A descriptor is already registered against the name >>>> "org.geotools.Hysteresis" under registry mode "rendered" >>>> Error in registry file at line number #33 >>>> A descriptor is already registered against the name >>>> "org.geotools.NodataFilter" under registry mode "rendered" >>>> Oct 20, 2006 3:12:46 PM >>>> it.geosolutions.utils.imagepyramid.PyramidBuilder getNotification >>>> INFO: Progress is at 0.0 >>>> attached message is: Original envelope is [180°00.0'W 60°00.0'S , >>>> 180°00.0'E 60°00.0'N] >>>> Oct 20, 2006 3:12:46 PM >>>> it.geosolutions.utils.imagepyramid.PyramidBuilder getNotification >>>> INFO: Progress is at 0.0 >>>> attached message is: Original range is GeneralGridRange[0..432000, >>>> 0..144000] >>>> Oct 20, 2006 3:12:46 PM >>>> it.geosolutions.utils.imagepyramid.PyramidBuilder getNotification >>>> INFO: Progress is at 0.0 >>>> attached message is: New dimension is (W,H)==(216000.0,72000.0) >>>> Oct 20, 2006 3:12:46 PM >>>> it.geosolutions.utils.imagepyramid.PyramidBuilder getNotification >>>> INFO: Progress is at 0.0 >>>> attached message is: New matrix dimension is (cols,rows)==(72,24) >>>> Oct 20, 2006 3:12:46 PM >>>> it.geosolutions.utils.imagepyramid.PyramidBuilder getNotification >>>> INFO: Progress is at 0.0 >>>> attached message is: Preparing tile (col,row)==(0,0) to file >>>> /mnt/amalia/srtm3/SRTM_Data_GeoTiff/2/mosaic_0.tiff >>>> Oct 20, 2006 3:12:46 PM >>>> it.geosolutions.utils.imagepyramid.PyramidBuilder getNotification >>>> INFO: Progress is at 0.0 >>>> attached message is: Reading with grid envelope [180°00.0'W 60°00.0'S , >>>> 175°00.0'W 55°00.0'S] >>>> Oct 20, 2006 3:12:46 PM >>>> it.geosolutions.utils.imagepyramid.PyramidBuilder getNotification >>>> INFO: Progress is at 0.0 >>>> attached message is: Scaling... >>>> Exception in thread "PyramidBuilder" java.lang.NullPointerException >>>> at >>>> org.geotools.coverage.processing.operation.ScaledGridCoverage2D.create(ScaledGridCoverage2D.java:88) >>>> >>>> at >>>> org.geotools.coverage.processing.operation.Scale.doOperation(Scale.java:148) >>>> >>>> at >>>> it.geosolutions.utils.imagepyramid.PyramidBuilder.run(PyramidBuilder.java:679) >>>> >>>> at java.lang.Thread.run(Thread.java:595) >>>> >>>> >>> >>> This error is quite strange. It seems that JAI is not able to register >>> some of the custom operations that we have in geotools, which by the >>> way are not used by the PyramidBuilder. Can you give some details >>> about he platform where this is happening? Anyway the code that >>> actaully does this registration comes Martin Desruisseaux (he is >>> cc-ed) hence I would like to know what he thinks of this error. >>> >> This is on gentoo linux, amd64. sun-jdk 1.5. >> >>>> This is using geotools 2.3 branch from today's svn. JAI and JAI-IO are >>>> from cvs couple of days ago. >>>> This error is independent from 1) tile size (tried 500, 1000 and 3000) >>>> and 2) scaling algorithm (tried nn, avg, bil). >>>> Now this is a pretty large mosaic, so maybe that is a problem? Earlier I >>>> tried running the pyramidBuilder on a subset, and that mostly went OK >>>> (not always, but alas the errors were erratic and not reproduceable). >>>> >>> At FOSS06 I showed people various mosaics, one had been made from Blue >>> marble , the base level add 1000 images, but I had another one >>> covering chicago where the base level had exaclty 10.000 images. I am >>> builing a new one for san diego with 20.000 image so, in thery 3000 >>> should not be that much :-). >>> >> OK, clear :) >> >>> >>>> I'll start some testing on smaller mosaics, but in the mean time want to >>>> ask: >>>> >>>> - should the pyramidBuilder be able to handle mosaics this large? >>>> >>> See above. >>> >>> >>>> - any idea what could be the problem? >>>> >>> See above. >>> >>> >>>> side note questions: >>>> - is it also possible to achieve the same performance by adding internal >>>> overviews to the original mosaic tiles? Or will pyramids outperform >>>> internal overviews? >>>> >>> Rationale of the ImagePyramid. If you have less then 300 images of a >>> reasonable size (up to 100 mb) usin the ImageMosaic with embedded >>> overviews should be fine. Beyond that it won't work ebcause you might >>> try to open too many files at the same time having problems very >>> quickly on some platforms. Pyramids are there to reduce not only the >>> resoulution but also the number of files to be used. Moreover you >>> avoid seeking into a big file and you get more parallelism. >>> >> Aha, hadn't thought of the open files thing. >> >>>> - and if I use internal overviews, will the imagemosaic plugin >>>> automatically use these when a lower resolution image is requested? >>>> >>> Yeah, I fixed a bug in the mosaic plugin just three days ago that was >>> giving strange results depending on the resolution. Now it work >>> perfectly (well, fine is probably better because it generates less >>> expectations :-) ). >>> >> Good to know. Will continue to try and get the pyramidBuilder working >> though. >> However, first rebuilding geotools and geoserver because when trying to >> visualize a mosaic geoserver complains about the new input stream being >> not found >> (<ServiceException>it/geosolutions/imageio/stream/input/FileImageInputStreamExtImpl</ServiceException> >> </ServiceExceptionReport>). Hope rebuilding geotools and geoserver and >> redeploying will solve this... >> Then I'll try some pyramids going with less files first, see if that >> changes things. I'll keep you updated. >> > > As I told you this error is quite strange, it is not relate at all to > the PyramidBuilder. As I said let's first see what Martin says, then > I'll look myself at it a but closer. > You running on linux right? > yep. Linux (gentoo) on amd64. JAI & JAI-IO from cvs (couple of days old), java 5 (sun-jdk), tomcat 5.5 (however that has nothing to do with the pyramidBuilder of course). Everything is running in native 64 bit mode.
I know the error is strange, and I know I am testing several things through each other at the same time which makes things hard to separate... Not a very fluent java developer myself (python, actually...), but I've been doing some java & JAI stuff (just creating processing chains and some simple custom operations, no real magic) myself not so long ago so I know a bit what we're talking about, and how hard it can be to solve (and find) JAI related errors. Wonderful library still, though :) Reason I mention testing with less images is because I did manage to get the pyramidBuilder to run on a small subset of srtm tiles some days ago. So either something happened to the codebase or my setup in between, or it actually is somehow related to the dataset (not necessarily to it's size). I'm of course curious about Martins comments too, though. Anyway, thanks for looking into this. I'll keep you and the list updated on my progress, and eagerly await any comments or hints. Vincent. > Simone. > > > >> Ciao, >> Vincenzo (been in Italy recently ;-)) >> >>>> Cheers, >>>> Vincent. >>>> >>>> >>> Ciao, >>> Simone. >>> >>>> ------------------------------------------------------------------------- >>>> >>>> Using Tomcat but need to do more? Need to support web services, >>>> security? >>>> Get stuff done quickly with pre-integrated technology to make your >>>> job easier >>>> Download IBM WebSphere Application Server v.1.0.1 based on Apache >>>> Geronimo >>>> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 >>>> _______________________________________________ >>>> Geoserver-devel mailing list >>>> [EMAIL PROTECTED] >>>> https://lists.sourceforge.net/lists/listinfo/geoserver-devel >>>> >>>> >>> >> >> > > > ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 _______________________________________________ Geotools-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/geotools-devel
