Deak Michael, please read below... ------------------------------------------------------- Ing. Simone Giannecchini GeoSolutions S.A.S. Founder - Software Engineer Via Carignoni 51 55041 Camaiore (LU) Italy
phone: +39 0584983027 fax: +39 0584983027 mob: +39 333 8128928 http://www.geo-solutions.it http://geo-solutions.blogspot.com/ http://www.linkedin.com/in/simonegiannecchini http://twitter.com/simogeo ------------------------------------------------------- On Thu, Jan 14, 2010 at 3:17 PM, Michael Härtel <[email protected]> wrote: > Hello list, > > I know that there was a posting to the list covering the exact topic but I > am still unable to read a small subset from a huge image. I would point out that this email should probably be posted to the JAI ImageIO list rather than to the GeoTools list. > > The JP2k image is 114500 x 28820 pixels in size and it is tiled. the tiles > have the dimension 15000 x 15000 (!). > > my first attempt was to read one tile with JAI only: > > > --- > > > FileImageInputStream fiis = null; > try{ > fiis = new FileImageInputStream(file); > }catch (Exception e) { > e.printStackTrace(); > } > > ImageReader ir = > getImageReaderForFileImageInputStreamFromOperationRegistry(fiis); > > // Reader is > // com.sun.media.imageioimpl.plugins.jpeg2000.J2KImageReaderSpi > > //Setting Area of Interest 512 x 512 Pixels > Rectangle roi = new Rectangle(1400, 200, 512, 512); > > // Setting Read parameter > J2KImageReadParam rp = new J2KImageReadParam(); > rp.setSourceRegion(roi); > > JAI jai = JAI.getDefaultInstance(); > > //setting cache > TileCache cache = jai.getTileCache(); > long size = 512*1024*1024L; > cache.setMemoryCapacity(size); > > JAI.setDefaultTileSize(new Dimension(512, 512)); > > //ParameterBlock for Reading > ParameterBlockJAI pb = new ParameterBlockJAI(new ImageReadDescriptor()); > pb.set(fiis, 0); > pb.set(0, 1); > pb.set(false, 2); > pb.set(false, 3); > pb.set(true, 4); > pb.set(null, 5); > pb.set(null, 6); > pb.set(rp, 7); > pb.set(ir, 8); > > //setting source file > Vector<Object> vec = new Vector<Object>(); > vec.add(fiis); > > > //setting ImageLayout to use smaller tiles > ImageLayout layout = new ImageLayout(); > layout.setTileGridXOffset(0); > layout.setTileGridYOffset(0); > layout.setTileWidth(512); > layout.setTileHeight(512); > > //Rendering Hints for the image Layout > RenderingHints rh = new RenderingHints(JAI.KEY_IMAGE_LAYOUT, layout); > > // finally, create the PlanarImage > PlanarImage pi = JAI.create("ImageRead", pb, rh); > > > // trying to receive the subset > Raster rst = null; > try{ > rst = pi.getData(); > wrst = rst.createCompatibleWritableRaster(); > }catch (Exception e) { > e.printStackTrace(); > } > Notice that this triggere a full copy of the image. > --- > > At the line > > "rst = pi.getData();" > > the program virtually stops. It computes for about 4 minutes and then returns > out of memory error: > --- > > java.lang.reflect.InvocationTargetException > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) > at java.lang.reflect.Method.invoke(Unknown Source) > at > org.eclipse.ve.internal.java.vce.launcher.remotevm.JavaBeansLauncher.main(JavaBeansLauncher.java:79) > Caused by: java.lang.OutOfMemoryError: Java heap space > at jj2000.j2k.image.DataBlkInt.<init>(DataBlkInt.java:97) > at > jj2000.j2k.wavelet.synthesis.InvWTFull.getInternCompData(InvWTFull.java:277)IWAV0052E > Invocation Target Exception creating com.haertelonline.test.JAItester > > at jj2000.j2k.image.ImgDataConverter.getData(ImgDataConverter.java:249) > at > jj2000.j2k.image.ImgDataConverter.getInternCompData(ImgDataConverter.java:205) > at > jj2000.j2k.image.invcomptransf.InvCompTransf.getInternCompData(InvCompTransf.java:360) > at > com.sun.media.imageioimpl.plugins.jpeg2000.J2KReadState.readSubsampledRaster(J2KReadState.java:831) > at > com.sun.media.imageioimpl.plugins.jpeg2000.J2KReadState.readBufferedImage(J2KReadState.java:384) > at > com.sun.media.imageioimpl.plugins.jpeg2000.J2KImageReader.read(J2KImageReader.java:454) > at > com.sun.media.jai.imageioimpl.ImageReadOpImage.computeTile(ImageReadOpImage.java:697) > at > com.sun.media.jai.util.SunTileScheduler.scheduleTile(SunTileScheduler.java:914) > at javax.media.jai.OpImage.getTile(OpImage.java:1138) > at javax.media.jai.PlanarImage.getData(PlanarImage.java:2085) > at javax.media.jai.PlanarImage.getData(PlanarImage.java:2016) > at javax.media.jai.RenderedOp.getData(RenderedOp.java:2266) > at com.haertelonline.test.JAItester.main(JAItester.java:225) > ... 5 more > > > I tried not to use cache, and omitting the rendering hints but it doesn't > change anything.\ Notice the lines jj2000.j2k.image...., this means that you are using JJ2000, the pure Java reference implementation of the JPEG2K spec. It is known to have limitations on large imagery since it cannot do tiled reading, hence the errors you are getting. You might want to try the ImageIO native bindings that contain the C reference Implementation, the Jasper library. It should be a bit better. > > Because I read that geotools can do this better I started a second attempt: > Where? > --- > ImageWorker iw = null; > try{ > iw = new ImageWorker(file); > }catch (Exception e) { > e.printStackTrace(); > } > > result: > > --- > java.lang.IllegalArgumentException: Dimensions (width=114500 height=28820) > are too large > at java.awt.image.SampleModel.<init>(Unknown Source) > at java.awt.image.ComponentSampleModel.<init>(Unknown Source) > at java.awt.image.PixelInterleavedSampleModel.<init>(Unknown Source) > at > java.awt.image.PixelInterleavedSampleModel.createCompatibleSampleModel(Unknown > Source) > at > com.sun.media.imageioimpl.plugins.jpeg2000.J2KReadState.readBufferedImage(J2KReadState.java:371) > at > com.sun.media.imageioimpl.plugins.jpeg2000.J2KImageReader.read(J2KImageReader.java:454) > at javax.imageio.ImageIO.read(Unknown Source) > at javax.imageio.ImageIO.read(Unknown Source) > at org.geotools.image.ImageWorker.<init>(ImageWorker.java:172) > at com.haertelonline.test.GeoToolsTester.main(GeoToolsTester.java:196) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) > at java.lang.reflect.Method.invoke(Unknown Source) > at > org.eclipse.ve.internal.java.vce.launcher.remotevm.JavaBeansLauncher.main(JavaBeansLauncher.java:79) > --- As you can see from the stacktrace, you are simply calling ImageIO with this method. So no deferred loading, you are loading the whole image in memory. > > OK, next attempt: > > JP2KReader jp2r = null; > try{ > jp2r = new JP2KReader(file); > }catch (Exception e) { > e.printStackTrace(); > } > --- > > result is (line "jp2r = new JP2KReader(file)" was the problem): > > org.geotools.data.DataSourceException: Unavailable envelope for this coverage > at > org.geotools.coverageio.jp2k.JP2KReader.setCoverageProperties(JP2KReader.java:157) > at org.geotools.coverageio.jp2k.JP2KReader.<init>(JP2KReader.java:523) > at org.geotools.coverageio.jp2k.JP2KReader.<init>(JP2KReader.java:120) > at com.haertelonline.test.GeoToolsTester.main(GeoToolsTester.java:212) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) > at java.lang.reflect.Method.invoke(Unknown Source) > at > org.eclipse.ve.internal.java.vce.launcher.remotevm.JavaBeansLauncher.main(JavaBeansLauncher.java:79) > > --- GeoTools, focuses on geospatial data, you jp2 does look like containig geospatial data, hence the error. > > Dear list, can someone please tell me how those images can be read? The > images I want to read are on the "ESRI DATA & MAPS CD". > > One more general question: What happens under the hood when trying to read > packed (LZW/RLE/Packbits/) images. How can the program find out at which byte > position of a stream a specific pixel coordinate is located? > > The tiles of the test image is 15000 x 15000, so there are 644MB RAM is > needed to load one tile in memory. I have 2GB Ram and the parameters > -Xms1024m -Xmx1024m are passed to the JVM when launching my small program. > Does the "code behind" JAI need to unpack the tiles first to find the roi I > requested? > Now a few points: 1> JAI ImageIO as it stands is really inefficient when it comes to reading JPEG2000 images 2> GeoTools, by default, relies on JAI ImageIO for doing I/O therefore inherits this deficiencies. Ok, now a path to get better results. A while ago we ( as in GeoSolutions) started a sibling project to add more I/O plugins to ImageIO, we called it ImageIO-ext (https://imageio-ext.dev.java.net/). GeoTools can use it depending on what it can find in the path (for native libs) and classpath (for jars) ( see http://docs.codehaus.org/display/GEOTOOLS/JP2K+Plugin). Now, inside ImageIO-ext we have various JPEG2000 ImageIO reader that geotools and/or your code can leverage on. - gdal-ecw uses the ECW SDK via GDAL - gdal-mrsid uses the MRSID SDK via GDAL - gdal-kakadu uses the Kakadu SDK via GDAL - kakadu uses the KAKADU sdk directly ECW and MrSID SDK are for reading purposes but not open source. The kakadu sdk has various licensing options. GDAL is a c/c++ raster I/O lib that is open source. So you options are, imho: - use either one of the gdal-exw. gdal-mrsid libs via imageio-ext - buy a developer license of kakadu (250 euro) and use it via imageio-ext (this is by far the best performing option). I would try to give a go to imageio-ext. If you have problems, send an email to the user forum we will help you out. Ciao, Simone. > Thanks for any answer, > > haerta > > > > ------------------------------------------------------------------------------ > Throughout its 18-year history, RSA Conference consistently attracts the > world's best and brightest in the field, creating opportunities for Conference > attendees to learn about information security's most important issues through > interactions with peers, luminaries and emerging and established companies. > http://p.sf.net/sfu/rsaconf-dev2dev > _______________________________________________ > Geotools-gt2-users mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/geotools-gt2-users > ------------------------------------------------------------------------------ Throughout its 18-year history, RSA Conference consistently attracts the world's best and brightest in the field, creating opportunities for Conference attendees to learn about information security's most important issues through interactions with peers, luminaries and emerging and established companies. http://p.sf.net/sfu/rsaconf-dev2dev _______________________________________________ Geotools-gt2-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/geotools-gt2-users
