Re: [Geotools-devel] ArcSDE raster suppor refurbished, concept check and call for review

[Saul Farber]

Gabriel, I think the move to JAI is *awesome*.  I haven't looked at the code specifically yet, but I really like that you've offloaded the implementation details that can be handled by JAI to code that doesn't need to be supported inside geotools. There's a little on the LZERO_ORIGIN_TILE parameter here http://docs.codehaus.org/display/GEOTDOC/ArcSDE+Raster+Support but the deeper explanation is below: In ArcSDE 9.1 or below, when you load a raster using a sequence of tiles the *first* tile that you load gets to be the "0,0" tile, and the raster "origin" is set to the lower-left corner of that tile.  This is unmodifiable after the raster is created. When querying a raster, one needs to know the x/y location of the upper-left tile and then work out the actual tiles to be queried based on the requested world-extent.  On raster pyramid levels 1 or above, the x/y position of the 0,0 tile is trustworthy and will be the upper-left tile in the raster. However, at level zero, there can be a problem in this specific case: * You're using ArcSDE 9.1 or below * You didn't set a raster_origin point before loading the raster * You loaded the raster using a sequence of files (probably .tif files) and you didn't load the upper-left tile as the FIRST tile If the above three conditions are true, then the ArcSDE Java API will happily tell you that the level zero 0,0 tile is at the upper-left of the raster, but when you actually ask for that tile, you'll find out that at level zero the 0,0 tile is the FIRST TILE LOADED. Therefore, in order to access tiles above or to the left of the first tile loaded, you would need to trick ArcSDE into fetching you data from "negatively indexed" tiles, but the Jsde api doesn't allow negatively indexed queries, so that's out.  This means that the data that is above or to the left of the first loaded tile is unavalaible at level zero, and you have to return level 1 (or 2, if skipLevelOne is set) data instead.  The LZERO_ORIGIN_TILE parameter allows you to tell geotools exactly which tile is the "first loaded" tile so that queries of tiles above or to the left of that tile at level zero can be returned with level 1 or 2 data. This is a "filed and known" bug in the JSDE api, and in ArcSDE 9.2 or above you can re-set the raster origin using "sderaster -o ..." which I believe recalculates the various tables that cause this problem. I'd say that unless you have just such a setup to test against, I wouldn't worry about keeping support for the LZERO_ORIGIN_TILE parameter.  If someone can come up with an sde raster that has this, err, "issue" and they can reproduce the problem, then all the code is in-place to support the fix...but without a working example it's a tough thing to code. hope that help gabriel, --saul Gabriel Roldan wrote: Hi guys, As you know I've been working on refurbishing the geotools arcsde module to support more raster formats. In the process, I've found the previous Saul work invaluable to understand how the arcsde raster api works, started adding support for some specific formats (16-bit unsigned, 32-bit float, 8bit-u colormapped) following the initial Saul's design. As a parallel inspiration I started spiking how to better integrate with JAI. It turned out to be easier than expected (once I learned a lot, remember I'm a newbie in gce land). In the end, I refactored a lot of code and come up with a new design, which makes use of an ImageInputStream implementation that reads arcsde tiles and is wrapped by a RawImageInputStream. So far so good, we have now a single raw arcsde raster format reader to rule them all. So, I would like to kindly ask you to answer a couple questions, before I go forward in wraping up the work and backporting to 2.5.x. - Saul: what is the intended use of the LZERO_ORIGIN_TILE parameter? I would need to check if it still applies. - All: I would love a concept check on the following explanation of the new workings of the module, and even a bit of a code review would be highly appreciated. Though there may be still a couple rough edges, I think I could make good use of your higher experience with gce to early catch on conceptual missunderstandings or technical mistakes. * How it used to work: Given a read request to the GT arcsde AbstractGridCoverage2DReader implementation: 1- connect to the raster table and gather raster layout and metadata information 2- create a BufferedImage compatible with the raster format, backed by the appropriate colormodel and samplemodel, and with the requested dimensions. Fail if there's no support for the raster's format and number of bands. 3- find out the most appropriate pyramid level to fullfill the requested extent and image dimension, as well as the matching tiles. 4- look up for a specific ArcSdeRasterReader(tm) for the given pixel depth and number of bands. 5- start fetching the raster tiles, for each tile, pass it to the ArcSdeRasterReader, together with the target BufferedImage's WritableRaster and the offset and band where to write the tile's content. The reader takes care of writing only the samples that match the target extent. 6- create and return the coverage backed by the BufferedImage. It matches exactly the requested extent and image size. * How it works now: Given a read request to the GT arcsde AbstractGridCoverage2DReader implementation: 1- Gather the raster layout and metadata. It is cached so fetch it if not yet available. It contains the raster dimensions, band, and pyramid info. 2- use the OVERVIEW_POLICY argument to determine the most appropriate pyramid level, default to QUALITY if not provided. This leverages the code in AbstractGridCoverage2DReader.setReadParams to select the overview (ie, pyramid level) 3- based on the requested extent and pyramid level, calculate the matching tiles that overlap the requested extent for the selected pyramid level, as well as the resulting image size and extent. This may well be different than the requested extent and image dimension, but we're going to return the native resolution and only the matching pixels, letting the client code deal with interpolating if needed. 4- If the request does not match the raster extent, create a zero area image, and return a null extent coverage. Otherwise create an ArcSDEImageInputStream to fetch the raw byte[] data for the whole matching tiles, and call the JAI ImageRead operation with an appropriate ImageTypeSpecifier and ImageLayout. 5- since the read image contains an area covered by full tiles, use the JAI crop operation to crop it to the image dimension that better match the requested extent. 6- create and return the coverage backed by the actual RendedImage and resulting extent, in native (pyramid level) resolution. Tha's quite it. Any comments will be appreciated. -- Gabriel Roldan OpenGeo - http://www.opengeo.org

------------------------------------------------------------------------------
Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM)
software. With Adobe AIR, Ajax developers can use existing skills and code to
build responsive, highly engaging applications that combine the power of local
resources and data with the reach of the web. Download the Adobe AIR SDK and
Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com
_______________________________________________
Geotools-devel mailing list
Geotools-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel