RE: [OSGeo-Discuss] Open File Formats andProprietaryAlgorithms[SEC=UNCLASSIFIED]
> 10:1 compression rates without noticeable image quality reductions, For some appropriate, workflow-specific definition of "noticeable". Everything is a tradeoff. I always tell people to run their own tests with their own datasets to determine what sort of quality they will achieve and what their users' workflows will require. -mpg -Original Message- From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Paul Ramsey Sent: Friday, August 21, 2009 11:36 AM To: OSGeo Discussions Subject: Re: [OSGeo-Discuss] Open File Formats andProprietaryAlgorithms[SEC=UNCLASSIFIED] So hung up on wavelets, we are. Internally tiled TIFF with JPEG compression and similarly formatted internal overviews can achieve 10:1 compression rates without noticeable image quality reductions, and as an added bonus can be decompressed a heck of a lot faster than wavelet-based formats. The wavelet stuff is k00l, in that there is no need for an overview pyramid (it's implicit in the compression math) and much higher compression rates can be achieved. But operationally, you can go a long way with the more primitive (open image format + open compression algorithm) approach. P. ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
Re: [OSGeo-Discuss] Open File Formats andProprietaryAlgorithms[SEC=UNCLASSIFIED]
So hung up on wavelets, we are. Internally tiled TIFF with JPEG compression and similarly formatted internal overviews can achieve 10:1 compression rates without noticeable image quality reductions, and as an added bonus can be decompressed a heck of a lot faster than wavelet-based formats. The wavelet stuff is k00l, in that there is no need for an overview pyramid (it's implicit in the compression math) and much higher compression rates can be achieved. But operationally, you can go a long way with the more primitive (open image format + open compression algorithm) approach. P. ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
Re: [OSGeo-Discuss] Open File Formats andProprietaryAlgorithms[SEC=UNCLASSIFIED]
On Fri, Aug 21, 2009 at 09:45:04AM -0700, Landon Blake wrote: > MPG wrote: "Tiling" essentially means you can take a large file and > compress pieces of it independently. This avoids having to deal with > the large memory footprint issues, but it can also lead to seam-line > artifacts under certain conditions. Ideally, one would prefer to have > the option of compressing large images without resorting to using > tiles." > > This is probably a stupid question, since I know absolutely nothing > about image compression, but couldn't you overlap the tiles slightly to > avoid the seam lines? > > This would obviously result in a slightly larger file size because some > pixels would be compressed twice. But that might be OK if you were > trying to compress a huge image. > > What about reading chunks of the image off disk, instead of trying to > put the whole image in memory? This would be slower, but might make an > impossible task possible. "Reading chunks of image off disk" == "tiling". With compression, bits aren't stored on th disk in a way that you can say "Okay, bytes 0-32768 are the first 720 pixels" in any way. Instead, you have to decompress the image, or part of it, to start to learn these things. Tiling lets you split the image up into many little chunks, which you can read individually. > We run into this problem with vector datasets to. Some datasets are just > to stinking BIG. One of my tasks for OpenJUMP is to write a core module > that displays vector data accessed directly from disk, instead of from > memory. This will be slower, but it is better than crashing the program > because there isn't enough RAM. Most Vector datasets have some lvel of "random access" -- I can look for feature 7, and get it, because i know where the start and end of feature 7 are. I don't know where the start and end of pixel 7 is -- because its' different depending on exactly how wth file is compressed. This is all a vast simplification, and some of it is probably not-entirely right, but the problems are -- as you suggested -- more complex than most people not working in imagery know. (And even more complex than some of them know, most likely :)) Regards, -- Christopher Schmidt Web Developer ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
RE: [OSGeo-Discuss] Open File Formats andProprietaryAlgorithms[SEC=UNCLASSIFIED]
Not a stupid question -- but it doesn't work that way. The artifacts are due to the wavelet processing of the pixels near the tile boundaries, and the boundaries have to be treated reflectively within their individual tiles (in order to keep each tile separate), which means you can sometimes "see" where those boundaries are. "Overlapping" doesn't help because the wavelet footprint spans a large width, in order to handle the lower-resolution scales. Which in turn means you need to be able "reach" far away parts of the image at various (some might say "arbitrary") stages in the wavelet pipeline. Just trust me, it is a nontrivial problem to solve. Brighter minds than ours have spent a lot of energy on this problem -- a literature search would reveal a number of PhD theses and patents. -mpg -Original Message- From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Landon Blake Sent: Friday, August 21, 2009 10:45 AM To: OSGeo Discussions Subject: RE: [OSGeo-Discuss] Open File Formats andProprietaryAlgorithms[SEC=UNCLASSIFIED] MPG wrote: "Tiling" essentially means you can take a large file and compress pieces of it independently. This avoids having to deal with the large memory footprint issues, but it can also lead to seam-line artifacts under certain conditions. Ideally, one would prefer to have the option of compressing large images without resorting to using tiles." This is probably a stupid question, since I know absolutely nothing about image compression, but couldn't you overlap the tiles slightly to avoid the seam lines? This would obviously result in a slightly larger file size because some pixels would be compressed twice. But that might be OK if you were trying to compress a huge image. What about reading chunks of the image off disk, instead of trying to put the whole image in memory? This would be slower, but might make an impossible task possible. We run into this problem with vector datasets to. Some datasets are just to stinking BIG. One of my tasks for OpenJUMP is to write a core module that displays vector data accessed directly from disk, instead of from memory. This will be slower, but it is better than crashing the program because there isn't enough RAM. Things must be more complicated than can be described in an e-mail, because we've got people a lot smarter than me working on these problems. I am just curious. (I tried reading about wavelet compression on Wikipedia yesterday and quickly got a headache.) :] Landon Office Phone Number: (209) 946-0268 Cell Phone Number: (209) 992-0658 -Original Message- From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Michael P. Gerlek Sent: Friday, August 21, 2009 9:36 AM To: OSGeo Discussions Subject: RE: [OSGeo-Discuss] Open File Formats andProprietaryAlgorithms[SEC=UNCLASSIFIED] "Tiling" essentially means you can take a large file and compress pieces of it independently. This avoids having to deal with the large memory footprint issues, but it can also lead to seam-line artifacts under certain conditions. Ideally, one would prefer to have the option of compressing large images without resorting to using tiles. Note too that, in addition to the large image issue, many of the JP2 implementations out there are either not fully compliant or are not tuned for performance. A viable solution would need both of these. -mpg -Original Message- From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Chris Puttick Sent: Friday, August 21, 2009 9:37 AM To: OSGeo Discussions Subject: Re: [OSGeo-Discuss] Open File Formats and ProprietaryAlgorithms[SEC=UNCLASSIFIED] Well, according to this page: http://jpeg2000.epfl.ch/ v.5.1, courtesy in part Eastman Kodak, provides "complete JP2 support at the decoding side" - not sure whether that covers the tiling or other geo needs, but doesn't it sound worth investigating? Chris - "Christopher Schmidt" wrote: > On Fri, Aug 21, 2009 at 08:27:13AM -0700, Landon Blake wrote: > > Thanks for the information Michael. I am downloading Opticks right > now. > > :] > > > > I also found this Java library for JP2, thought I'm not sure how > > complete/up-to-date it is: > > > > http://jj2000.epfl.ch/ > > > > Maybe we need a JPEG 2000 page on the OSGeo wiki. > > Note that "JPEG 2000 support" is different from "JPEG 2000 support > which > works on geo-sized images." The tiling (or 'paging'? as Michael calls > it) support that's supposed to be provided by OpenJPEG2000 has been > coming 'real soon now' for about 18 months now from my uneducated > observations, and until it's there, most tools using Ope
RE: [OSGeo-Discuss] Open File Formats andProprietaryAlgorithms[SEC=UNCLASSIFIED]
MPG wrote: "Tiling" essentially means you can take a large file and compress pieces of it independently. This avoids having to deal with the large memory footprint issues, but it can also lead to seam-line artifacts under certain conditions. Ideally, one would prefer to have the option of compressing large images without resorting to using tiles." This is probably a stupid question, since I know absolutely nothing about image compression, but couldn't you overlap the tiles slightly to avoid the seam lines? This would obviously result in a slightly larger file size because some pixels would be compressed twice. But that might be OK if you were trying to compress a huge image. What about reading chunks of the image off disk, instead of trying to put the whole image in memory? This would be slower, but might make an impossible task possible. We run into this problem with vector datasets to. Some datasets are just to stinking BIG. One of my tasks for OpenJUMP is to write a core module that displays vector data accessed directly from disk, instead of from memory. This will be slower, but it is better than crashing the program because there isn't enough RAM. Things must be more complicated than can be described in an e-mail, because we've got people a lot smarter than me working on these problems. I am just curious. (I tried reading about wavelet compression on Wikipedia yesterday and quickly got a headache.) :] Landon Office Phone Number: (209) 946-0268 Cell Phone Number: (209) 992-0658 -Original Message- From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Michael P. Gerlek Sent: Friday, August 21, 2009 9:36 AM To: OSGeo Discussions Subject: RE: [OSGeo-Discuss] Open File Formats andProprietaryAlgorithms[SEC=UNCLASSIFIED] "Tiling" essentially means you can take a large file and compress pieces of it independently. This avoids having to deal with the large memory footprint issues, but it can also lead to seam-line artifacts under certain conditions. Ideally, one would prefer to have the option of compressing large images without resorting to using tiles. Note too that, in addition to the large image issue, many of the JP2 implementations out there are either not fully compliant or are not tuned for performance. A viable solution would need both of these. -mpg -Original Message- From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Chris Puttick Sent: Friday, August 21, 2009 9:37 AM To: OSGeo Discussions Subject: Re: [OSGeo-Discuss] Open File Formats and ProprietaryAlgorithms[SEC=UNCLASSIFIED] Well, according to this page: http://jpeg2000.epfl.ch/ v.5.1, courtesy in part Eastman Kodak, provides "complete JP2 support at the decoding side" - not sure whether that covers the tiling or other geo needs, but doesn't it sound worth investigating? Chris - "Christopher Schmidt" wrote: > On Fri, Aug 21, 2009 at 08:27:13AM -0700, Landon Blake wrote: > > Thanks for the information Michael. I am downloading Opticks right > now. > > :] > > > > I also found this Java library for JP2, thought I'm not sure how > > complete/up-to-date it is: > > > > http://jj2000.epfl.ch/ > > > > Maybe we need a JPEG 2000 page on the OSGeo wiki. > > Note that "JPEG 2000 support" is different from "JPEG 2000 support > which > works on geo-sized images." The tiling (or 'paging'? as Michael calls > it) support that's supposed to be provided by OpenJPEG2000 has been > coming 'real soon now' for about 18 months now from my uneducated > observations, and until it's there, most tools using OpenJPEG for JP2s > are > going to suffering under much the same limitations. > > -- Chris > > > Landon > > Office Phone Number: (209) 946-0268 > > Cell Phone Number: (209) 992-0658 > > > > > > > > -Original Message- > > From: discuss-boun...@lists.osgeo.org > > [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Considine, > Michael > > Sent: Friday, August 21, 2009 8:09 AM > > To: OSGeo Discussions > > Subject: RE: [OSGeo-Discuss] Open File Formats and > > ProprietaryAlgorithms[SEC=UNCLASSIFIED] > > > > All, > > > > Opticks is an open source remote sensing application and > development > > framework. We recently started the process to add JPEG 2000 support > to > > our framework. We picked OpenJpeg to add JPEG 2000 support to our > > application. They are also open source. We currently support > importing > > JPEG 2000 files but we are currently limited to the 4GB memory size > > after decoding. > > > > Our plan is to continue development and to upgrade t
