Re: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms
Excellent idea! It would need a little bit of content management to avoid getting spam in these fields and getting feedback preferably back to the source, but that might be overcome with some brainstorming :-) Cheers, Jeroen On Aug 26, 2009, at 4:51 AM, Ted Habermann wrote: Lots of discussions on standardization of data formats, lots of challenges there. One thing that has not been mentioned (I think) is the idea of standardizing responses from users about 1) uses of data and 2) limitations of data (files too big fits in here). Both of these are included in the MD_Usage object that is part of the ISO 19115 Standard included in GeoNetwork. Would be very cool to build into GeoNetwork the capability to accept user feedback about datasets and to associate that feedback with the appropriate datasets in the GeoNetwork catalog and to make it available to 1) data providers and 2) future users... Seems straightforward, Ted -- Ted Habermann === Enterprise Data Systems Group Leader NOAA, National Geophysical Data Center V: 303.497.6472 F: 303.497.6513 If you want to go quickly, go alone. If you want to go far, go together Old Proverb ted.haberm...@noaa.gov == ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
RE: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms
OK. I'm not an expert on images, but I'd be interested in working with you. However, I avoid C++ like a lethal strain of the Swine Flu. :] I may give some more thought to some of Bob's ideas about making it easier to work with image tiles. Thanks. Landon Office Phone Number: (209) 946-0268 Cell Phone Number: (209) 992-0658 -Original Message- From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Michael P. Gerlek Sent: Monday, August 24, 2009 3:19 PM To: OSGeo Discussions Subject: RE: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms I've not given it much thought recently, to be honest. I'd need to review the current state of things in OpenJp2 (or whatever it's called) to see where they are at, what changes would be realistic and viable, how amenable they'd be to taking patches versus a fork, etc. Done properly, the work would have no geo specific component at all -- it would just be a new version of some of the internal algorithms. The test case would simply be to encode and decode an 500 GB(?) raw file on a box with 2 GB (?) of RAM. I would certainly not want anyone to have to build a whole new jp2 library from scratch, if that's what you meant. I'd really only be interested in C++ (or possibly mono-safe C#). -mpg -Original Message- From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Landon Blake Sent: Monday, August 24, 2009 4:11 PM To: OSGeo Discussions Subject: RE: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms MPG: When you say effort do you mean some sort of library to support JP2 geo side of things? What programming language would you be most interested in? C++? Landon Office Phone Number: (209) 946-0268 Cell Phone Number: (209) 992-0658 -Original Message- From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Michael P. Gerlek Sent: Monday, August 24, 2009 1:59 PM To: OSGeo Discussions Subject: RE: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms It would not be a good SoC thing, due to the level of expertise and time required. I (LizardTech) would likely be willing to contribute to such an effort. -mpg -Original Message- From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Fawcett, David Sent: Monday, August 24, 2009 8:11 AM To: OSGeo Discussions Subject: RE: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms I realize that there are likely not a large number of people who have the expertise and experience to write this kind of code. Is this a project that should be shopped around for funding? Google Summer of Code? A grant from our ~benevolent overlord Google? Some other foundation or org interested in open data formats? David. -Original Message- From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Michael P. Gerlek Sent: Thursday, August 20, 2009 4:36 PM To: OSGeo Discussions Subject: RE: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms snip Do you know why there hasn't been a broader adoption of JP2? Not through lack of trying on my part :-) I think the two biggest reasons are: (1) The algorithms for handling large images in memory really are rocket science, and no one in the FOSS community has gotten the itch sufficiently bad enough to go and do the work needed inside the existing open source packages. Hopefully someday someone will. ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss Warning: Information provided via electronic media is not guaranteed against defects including translation and transmission errors. If the reader is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this information in error, please notify the sender immediately. ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
RE: [OSGeo-Discuss] Open File Formats andProprietaryAlgorithms
Bob, What did you have in mind when you said pre-processing. Landon Office Phone Number: (209) 946-0268 Cell Phone Number: (209) 992-0658 From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Bob Basques Sent: Monday, August 24, 2009 9:21 AM To: OSGeo Discussions Subject: RE: [OSGeo-Discuss] Open File Formats andProprietaryAlgorithms Landon, Just had another thought . . . What about setting up a (openSource) tool set specifically for handling Raster images for pre-processing purposes. Might even be something that publishers could re-distribute with their datasets, as in this processor stack works with our data. Just thinking on the run here, no detail (or ramifications though through [at all :c) ]) bobb Bob Basques bob.basq...@ci.stpaul.mn.us wrote: Landon, It would be interesting to come up with a standard structure on a computer file system that could be used to accessed tiled raster data, if this hasn't been done already. One the file system structure was defined, it would be fairly easy to write open source software that accessed this structure and provided individual tiles as a service to desktop GIS applications. Hmm, interesting angle, to expand on your idea a bit more, what about a processing suite (or set of suites) that process data for different types of uses, visual display, DEM analysis, etc. Each processor stack would/could have it's own rules associated with data resolution vs files sizes, etc. bobb Landon Blake lbl...@ksninc.com wrote: Bobb wrote: Here's my reasoning, we're never (ever?) going to hit the top end on how big files ever get, resolution just keeps going up and up, so there is always going to be some upper limit that will need to be breached somehow. Working out a proper method for segregating the data up front (dare I say it), as some sort of standard (which can be adjusted as time passes) will make everything work nicely, then all will work with available tools when they are available, if tools to handle larger datasets become available, and the community feels there is a reason/need that these new larger files need to be handled, then they get to change the standard. I agree with some of the points you are making in your argument Bobb. There is certainly a practical limit to how much you data you should put in a single file. That is why we have lumber cut to 8 foot lengths. You don't need a flatbed semi to carry it to your house. :] When you refer to a standard for splitting data up front, what do you mean? It would be interesting to come up with a standard structure on a computer file system that could be used to accessed tiled raster data, if this hasn't been done already. One the file system structure was defined, it would be fairly easy to write open source software that accessed this structure and provided individual tiles as a service to desktop GIS applications. Landon From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Bob Basques Sent: Monday, August 24, 2009 7:33 AM To: OSGeo Discussions Subject: RE: [OSGeo-Discuss] Open File Formats and ProprietaryAlgorithms All, Ok, I'm probably going to get someone irritated, but here goes . . Why not approach this from the other end of the spectrum and work at making the original files smaller. Work with the providers to make the images smaller in the first place, or at least come up with a maximum practical size to work with, I mean if this is the only (or biggest reason) for implementing JP2, then getting folks to make the smaller deliverables seems like a better long term approach. Here's my reasoning, we're never (ever?) going to hit the top end on how big files ever get, resolution just keeps going up and up, so there is always going to be some upper limit that will need to be breached somehow. Working out a proper method for segregating the data up front (dare I say it), as some sort of standard (which can be adjusted as time passes) will make everything work nicely, then all will work with available tools when they are available, if tools to handle larger datasets become available, and the community feels there is a reason/need that these new larger files need to be handled, then they get to change the standard. bobb Fawcett, David david.fawc...@state.mn.us wrote: I realize that there are likely not a large number of people who have the expertise and experience to write this kind of code. Is this a project that should be shopped around for funding? Google Summer of Code? A grant from our ~benevolent overlord Google
RE: [OSGeo-Discuss] Open File Formats andProprietaryAlgorithms
Landon, Rather than sput off about something theoretical, I'll relate our use of MRSID deliverables. This is our Typical publishing process for MRSID format data that we receive, which happens fairly regularly. Our use case in this instance is for visual inspection of aerial photography. * First we extract JPG tile pyramids from the MrSID format into MapServer ready tile layers doubling/halving the resolution of the output for each level of tiles along with generating WORLD files for each tile. (we actually have a static grid that we use for tiling that makes this a one time process, since the WORLD files are reusable across data sets.) * Then, index the tiles for each level for Mapserver's use We start at Level 0 (L0 = 6 resolution) and increase by doubling resolution until we have a single tile at the City view level, ~L7. * Mapserver is setup with a multi-level display with cutoffs for each tiled layer at the appropriate scales. * This approach guarantees only four tiles at most (as long as the client viewer pixel width isn't bigger than the tile pixel size of each tile) are ever retrieved by MapServer to build a composite output image. The first three steps could be automated into a package that ran automatically for example from most MRSID deliverables. Another use case might be DEM analysis, which might require converting into other formats as well as the data originating in some other format besides MRSID. Still another use case might be spectral separation and handling the consequent analysis tasks. Anyway, the list goes on and on I sppose. bobb Landon Blake lbl...@ksninc.com wrote: Bob, What did you have in mind when you said “pre-processing”. Landon Office Phone Number: (209) 946-0268 Cell Phone Number: (209) 992-0658 From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Bob Basques Sent: Monday, August 24, 2009 9:21 AM To: OSGeo Discussions Subject: RE: [OSGeo-Discuss] Open File Formats andProprietaryAlgorithms Landon, Just had another thought . . . What about setting up a (openSource) tool set specifically for handling Raster images for pre-processing purposes. Might even be something that publishers could re-distribute with their datasets, as in this processor stack works with our data. Just thinking on the run here, no detail (or ramifications though through [at all :c) ]) bobb Bob Basques bob.basq...@ci.stpaul.mn.us wrote: Landon, It would be interesting to come up with a standard structure on a computer file system that could be used to accessed tiled raster data, if this hasn’t been done already. One the file system structure was defined, it would be fairly easy to write open source software that accessed this structure and provided individual tiles as a service to desktop GIS applications. Hmm, interesting angle, to expand on your idea a bit more, what about a processing suite (or set of suites) that process data for different types of uses, visual display, DEM analysis, etc. Each processor stack would/could have it's own rules associated with data resolution vs files sizes, etc. bobb Landon Blake lbl...@ksninc.com wrote: Bobb wrote: “Here's my reasoning, we're never (ever?) going to hit the top end on how big files ever get, resolution just keeps going up and up, so there is always going to be some upper limit that will need to be breached somehow. Working out a proper method for segregating the data up front (dare I say it), as some sort of standard (which can be adjusted as time passes) will make everything work nicely, then all will work with available tools when they are available, if tools to handle larger datasets become available, and the communityI agree with some of the points you are making in your argument Bobb. There is certainly a practical limit to how much you data you should put in a single file. That is why we have lumber cut to 8 foot lengths. You don’t need a flatbed semi to carry it to your house. :] When you refer to a standard for splitting data up front, what do you mean? It would be interesting to come up with a standard structure on a computer file system that could be used to accessed tiled raster data, if this hasn’t been done already. One the file system structure was defined, it would be fairly easy to write open source software that accessed this structure and provided individual tiles as a service to desktop GIS applications. Landon From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Bob Basques Sent: Monday, August 24, 2009 7:33 AM To: OSGeo Discussions Subject: RE: [OSGeo-Discuss] Open File Formats and ProprietaryAlgorithms
RE: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms
To be clear, it's not an effort I have the bandwidth for personally. But if there were qualified developers interested in taking it on, I might be in a position to offer project guidance and a small amount of funding. -mpg -Original Message- From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Landon Blake Sent: Tuesday, August 25, 2009 11:12 AM To: OSGeo Discussions Subject: RE: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms OK. I'm not an expert on images, but I'd be interested in working with you. However, I avoid C++ like a lethal strain of the Swine Flu. :] I may give some more thought to some of Bob's ideas about making it easier to work with image tiles. Thanks. Landon Office Phone Number: (209) 946-0268 Cell Phone Number: (209) 992-0658 -Original Message- From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Michael P. Gerlek Sent: Monday, August 24, 2009 3:19 PM To: OSGeo Discussions Subject: RE: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms I've not given it much thought recently, to be honest. I'd need to review the current state of things in OpenJp2 (or whatever it's called) to see where they are at, what changes would be realistic and viable, how amenable they'd be to taking patches versus a fork, etc. Done properly, the work would have no geo specific component at all -- it would just be a new version of some of the internal algorithms. The test case would simply be to encode and decode an 500 GB(?) raw file on a box with 2 GB (?) of RAM. I would certainly not want anyone to have to build a whole new jp2 library from scratch, if that's what you meant. I'd really only be interested in C++ (or possibly mono-safe C#). -mpg -Original Message- From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Landon Blake Sent: Monday, August 24, 2009 4:11 PM To: OSGeo Discussions Subject: RE: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms MPG: When you say effort do you mean some sort of library to support JP2 geo side of things? What programming language would you be most interested in? C++? Landon Office Phone Number: (209) 946-0268 Cell Phone Number: (209) 992-0658 -Original Message- From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Michael P. Gerlek Sent: Monday, August 24, 2009 1:59 PM To: OSGeo Discussions Subject: RE: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms It would not be a good SoC thing, due to the level of expertise and time required. I (LizardTech) would likely be willing to contribute to such an effort. -mpg -Original Message- From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Fawcett, David Sent: Monday, August 24, 2009 8:11 AM To: OSGeo Discussions Subject: RE: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms I realize that there are likely not a large number of people who have the expertise and experience to write this kind of code. Is this a project that should be shopped around for funding? Google Summer of Code? A grant from our ~benevolent overlord Google? Some other foundation or org interested in open data formats? David. -Original Message- From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Michael P. Gerlek Sent: Thursday, August 20, 2009 4:36 PM To: OSGeo Discussions Subject: RE: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms snip Do you know why there hasn't been a broader adoption of JP2? Not through lack of trying on my part :-) I think the two biggest reasons are: (1) The algorithms for handling large images in memory really are rocket science, and no one in the FOSS community has gotten the itch sufficiently bad enough to go and do the work needed inside the existing open source packages. Hopefully someday someone will. ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss Warning: Information provided via electronic media is not guaranteed against defects including translation and transmission errors. If the reader is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this information in error, please notify the sender immediately. ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
Re: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms
Lots of discussions on standardization of data formats, lots of challenges there. One thing that has not been mentioned (I think) is the idea of standardizing responses from users about 1) uses of data and 2) limitations of data (files too big fits in here). Both of these are included in the MD_Usage object that is part of the ISO 19115 Standard included in GeoNetwork. Would be very cool to build into GeoNetwork the capability to accept user feedback about datasets and to associate that feedback with the appropriate datasets in the GeoNetwork catalog and to make it available to 1) data providers and 2) future users... Seems straightforward, Ted -- Ted Habermann === Enterprise Data Systems Group Leader NOAA, National Geophysical Data Center V: 303.497.6472 F: 303.497.6513 If you want to go quickly, go alone. If you want to go far, go together Old Proverb ted.haberm...@noaa.gov == ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
Re: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms
Fawcett, David wrote: I realize that there are likely not a large number of people who have the expertise and experience to write this kind of code. Is this a project that should be shopped around for funding? I'd say so. Google Summer of Code? IMHO, it definitely doesn't such timing/schedule. Best regards, -- Mateusz Loskot, http://mateusz.loskot.net ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
RE: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms
All, Ok, I'm probably going to get someone irritated, but here goes . . . Why not approach this from the other end of the spectrum and work at making the original files smaller. Work with the providers to make the images smaller in the first place, or at least come up with a maximum practical size to work with, I mean if this is the only (or biggest reason) for implementing JP2, then getting folks to make the smaller deliverables seems like a better long term approach. Here's my reasoning, we're never (ever?) going to hit the top end on how big files ever get, resolution just keeps going up and up, so there is always going to be some upper limit that will need to be breached somehow. Working out a proper method for segregating the data up front (dare I say it), as some sort of standard (which can be adjusted as time passes) will make everything work nicely, then all will work with available tools when they are available, if tools to handle larger datasets become available, and the community feels there is a reason/need that these new larger files need to be handled, then they get to change the standard. bobb Fawcett, David david.fawc...@state.mn.us wrote: I realize that there are likely not a large number of people who have the expertise and experience to write this kind of code. Is this a project that should be shopped around for funding? Google Summer of Code? A grant from our ~benevolent overlord Google? Some other foundation or org interested in open data formats? David. -Original Message- From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Michael P. Gerlek Sent: Thursday, August 20, 2009 4:36 PM To: OSGeo Discussions Subject: RE: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms snip Do you know why there hasn't been a broader adoption of JP2? Not through lack of trying on my part :-) I think the two biggest reasons are: (1) The algorithms for handling large images in memory really are rocket science, and no one in the FOSS community has gotten the itch sufficiently bad enough to go and do the work needed inside the existing open source packages. Hopefully someday someone will. ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
Re: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms
I agree, primarily because I just got a dataset from the city that was a 5Gb raster. I know hardrive space is cheap and so is processing power but still, it took literally hours to get anything meaningful out of it. Picking a more appropriate resolution, better compression and eventually switching file formats would have helped immensely but wasn't done since the prevailing attitude is that bigger is better. This attitude is really the same as in the programming world, where programs keep getting slower and slower (in terms of time complexity) but it's deemed okay since computers are also getting faster. I don't think this attitude is going to change any time soon though and making some form of standard would simply not work. How could we standardize what resolution and compression we should be using on specific datasets for specific applications? There are uses we haven't even thought up yet, how could we standardize for those future uses? Just my 0.02$ R Bob Basques wrote: All, Ok, I'm probably going to get someone irritated, but here goes . . . Why not approach this from the other end of the spectrum and work at making the original files smaller. Work with the providers to make the images smaller in the first place, or at least come up with a maximum practical size to work with, I mean if this is the only (or biggest reason) for implementing JP2, then getting folks to make the smaller deliverables seems like a better long term approach. Here's my reasoning, we're never (ever?) going to hit the top end on how big files ever get, resolution just keeps going up and up, so there is always going to be some upper limit that will need to be breached somehow. Working out a proper method for segregating the data up front (dare I say it), as some sort of standard (which can be adjusted as time passes) will make everything work nicely, then all will work with available tools when they are available, if tools to handle larger datasets become available, and the community feels there is a reason/need that these new larger files need to be handled, then they get to change the standard. bobb Fawcett, David david.fawc...@state.mn.us wrote: I realize that there are likely not a large number of people who have the expertise and experience to write this kind of code. Is this a project that should be shopped around for funding? Google Summer of Code? A grant from our ~benevolent overlord Google? Some other foundation or org interested in open data formats? David. -Original Message- From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Michael P. Gerlek Sent: Thursday, August 20, 2009 4:36 PM To: OSGeo Discussions Subject: RE: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms snip Do you know why there hasn't been a broader adoption of JP2? Not through lack of trying on my part :-) I think the two biggest reasons are: (1) The algorithms for handling large images in memory really are rocket science, and no one in the FOSS community has gotten the itch sufficiently bad enough to go and do the work needed inside the existing open source packages. Hopefully someday someone will. ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
RE: [OSGeo-Discuss] Open File Formats and ProprietaryAlgorithms
Bobb wrote: Here's my reasoning, we're never (ever?) going to hit the top end on how big files ever get, resolution just keeps going up and up, so there is always going to be some upper limit that will need to be breached somehow. Working out a proper method for segregating the data up front (dare I say it), as some sort of standard (which can be adjusted as time passes) will make everything work nicely, then all will work with available tools when they are available, if tools to handle larger datasets become available, and the community feels there is a reason/need that these new larger files need to be handled, then they get to change the standard. I agree with some of the points you are making in your argument Bobb. There is certainly a practical limit to how much you data you should put in a single file. That is why we have lumber cut to 8 foot lengths. You don't need a flatbed semi to carry it to your house. :] When you refer to a standard for splitting data up front, what do you mean? It would be interesting to come up with a standard structure on a computer file system that could be used to accessed tiled raster data, if this hasn't been done already. One the file system structure was defined, it would be fairly easy to write open source software that accessed this structure and provided individual tiles as a service to desktop GIS applications. Landon From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Bob Basques Sent: Monday, August 24, 2009 7:33 AM To: OSGeo Discussions Subject: RE: [OSGeo-Discuss] Open File Formats and ProprietaryAlgorithms All, Ok, I'm probably going to get someone irritated, but here goes . . . Why not approach this from the other end of the spectrum and work at making the original files smaller. Work with the providers to make the images smaller in the first place, or at least come up with a maximum practical size to work with, I mean if this is the only (or biggest reason) for implementing JP2, then getting folks to make the smaller deliverables seems like a better long term approach. Here's my reasoning, we're never (ever?) going to hit the top end on how big files ever get, resolution just keeps going up and up, so there is always going to be some upper limit that will need to be breached somehow. Working out a proper method for segregating the data up front (dare I say it), as some sort of standard (which can be adjusted as time passes) will make everything work nicely, then all will work with available tools when they are available, if tools to handle larger datasets become available, and the community feels there is a reason/need that these new larger files need to be handled, then they get to change the standard. bobb Fawcett, David david.fawc...@state.mn.us wrote: I realize that there are likely not a large number of people who have the expertise and experience to write this kind of code. Is this a project that should be shopped around for funding? Google Summer of Code? A grant from our ~benevolent overlord Google? Some other foundation or org interested in open data formats? David. -Original Message- From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Michael P. Gerlek Sent: Thursday, August 20, 2009 4:36 PM To: OSGeo Discussions Subject: RE: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms snip Do you know why there hasn't been a broader adoption of JP2? Not through lack of trying on my part :-) I think the two biggest reasons are: (1) The algorithms for handling large images in memory really are rocket science, and no one in the FOSS community has gotten the itch sufficiently bad enough to go and do the work needed inside the existing open source packages. Hopefully someday someone will. ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss Warning: Information provided via electronic media is not guaranteed against defects including translation and transmission errors. If the reader is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this information in error, please notify the sender immediately.___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
Re: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms
Rene, how could we standardize for those future uses? I was thinking more along the lines of a standard file size more than anything. Nto all deliverable are even able to accomplish capturing a whole contract in a single file, so if even a separation into more than one file is needed, why not come up with some form of sizing standard, I realize that there are uses other than those that I have, but some form of data tiling seems appropriate with this discussion. bobb René A. Enguehard ahugen...@gmail.com wrote: I agree, primarily because I just got a dataset from the city that was a 5Gb raster. I know hardrive space is cheap and so is processing power but still, it took literally hours to get anything meaningful out of it. Picking a more appropriate resolution, better compression and eventually switching file formats would have helped immensely but wasn't done since the prevailing attitude is that bigger is better. This attitude is really the same as in the programming world, where programs keep getting slower and slower (in terms of time complexity) but it's deemed okay since computers are also getting faster. I don't think this attitude is going to change any time soon though and making some form of standard would simply not work. How could we standardize what resolution and compression we should be using on specific datasets for specific applications? There are uses we haven't even thought up yet, how could we standardize for those future uses? Just my 0.02$ R Bob Basques wrote: All, Ok, I'm probably going to get someone irritated, but here goes . . . Why not approach this from the other end of the spectrum and work at making the original files smaller. Work with the providers to make the images smaller in the first place, or at least come up with a maximum practical size to work with, I mean if this is the only (or biggest reason) for implementing JP2, then getting folks to make the smaller deliverables seems like a better long term approach. Here's my reasoning, we're never (ever?) going to hit the top end on how big files ever get, resolution just keeps going up and up, so there is always going to be some upper limit that will need to be breached somehow. Working out a proper method for segregating the data up front (dare I say it), as some sort of standard (which can be adjusted as time passes) will make everything work nicely, then all will work with available tools when they are available, if tools to handle larger datasets become available, and the community feels there is a reason/need that these new larger files need to be handled, then they get to change the standard. bobb Fawcett, David david.fawc...@state.mn.us wrote: I realize that there are likely not a large number of people who have the expertise and experience to write this kind of code. Is this a project that should be shopped around for funding? Google Summer of Code? A grant from our ~benevolent overlord Google? Some other foundation or org interested in open data formats? David. -Original Message- From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Michael P. Gerlek Sent: Thursday, August 20, 2009 4:36 PM To: OSGeo Discussions Subject: RE: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms snip Do you know why there hasn't been a broader adoption of JP2? Not through lack of trying on my part :-) I think the two biggest reasons are: (1) The algorithms for handling large images in memory really are rocket science, and no one in the FOSS community has gotten the itch sufficiently bad enough to go and do the work needed inside the existing open source packages. Hopefully someday someone will. ___ Discuss mailing list Discuss@lists.osgeo.org http://lists. Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
RE: [OSGeo-Discuss] Open File Formats and ProprietaryAlgorithms
Landon, It would be interesting to come up with a standard structure on a computer file system that could be used to accessed tiled raster data, if this hasn’t been done already. One the file system structure was defined, it would be fairly easy to write open source software that accessed this structure and provided individual tiles as a service to desktop GIS applications. Hmm, interesting angle, to expand on your idea a bit more, what about a processing suite (or set of suites) that process data for different types of uses, visual display, DEM analysis, etc. Each processor stack would/could have it's own rules associated with data resolution vs files sizes, etc. bobb Landon Blake lbl...@ksninc.com wrote: Bobb wrote: “Here's my reasoning, we're never (ever?) going to hit the top end on how big files ever get, resolution just keeps going up and up, so there is always going to be some upper limit that will need to be breached somehow. Working out a proper method for segregating the data up front (dare I say it), as some sort of standard (which can be adjusted as time passes) will make everything work nicely, then all will work with available tools when they are available, if tools to handle larger datasets become available, and the community feels there is a reason/need that these new larger files need to be handled, then they get to change the standard.” I agree with some of the points you are making in your argument Bobb. There is certainly a practical limit to how much you data you should put in a single file. That is why we have lumber cut to 8 foot lengths. You don’t need a flatbed semi to carry it to your house. :] When you refer to a standard for splitting data up front, what do you mean? It would be interesting to come up with a standard structure on a computer file system that could be used to accessed tiled raster data, if this hasn’t been done already. One the file system structure was defined, it would be fairly easy to write open source software that accessed this structure and provided individual tiles as a service to desktop GIS applications. Landon From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Bob Basques Sent: Monday, August 24, 2009 7:33 AM To: OSGeo Discussions Subject: RE: [OSGeo-Discuss] Open File Formats and ProprietaryAlgorithms All, Ok, I'm probably going to get someone irritated, but here goes . . Why not approach this from the other end of the spectrum and work at making the original files smaller. Work with the providers to make the images smaller in the first place, or at least come up with a maximum practical size to work with, I mean if this is the only (or biggest reason) for implementing JP2, then getting folks to make the smaller deliverables seems like a better long term approach. Here's my reasoning, we're never (ever?) going to hit the top end on how big files ever get, resolution just keeps going up and up, so there is always going to be some upper limit that will need to be breached somehow. Working out a proper method for segregating the data up front (dare I say it), as some sort of standard (which can be adjusted as time passes) will make everything work nicely, then all will work with available tools when they are available, if tools to handle larger datasets become available, and the community feels there is a reason/need that these new larger files need to be handled, then they get to change the standard. bobb Fawcett, David david.fawc...@state.mn.us wrote: I realize that there are likely not a large number of people who have the expertise and experience to write this kind of code. Is this a project that should be shopped around for funding? Google Summer of Code? A grant from our ~benevolent overlord Google? Some[mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Michael P. Gerlek Sent: Thursday, August 20, 2009 4:36 PM To: OSGeo Discussions Subject: RE: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms snip Do you know why there hasn't been a broader adoption of JP2? Not through lack of trying on my part :-) I think the two biggest reasons are: (1) The algorithms for handling large images in memory really are rocket science, and no one in the FOSS community has gotten the itch sufficiently bad enough to go and do the work needed inside the existing open source packages. Hopefully someday someone will. ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss Warning: Information provided via electronic media is not guaranteed against defects including translation and transmission errors. If the reader is not the intended recipient
RE: [OSGeo-Discuss] Open File Formats andProprietaryAlgorithms
Landon, Just had another thought . . . What about setting up a (openSource) tool set specifically for handling Raster images for pre-processing purposes. Might even be something that publishers could re-distribute with their datasets, as in this processor stack works with our data. Just thinking on the run here, no detail (or ramifications though through [at all :c) ]) bobb Bob Basques bob.basq...@ci.stpaul.mn.us wrote: Landon, It would be interesting to come up with a standard structure on a computer file system that could be used to accessed tiled raster data, if this hasn’t been done already. One the file system structure was defined, it would be fairly easy to write open source software that accessed this structure and provided individual tiles as a service to desktop GIS applications. Hmm, interesting angle, to expand on your idea a bit more, what about a processing suite (or set of suites) that process data for different types of uses, visual display, DEM analysis, etc. Each processor stack would/could have it's own rules associated with data resolution vs files sizes, etc. bobb Landon Blake lbl...@ksninc.com wrote: Bobb wrote: “Here's my reasoning, we're never (ever?) going to hit the top end on how big files ever get, resolution just keeps going up and up, so there is always going to be some upper limit that will need to be breached somehow. Working out a proper method for segregating the data up front (dare I say it), as some sort of standard (which can be adjusted as time passes) will make everything work nicely, then all will work with available tools when they are available, if tools to handle larger datasets become available, and the community feels there is a reason/need that these new larger files need to be handled, then they get to change the standard.” I agree with some of the points you are making in your argument Bobb. There is certainly a practical limit to how much you data you should put in a single file. That is why we have lumber cut to 8 foot lengths. You don’t need a flatbed semi to carry it to your house. :] When you refer to a standard for splitting data up front, what do you mean? It would be interesting to come up with a standard structure on a computer file system that could be used to accessed tiled raster data, if this hasn’t been done already. One the file system structure was defined, it would be fairly easy to write open source software that accessed this structure and provided individual tiles as a service to desktop GIS applications. Landon From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Bob Basques Sent: Monday, August 24, 2009 7:33 AM To: OSGeo Discussions Subject: RE: [OSGeo-Discuss] Open File Formats and ProprietaryAlgorithms All, Ok, I'm probably going to get someone irritated, but here goes . . Why not approach this from the other end of the spectrum and work at making the original files smaller. Work with the providers to make the images smaller in the first place, or at least come up with a maximum practical size to work with, I mean if this is the only (or biggest reason) for implementing JP2, then getting folks to make the smaller deliverables seems like a better long term approach. Here's my reasoning, we're never (ever?) going to hit the top end on how big files ever get, resolution just keeps going up and up, so there is always going to be some upper limit that will need to be breached somehow. Working out a proper method for segregating the data up front (dare I say it), as some sort of standard (which can be adjusted as time passes) will make everything work nicely, then all reason/need that these new larger files need to be handled, then they get to change the standard. bobb Fawcett, David david.fawc...@state.mn.us wrote: I realize that there are likely not a large number of people who have the expertise and experience to write this kind of code. Is this a project that should be shopped around for funding? Google Summer of Code? A grant from our ~benevolent overlord Google? Some other foundation or org interested in open data formats? David. -Original Message- From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Michael P. Gerlek Sent: Thursday, August 20, 2009 4:36 PM To: OSGeo Discussions Subject: RE: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms snip Do you know why there hasn't been a broader adoption of JP2? Not through lack of trying on my part :-) I think the two biggest reasons are: (1) The algorithms for handling large images in memory really are rocket science, and no one in the FOSS community has gotten the itch
Re: [OSGeo-Discuss] Open File Formats andProprietaryAlgorithms
Markus, What's the standard (OGC?) part of this, just the calling structure, and consequent responses? It looks like anything can be on the backenda s the processor, and the only enforcements are in the calling and results for the getcapabilities, maybe in the service(s) requests as well . . . Still looking . . . bobb Markus Neteler nete...@osgeo.org wrote: On Mon, Aug 24, 2009 at 6:20 PM, Bob Basquesbob.basq...@ci.stpaul.mn.us wrote: Landon, Just had another thought . . . What about setting up a (openSource) tool set specifically for handling Raster images for pre-processing purposes. Might even be something that publishers could re-distribute with their datasets, as in this processor stack works with our data. Try this :) http://pywps.wald.intevation.org/ Markus ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
RE: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms
It would not be a good SoC thing, due to the level of expertise and time required. I (LizardTech) would likely be willing to contribute to such an effort. -mpg -Original Message- From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Fawcett, David Sent: Monday, August 24, 2009 8:11 AM To: OSGeo Discussions Subject: RE: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms I realize that there are likely not a large number of people who have the expertise and experience to write this kind of code. Is this a project that should be shopped around for funding? Google Summer of Code? A grant from our ~benevolent overlord Google? Some other foundation or org interested in open data formats? David. -Original Message- From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Michael P. Gerlek Sent: Thursday, August 20, 2009 4:36 PM To: OSGeo Discussions Subject: RE: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms snip Do you know why there hasn't been a broader adoption of JP2? Not through lack of trying on my part :-) I think the two biggest reasons are: (1) The algorithms for handling large images in memory really are rocket science, and no one in the FOSS community has gotten the itch sufficiently bad enough to go and do the work needed inside the existing open source packages. Hopefully someday someone will. ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
RE: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms
MPG: When you say effort do you mean some sort of library to support JP2 geo side of things? What programming language would you be most interested in? C++? Landon Office Phone Number: (209) 946-0268 Cell Phone Number: (209) 992-0658 -Original Message- From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Michael P. Gerlek Sent: Monday, August 24, 2009 1:59 PM To: OSGeo Discussions Subject: RE: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms It would not be a good SoC thing, due to the level of expertise and time required. I (LizardTech) would likely be willing to contribute to such an effort. -mpg -Original Message- From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Fawcett, David Sent: Monday, August 24, 2009 8:11 AM To: OSGeo Discussions Subject: RE: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms I realize that there are likely not a large number of people who have the expertise and experience to write this kind of code. Is this a project that should be shopped around for funding? Google Summer of Code? A grant from our ~benevolent overlord Google? Some other foundation or org interested in open data formats? David. -Original Message- From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Michael P. Gerlek Sent: Thursday, August 20, 2009 4:36 PM To: OSGeo Discussions Subject: RE: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms snip Do you know why there hasn't been a broader adoption of JP2? Not through lack of trying on my part :-) I think the two biggest reasons are: (1) The algorithms for handling large images in memory really are rocket science, and no one in the FOSS community has gotten the itch sufficiently bad enough to go and do the work needed inside the existing open source packages. Hopefully someday someone will. ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss Warning: Information provided via electronic media is not guaranteed against defects including translation and transmission errors. If the reader is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this information in error, please notify the sender immediately. ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
RE: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms
I've not given it much thought recently, to be honest. I'd need to review the current state of things in OpenJp2 (or whatever it's called) to see where they are at, what changes would be realistic and viable, how amenable they'd be to taking patches versus a fork, etc. Done properly, the work would have no geo specific component at all -- it would just be a new version of some of the internal algorithms. The test case would simply be to encode and decode an 500 GB(?) raw file on a box with 2 GB (?) of RAM. I would certainly not want anyone to have to build a whole new jp2 library from scratch, if that's what you meant. I'd really only be interested in C++ (or possibly mono-safe C#). -mpg -Original Message- From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Landon Blake Sent: Monday, August 24, 2009 4:11 PM To: OSGeo Discussions Subject: RE: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms MPG: When you say effort do you mean some sort of library to support JP2 geo side of things? What programming language would you be most interested in? C++? Landon Office Phone Number: (209) 946-0268 Cell Phone Number: (209) 992-0658 -Original Message- From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Michael P. Gerlek Sent: Monday, August 24, 2009 1:59 PM To: OSGeo Discussions Subject: RE: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms It would not be a good SoC thing, due to the level of expertise and time required. I (LizardTech) would likely be willing to contribute to such an effort. -mpg -Original Message- From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Fawcett, David Sent: Monday, August 24, 2009 8:11 AM To: OSGeo Discussions Subject: RE: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms I realize that there are likely not a large number of people who have the expertise and experience to write this kind of code. Is this a project that should be shopped around for funding? Google Summer of Code? A grant from our ~benevolent overlord Google? Some other foundation or org interested in open data formats? David. -Original Message- From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Michael P. Gerlek Sent: Thursday, August 20, 2009 4:36 PM To: OSGeo Discussions Subject: RE: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms snip Do you know why there hasn't been a broader adoption of JP2? Not through lack of trying on my part :-) I think the two biggest reasons are: (1) The algorithms for handling large images in memory really are rocket science, and no one in the FOSS community has gotten the itch sufficiently bad enough to go and do the work needed inside the existing open source packages. Hopefully someday someone will. ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss Warning: Information provided via electronic media is not guaranteed against defects including translation and transmission errors. If the reader is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this information in error, please notify the sender immediately. ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
Re: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms [SEC=UNCLASSIFIED]
Bruce, For your information, I expect to be talking to Chris, from the National Archives Australia this coming week. Apparently they store their data sets in Open formats, but don't know how to store GIS datasets in open formats. I'm hoping that we can help. Bruce Bannerman wrote: IMO: Just another thought on this issue (though we do seem to be recycling arguments over the years...): Assuming that I have a very large archive of spatial data, be it imagery or any other spatial format and that I store my data in a variety of proprietary formats: In ten years from now, can I be sure that: - the company that created, understands, and holds the IP in the data format will still be around? - there will still be software that runs on the then current operating environment, that can read and 'fully exploit' the data in the proprietary standard? - that this future software will work seamlessly with my then current spatial environment? - if all of the above risks prove to eventuate, can I be sure that I'll be able to salvage my data into another format, retaining its complete semantic context? IMO, it is a high risk proposition to lock public (or private) archives away in proprietary data formats. It makes more sense to use open standards and formats that are publically available. Bruce Bannerman -Original Message- From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Michael P. Gerlek Sent: Friday, 21 August 2009 6:55 AM To: OSGeo Discussions Subject: RE: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms Some clarifications: - MrSID has both lossy and lossless modes - MrSID is not fractal based; it uses wavelets (and arithmetic encoding) - you can't copyright algorithms; the MrSID source code certainly is, however - MrSID relies on a number of patents, not all of which are owned by LizardTech - reading MrSID does not require any fees; we have libraries you can download, although they are not open source That said, some editorial comments (although I'm now wishing I hadn't been so quick to rise to Landon's bait :-) - Some of you know the history of trying to open source MrSID; I won't go into that here, except to say that LizardTech doesn't own all of the required IP needed to make that happen. - If we are speaking of the NAIP data, then no, it is not exclusively available in MrSID format; it is also shipped as GeoTIFFs. - JPEG 2000 is a very robust open standard alternative to MrSID, and a number of players already support it (including LizardTech), but not enough to make it viable for certain domains like NAIP. - some of you also know the history on open JP2 support: there is today no open source implementation of JP2 that is suitable for geo work. Alas. -mpg From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Eric Wolf Sent: Thursday, August 20, 2009 2:15 PM To: OSGeo Discussions Subject: Re: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms The MRSID format is a very special case - and perhaps an opportunity for a new FOSS file format. MRSID is a lossless, fractal-based, multi-scale raster compression format. LizardTech has the algorithms to encode and decode MRSID locked up in copyrights, and I believe, patents. Even companies like ESRI shell out big bucks to LizardTech to be able to read and write the MRSID format. I guess I missed the context of the discussion. Is the government releasing certain data exclusively in this format? If so, I think the argument can be made against this practice. The different in compression between MRSID and gziped TIFFs isn't really that great in this day of cheap disks and fat pipes. -Eric -=--=---===---=--=-=--=---==---=--=-=- Eric B. WolfNew! 720-334-7734 USGS Geographer Center of Excellence in GIScience PhD Student CU-Boulder - Geography ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss -- Cameron Shorter Geospatial Systems Architect Tel: +61 (0)2 8570 5050 Mob: +61 (0)419 142 254 Think Globally, Fix Locally Geospatial Solutions enhanced with Open Standards and Open Source http://www.lisasoft.com ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
RE: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms[SEC=UNCLASSIFIED]
All, Opticks is an open source remote sensing application and development framework. We recently started the process to add JPEG 2000 support to our framework. We picked OpenJpeg to add JPEG 2000 support to our application. They are also open source. We currently support importing JPEG 2000 files but we are currently limited to the 4GB memory size after decoding. Our plan is to continue development and to upgrade to OpenJpeg 2.0 once they have a stable release. That will allow Opticks to use a pager to display and support much larger files. Michael Considine -Original Message- From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Bruce Bannerman Sent: Thursday, August 20, 2009 8:15 PM To: 'OSGeo Discussions' Subject: RE: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms[SEC=UNCLASSIFIED] IMO: Just another thought on this issue (though we do seem to be recycling arguments over the years...): Assuming that I have a very large archive of spatial data, be it imagery or any other spatial format and that I store my data in a variety of proprietary formats: In ten years from now, can I be sure that: - the company that created, understands, and holds the IP in the data format will still be around? - there will still be software that runs on the then current operating environment, that can read and 'fully exploit' the data in the proprietary standard? - that this future software will work seamlessly with my then current spatial environment? - if all of the above risks prove to eventuate, can I be sure that I'll be able to salvage my data into another format, retaining its complete semantic context? IMO, it is a high risk proposition to lock public (or private) archives away in proprietary data formats. It makes more sense to use open standards and formats that are publically available. Bruce Bannerman -Original Message- From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Michael P. Gerlek Sent: Friday, 21 August 2009 6:55 AM To: OSGeo Discussions Subject: RE: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms Some clarifications: - MrSID has both lossy and lossless modes - MrSID is not fractal based; it uses wavelets (and arithmetic encoding) - you can't copyright algorithms; the MrSID source code certainly is, however - MrSID relies on a number of patents, not all of which are owned by LizardTech - reading MrSID does not require any fees; we have libraries you can download, although they are not open source That said, some editorial comments (although I'm now wishing I hadn't been so quick to rise to Landon's bait :-) - Some of you know the history of trying to open source MrSID; I won't go into that here, except to say that LizardTech doesn't own all of the required IP needed to make that happen. - If we are speaking of the NAIP data, then no, it is not exclusively available in MrSID format; it is also shipped as GeoTIFFs. - JPEG 2000 is a very robust open standard alternative to MrSID, and a number of players already support it (including LizardTech), but not enough to make it viable for certain domains like NAIP. - some of you also know the history on open JP2 support: there is today no open source implementation of JP2 that is suitable for geo work. Alas. -mpg From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Eric Wolf Sent: Thursday, August 20, 2009 2:15 PM To: OSGeo Discussions Subject: Re: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms The MRSID format is a very special case - and perhaps an opportunity for a new FOSS file format. MRSID is a lossless, fractal-based, multi-scale raster compression format. LizardTech has the algorithms to encode and decode MRSID locked up in copyrights, and I believe, patents. Even companies like ESRI shell out big bucks to LizardTech to be able to read and write the MRSID format. I guess I missed the context of the discussion. Is the government releasing certain data exclusively in this format? If so, I think the argument can be made against this practice. The different in compression between MRSID and gziped TIFFs isn't really that great in this day of cheap disks and fat pipes. -Eric -=--=---===---=--=-=--=---==---=--=-=- Eric B. WolfNew! 720-334-7734 USGS Geographer Center of Excellence in GIScience PhD Student CU-Boulder - Geography ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss This message and any enclosures are intended only for the addressee. Please notify the sender by email if you are not the intended recipient. If you are not the intended
RE: [OSGeo-Discuss] Open File Formats and ProprietaryAlgorithms[SEC=UNCLASSIFIED]
Thanks for the information Michael. I am downloading Opticks right now. :] I also found this Java library for JP2, thought I'm not sure how complete/up-to-date it is: http://jj2000.epfl.ch/ Maybe we need a JPEG 2000 page on the OSGeo wiki. Landon Office Phone Number: (209) 946-0268 Cell Phone Number: (209) 992-0658 -Original Message- From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Considine, Michael Sent: Friday, August 21, 2009 8:09 AM To: OSGeo Discussions Subject: RE: [OSGeo-Discuss] Open File Formats and ProprietaryAlgorithms[SEC=UNCLASSIFIED] All, Opticks is an open source remote sensing application and development framework. We recently started the process to add JPEG 2000 support to our framework. We picked OpenJpeg to add JPEG 2000 support to our application. They are also open source. We currently support importing JPEG 2000 files but we are currently limited to the 4GB memory size after decoding. Our plan is to continue development and to upgrade to OpenJpeg 2.0 once they have a stable release. That will allow Opticks to use a pager to display and support much larger files. Michael Considine -Original Message- From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Bruce Bannerman Sent: Thursday, August 20, 2009 8:15 PM To: 'OSGeo Discussions' Subject: RE: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms[SEC=UNCLASSIFIED] IMO: Just another thought on this issue (though we do seem to be recycling arguments over the years...): Assuming that I have a very large archive of spatial data, be it imagery or any other spatial format and that I store my data in a variety of proprietary formats: In ten years from now, can I be sure that: - the company that created, understands, and holds the IP in the data format will still be around? - there will still be software that runs on the then current operating environment, that can read and 'fully exploit' the data in the proprietary standard? - that this future software will work seamlessly with my then current spatial environment? - if all of the above risks prove to eventuate, can I be sure that I'll be able to salvage my data into another format, retaining its complete semantic context? IMO, it is a high risk proposition to lock public (or private) archives away in proprietary data formats. It makes more sense to use open standards and formats that are publically available. Bruce Bannerman -Original Message- From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Michael P. Gerlek Sent: Friday, 21 August 2009 6:55 AM To: OSGeo Discussions Subject: RE: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms Some clarifications: - MrSID has both lossy and lossless modes - MrSID is not fractal based; it uses wavelets (and arithmetic encoding) - you can't copyright algorithms; the MrSID source code certainly is, however - MrSID relies on a number of patents, not all of which are owned by LizardTech - reading MrSID does not require any fees; we have libraries you can download, although they are not open source That said, some editorial comments (although I'm now wishing I hadn't been so quick to rise to Landon's bait :-) - Some of you know the history of trying to open source MrSID; I won't go into that here, except to say that LizardTech doesn't own all of the required IP needed to make that happen. - If we are speaking of the NAIP data, then no, it is not exclusively available in MrSID format; it is also shipped as GeoTIFFs. - JPEG 2000 is a very robust open standard alternative to MrSID, and a number of players already support it (including LizardTech), but not enough to make it viable for certain domains like NAIP. - some of you also know the history on open JP2 support: there is today no open source implementation of JP2 that is suitable for geo work. Alas. -mpg From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Eric Wolf Sent: Thursday, August 20, 2009 2:15 PM To: OSGeo Discussions Subject: Re: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms The MRSID format is a very special case - and perhaps an opportunity for a new FOSS file format. MRSID is a lossless, fractal-based, multi-scale raster compression format. LizardTech has the algorithms to encode and decode MRSID locked up in copyrights, and I believe, patents. Even companies like ESRI shell out big bucks to LizardTech to be able to read and write the MRSID format. I guess I missed the context of the discussion. Is the government releasing certain data exclusively in this format? If so, I think the argument can be made against this practice. The different in compression between MRSID and gziped
Re: [OSGeo-Discuss] Open File Formats and ProprietaryAlgorithms[SEC=UNCLASSIFIED]
Well, according to this page: http://jpeg2000.epfl.ch/ v.5.1, courtesy in part Eastman Kodak, provides complete JP2 support at the decoding side - not sure whether that covers the tiling or other geo needs, but doesn't it sound worth investigating? Chris - Christopher Schmidt crschm...@crschmidt.net wrote: On Fri, Aug 21, 2009 at 08:27:13AM -0700, Landon Blake wrote: Thanks for the information Michael. I am downloading Opticks right now. :] I also found this Java library for JP2, thought I'm not sure how complete/up-to-date it is: http://jj2000.epfl.ch/ Maybe we need a JPEG 2000 page on the OSGeo wiki. Note that JPEG 2000 support is different from JPEG 2000 support which works on geo-sized images. The tiling (or 'paging'? as Michael calls it) support that's supposed to be provided by OpenJPEG2000 has been coming 'real soon now' for about 18 months now from my uneducated observations, and until it's there, most tools using OpenJPEG for JP2s are going to suffering under much the same limitations. -- Chris Landon Office Phone Number: (209) 946-0268 Cell Phone Number: (209) 992-0658 -Original Message- From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Considine, Michael Sent: Friday, August 21, 2009 8:09 AM To: OSGeo Discussions Subject: RE: [OSGeo-Discuss] Open File Formats and ProprietaryAlgorithms[SEC=UNCLASSIFIED] All, Opticks is an open source remote sensing application and development framework. We recently started the process to add JPEG 2000 support to our framework. We picked OpenJpeg to add JPEG 2000 support to our application. They are also open source. We currently support importing JPEG 2000 files but we are currently limited to the 4GB memory size after decoding. Our plan is to continue development and to upgrade to OpenJpeg 2.0 once they have a stable release. That will allow Opticks to use a pager to display and support much larger files. Michael Considine -Original Message- From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Bruce Bannerman Sent: Thursday, August 20, 2009 8:15 PM To: 'OSGeo Discussions' Subject: RE: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms[SEC=UNCLASSIFIED] IMO: Just another thought on this issue (though we do seem to be recycling arguments over the years...): Assuming that I have a very large archive of spatial data, be it imagery or any other spatial format and that I store my data in a variety of proprietary formats: In ten years from now, can I be sure that: - the company that created, understands, and holds the IP in the data format will still be around? - there will still be software that runs on the then current operating environment, that can read and 'fully exploit' the data in the proprietary standard? - that this future software will work seamlessly with my then current spatial environment? - if all of the above risks prove to eventuate, can I be sure that I'll be able to salvage my data into another format, retaining its complete semantic context? IMO, it is a high risk proposition to lock public (or private) archives away in proprietary data formats. It makes more sense to use open standards and formats that are publically available. Bruce Bannerman -Original Message- From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Michael P. Gerlek Sent: Friday, 21 August 2009 6:55 AM To: OSGeo Discussions Subject: RE: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms Some clarifications: - MrSID has both lossy and lossless modes - MrSID is not fractal based; it uses wavelets (and arithmetic encoding) - you can't copyright algorithms; the MrSID source code certainly is, however - MrSID relies on a number of patents, not all of which are owned by LizardTech - reading MrSID does not require any fees; we have libraries you can download, although they are not open source That said, some editorial comments (although I'm now wishing I hadn't been so quick to rise to Landon's bait :-) - Some of you know the history of trying to open source MrSID; I won't go into that here, except to say that LizardTech doesn't own all of the required IP needed to make that happen. - If we are speaking of the NAIP data, then no, it is not exclusively available in MrSID format; it is also shipped as GeoTIFFs. - JPEG 2000 is a very robust open standard alternative to MrSID, and a number of players already support it (including LizardTech), but not enough to make it viable for certain domains
RE: [OSGeo-Discuss] Open File Formats and ProprietaryAlgorithms[SEC=UNCLASSIFIED]
Tiling essentially means you can take a large file and compress pieces of it independently. This avoids having to deal with the large memory footprint issues, but it can also lead to seam-line artifacts under certain conditions. Ideally, one would prefer to have the option of compressing large images without resorting to using tiles. Note too that, in addition to the large image issue, many of the JP2 implementations out there are either not fully compliant or are not tuned for performance. A viable solution would need both of these. -mpg -Original Message- From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Chris Puttick Sent: Friday, August 21, 2009 9:37 AM To: OSGeo Discussions Subject: Re: [OSGeo-Discuss] Open File Formats and ProprietaryAlgorithms[SEC=UNCLASSIFIED] Well, according to this page: http://jpeg2000.epfl.ch/ v.5.1, courtesy in part Eastman Kodak, provides complete JP2 support at the decoding side - not sure whether that covers the tiling or other geo needs, but doesn't it sound worth investigating? Chris - Christopher Schmidt crschm...@crschmidt.net wrote: On Fri, Aug 21, 2009 at 08:27:13AM -0700, Landon Blake wrote: Thanks for the information Michael. I am downloading Opticks right now. :] I also found this Java library for JP2, thought I'm not sure how complete/up-to-date it is: http://jj2000.epfl.ch/ Maybe we need a JPEG 2000 page on the OSGeo wiki. Note that JPEG 2000 support is different from JPEG 2000 support which works on geo-sized images. The tiling (or 'paging'? as Michael calls it) support that's supposed to be provided by OpenJPEG2000 has been coming 'real soon now' for about 18 months now from my uneducated observations, and until it's there, most tools using OpenJPEG for JP2s are going to suffering under much the same limitations. -- Chris Landon Office Phone Number: (209) 946-0268 Cell Phone Number: (209) 992-0658 -Original Message- From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Considine, Michael Sent: Friday, August 21, 2009 8:09 AM To: OSGeo Discussions Subject: RE: [OSGeo-Discuss] Open File Formats and ProprietaryAlgorithms[SEC=UNCLASSIFIED] All, Opticks is an open source remote sensing application and development framework. We recently started the process to add JPEG 2000 support to our framework. We picked OpenJpeg to add JPEG 2000 support to our application. They are also open source. We currently support importing JPEG 2000 files but we are currently limited to the 4GB memory size after decoding. Our plan is to continue development and to upgrade to OpenJpeg 2.0 once they have a stable release. That will allow Opticks to use a pager to display and support much larger files. Michael Considine -Original Message- From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Bruce Bannerman Sent: Thursday, August 20, 2009 8:15 PM To: 'OSGeo Discussions' Subject: RE: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms[SEC=UNCLASSIFIED] IMO: Just another thought on this issue (though we do seem to be recycling arguments over the years...): Assuming that I have a very large archive of spatial data, be it imagery or any other spatial format and that I store my data in a variety of proprietary formats: In ten years from now, can I be sure that: - the company that created, understands, and holds the IP in the data format will still be around? - there will still be software that runs on the then current operating environment, that can read and 'fully exploit' the data in the proprietary standard? - that this future software will work seamlessly with my then current spatial environment? - if all of the above risks prove to eventuate, can I be sure that I'll be able to salvage my data into another format, retaining its complete semantic context? IMO, it is a high risk proposition to lock public (or private) archives away in proprietary data formats. It makes more sense to use open standards and formats that are publically available. Bruce Bannerman -Original Message- From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Michael P. Gerlek Sent: Friday, 21 August 2009 6:55 AM To: OSGeo Discussions Subject: RE: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms Some clarifications: - MrSID has both lossy and lossless modes - MrSID is not fractal based; it uses wavelets (and arithmetic encoding) - you can't copyright algorithms; the MrSID source code certainly is, however - MrSID relies on a number of patents, not all
RE: [OSGeo-Discuss] Open File Formats andProprietaryAlgorithms[SEC=UNCLASSIFIED]
MPG wrote: Tiling essentially means you can take a large file and compress pieces of it independently. This avoids having to deal with the large memory footprint issues, but it can also lead to seam-line artifacts under certain conditions. Ideally, one would prefer to have the option of compressing large images without resorting to using tiles. This is probably a stupid question, since I know absolutely nothing about image compression, but couldn't you overlap the tiles slightly to avoid the seam lines? This would obviously result in a slightly larger file size because some pixels would be compressed twice. But that might be OK if you were trying to compress a huge image. What about reading chunks of the image off disk, instead of trying to put the whole image in memory? This would be slower, but might make an impossible task possible. We run into this problem with vector datasets to. Some datasets are just to stinking BIG. One of my tasks for OpenJUMP is to write a core module that displays vector data accessed directly from disk, instead of from memory. This will be slower, but it is better than crashing the program because there isn't enough RAM. Things must be more complicated than can be described in an e-mail, because we've got people a lot smarter than me working on these problems. I am just curious. (I tried reading about wavelet compression on Wikipedia yesterday and quickly got a headache.) :] Landon Office Phone Number: (209) 946-0268 Cell Phone Number: (209) 992-0658 -Original Message- From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Michael P. Gerlek Sent: Friday, August 21, 2009 9:36 AM To: OSGeo Discussions Subject: RE: [OSGeo-Discuss] Open File Formats andProprietaryAlgorithms[SEC=UNCLASSIFIED] Tiling essentially means you can take a large file and compress pieces of it independently. This avoids having to deal with the large memory footprint issues, but it can also lead to seam-line artifacts under certain conditions. Ideally, one would prefer to have the option of compressing large images without resorting to using tiles. Note too that, in addition to the large image issue, many of the JP2 implementations out there are either not fully compliant or are not tuned for performance. A viable solution would need both of these. -mpg -Original Message- From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Chris Puttick Sent: Friday, August 21, 2009 9:37 AM To: OSGeo Discussions Subject: Re: [OSGeo-Discuss] Open File Formats and ProprietaryAlgorithms[SEC=UNCLASSIFIED] Well, according to this page: http://jpeg2000.epfl.ch/ v.5.1, courtesy in part Eastman Kodak, provides complete JP2 support at the decoding side - not sure whether that covers the tiling or other geo needs, but doesn't it sound worth investigating? Chris - Christopher Schmidt crschm...@crschmidt.net wrote: On Fri, Aug 21, 2009 at 08:27:13AM -0700, Landon Blake wrote: Thanks for the information Michael. I am downloading Opticks right now. :] I also found this Java library for JP2, thought I'm not sure how complete/up-to-date it is: http://jj2000.epfl.ch/ Maybe we need a JPEG 2000 page on the OSGeo wiki. Note that JPEG 2000 support is different from JPEG 2000 support which works on geo-sized images. The tiling (or 'paging'? as Michael calls it) support that's supposed to be provided by OpenJPEG2000 has been coming 'real soon now' for about 18 months now from my uneducated observations, and until it's there, most tools using OpenJPEG for JP2s are going to suffering under much the same limitations. -- Chris Landon Office Phone Number: (209) 946-0268 Cell Phone Number: (209) 992-0658 -Original Message- From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Considine, Michael Sent: Friday, August 21, 2009 8:09 AM To: OSGeo Discussions Subject: RE: [OSGeo-Discuss] Open File Formats and ProprietaryAlgorithms[SEC=UNCLASSIFIED] All, Opticks is an open source remote sensing application and development framework. We recently started the process to add JPEG 2000 support to our framework. We picked OpenJpeg to add JPEG 2000 support to our application. They are also open source. We currently support importing JPEG 2000 files but we are currently limited to the 4GB memory size after decoding. Our plan is to continue development and to upgrade to OpenJpeg 2.0 once they have a stable release. That will allow Opticks to use a pager to display and support much larger files. Michael Considine -Original Message- From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Bruce Bannerman Sent: Thursday, August 20, 2009 8:15 PM To: 'OSGeo Discussions' Subject: RE: [OSGeo-Discuss] Open File Formats and Proprietary
RE: [OSGeo-Discuss] Open File Formats andProprietaryAlgorithms[SEC=UNCLASSIFIED]
Not a stupid question -- but it doesn't work that way. The artifacts are due to the wavelet processing of the pixels near the tile boundaries, and the boundaries have to be treated reflectively within their individual tiles (in order to keep each tile separate), which means you can sometimes see where those boundaries are. Overlapping doesn't help because the wavelet footprint spans a large width, in order to handle the lower-resolution scales. Which in turn means you need to be able reach far away parts of the image at various (some might say arbitrary) stages in the wavelet pipeline. Just trust me, it is a nontrivial problem to solve. Brighter minds than ours have spent a lot of energy on this problem -- a literature search would reveal a number of PhD theses and patents. -mpg -Original Message- From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Landon Blake Sent: Friday, August 21, 2009 10:45 AM To: OSGeo Discussions Subject: RE: [OSGeo-Discuss] Open File Formats andProprietaryAlgorithms[SEC=UNCLASSIFIED] MPG wrote: Tiling essentially means you can take a large file and compress pieces of it independently. This avoids having to deal with the large memory footprint issues, but it can also lead to seam-line artifacts under certain conditions. Ideally, one would prefer to have the option of compressing large images without resorting to using tiles. This is probably a stupid question, since I know absolutely nothing about image compression, but couldn't you overlap the tiles slightly to avoid the seam lines? This would obviously result in a slightly larger file size because some pixels would be compressed twice. But that might be OK if you were trying to compress a huge image. What about reading chunks of the image off disk, instead of trying to put the whole image in memory? This would be slower, but might make an impossible task possible. We run into this problem with vector datasets to. Some datasets are just to stinking BIG. One of my tasks for OpenJUMP is to write a core module that displays vector data accessed directly from disk, instead of from memory. This will be slower, but it is better than crashing the program because there isn't enough RAM. Things must be more complicated than can be described in an e-mail, because we've got people a lot smarter than me working on these problems. I am just curious. (I tried reading about wavelet compression on Wikipedia yesterday and quickly got a headache.) :] Landon Office Phone Number: (209) 946-0268 Cell Phone Number: (209) 992-0658 -Original Message- From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Michael P. Gerlek Sent: Friday, August 21, 2009 9:36 AM To: OSGeo Discussions Subject: RE: [OSGeo-Discuss] Open File Formats andProprietaryAlgorithms[SEC=UNCLASSIFIED] Tiling essentially means you can take a large file and compress pieces of it independently. This avoids having to deal with the large memory footprint issues, but it can also lead to seam-line artifacts under certain conditions. Ideally, one would prefer to have the option of compressing large images without resorting to using tiles. Note too that, in addition to the large image issue, many of the JP2 implementations out there are either not fully compliant or are not tuned for performance. A viable solution would need both of these. -mpg -Original Message- From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Chris Puttick Sent: Friday, August 21, 2009 9:37 AM To: OSGeo Discussions Subject: Re: [OSGeo-Discuss] Open File Formats and ProprietaryAlgorithms[SEC=UNCLASSIFIED] Well, according to this page: http://jpeg2000.epfl.ch/ v.5.1, courtesy in part Eastman Kodak, provides complete JP2 support at the decoding side - not sure whether that covers the tiling or other geo needs, but doesn't it sound worth investigating? Chris - Christopher Schmidt crschm...@crschmidt.net wrote: On Fri, Aug 21, 2009 at 08:27:13AM -0700, Landon Blake wrote: Thanks for the information Michael. I am downloading Opticks right now. :] I also found this Java library for JP2, thought I'm not sure how complete/up-to-date it is: http://jj2000.epfl.ch/ Maybe we need a JPEG 2000 page on the OSGeo wiki. Note that JPEG 2000 support is different from JPEG 2000 support which works on geo-sized images. The tiling (or 'paging'? as Michael calls it) support that's supposed to be provided by OpenJPEG2000 has been coming 'real soon now' for about 18 months now from my uneducated observations, and until it's there, most tools using OpenJPEG for JP2s are going to suffering under much the same limitations. -- Chris Landon Office Phone Number: (209) 946-0268 Cell Phone Number: (209) 992-0658 -Original Message- From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun
Re: [OSGeo-Discuss] Open File Formats andProprietaryAlgorithms[SEC=UNCLASSIFIED]
On Fri, Aug 21, 2009 at 09:45:04AM -0700, Landon Blake wrote: MPG wrote: Tiling essentially means you can take a large file and compress pieces of it independently. This avoids having to deal with the large memory footprint issues, but it can also lead to seam-line artifacts under certain conditions. Ideally, one would prefer to have the option of compressing large images without resorting to using tiles. This is probably a stupid question, since I know absolutely nothing about image compression, but couldn't you overlap the tiles slightly to avoid the seam lines? This would obviously result in a slightly larger file size because some pixels would be compressed twice. But that might be OK if you were trying to compress a huge image. What about reading chunks of the image off disk, instead of trying to put the whole image in memory? This would be slower, but might make an impossible task possible. Reading chunks of image off disk == tiling. With compression, bits aren't stored on th disk in a way that you can say Okay, bytes 0-32768 are the first 720 pixels in any way. Instead, you have to decompress the image, or part of it, to start to learn these things. Tiling lets you split the image up into many little chunks, which you can read individually. We run into this problem with vector datasets to. Some datasets are just to stinking BIG. One of my tasks for OpenJUMP is to write a core module that displays vector data accessed directly from disk, instead of from memory. This will be slower, but it is better than crashing the program because there isn't enough RAM. Most Vector datasets have some lvel of random access -- I can look for feature 7, and get it, because i know where the start and end of feature 7 are. I don't know where the start and end of pixel 7 is -- because its' different depending on exactly how wth file is compressed. This is all a vast simplification, and some of it is probably not-entirely right, but the problems are -- as you suggested -- more complex than most people not working in imagery know. (And even more complex than some of them know, most likely :)) Regards, -- Christopher Schmidt Web Developer ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
Re: [OSGeo-Discuss] Open File Formats andProprietaryAlgorithms[SEC=UNCLASSIFIED]
So hung up on wavelets, we are. Internally tiled TIFF with JPEG compression and similarly formatted internal overviews can achieve 10:1 compression rates without noticeable image quality reductions, and as an added bonus can be decompressed a heck of a lot faster than wavelet-based formats. The wavelet stuff is k00l, in that there is no need for an overview pyramid (it's implicit in the compression math) and much higher compression rates can be achieved. But operationally, you can go a long way with the more primitive (open image format + open compression algorithm) approach. P. ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
[OSGeo-Discuss] Open File Formats
I think we were talking about the easiest way to implement saving space. :] I was trying to point out (and maybe Paul was also) that ease of implementation (now or in the future) should be a factor the government chooses when selection a file format/technology for data storage and distribution to the public. There are a number of other factors, as MPG mentioned. This has been a really good discussion, and I will try to get some content about open file formats on the wiki as Arnulf requested. Maybe next week? Landon Office Phone Number: (209) 946-0268 Cell Phone Number: (209) 992-0658 From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Bob Basques Sent: Friday, August 21, 2009 11:28 AM To: OSGeo Discussions; Ivan Lucena Subject: RE: [OSGeo-Discuss] OpenFile FormatsandProprietaryAlgorithms[SEC=UNCLASSIFIED] All, Can someone remind me again, are we talking about saving space, or making it easier to implement something . . . :c) I personally prefer nice simple internal pyramided tiles with indexing, about 10% extra space, and very good performance. Someone earlier in this thread spoke about some of these technologies being somewhat obsolete what with the new network and bandwidth speeds available for publishing. bobb Lucena, Ivan ivan.luc...@pmldnet.com wrote: But you can't compress data types other than byte in JPG. Can you do that in JP2K? ---Original Message--- From: Landon Blake lbl...@ksninc.com Subject: RE: [OSGeo-Discuss] Open FileFormatsandProprietaryAlgorithms[SEC=UNCLASSIFIED] Sent: Aug 21 '09 12:42 Paul, I was wondering the same thing. It seems a little like choosing to drive a Honda Accord, or a Ferrari. The Ferrari is a lot faster and comes with a better looking trophy wife (or husband), but the Honda is a lot easier to fix. (Try finding an affordable Ferrari mechanic in Stockton, California.) To tie this back into our original discussion, it seems like the government should be choosing to drive a Honda Accord when it can, instead of the Ferrari. I guess you'd really have to crunch the numbers and see if the savings in bandwidth/disk space costs were really worth the compression savings that result from a proprietary compression scheme (wavelet black magic). The problem with this is a lot of the benefits that come from the Honda Accord (open image format + open compression algorithm) aren't easily calculated in dollars and cents. Still, this speaks to an important truth I have discovered in open source development: Simple is better, even when it isn't necessarily faster and smaller. I'd rather have code that I can understand, or a file format that a programmer in 20 years will understand, than a Ferrari you can't drive unless you have a PHD and did a thesis on wavelet compression. :] Landon Office Phone Number: (209) 946-0268 Cell Phone Number: (209) 992-0658 -Original Message- From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Paul Ramsey Sent: Friday, August 21, 2009 10:36 AM To: OSGeo Discussions Subject: Re: [OSGeo-Discuss] Open File FormatsandProprietaryAlgorithms[SEC=UNCLASSIFIED] So hung up on wavelets, we are. Internally tiled TIFF with JPEG compression and similarly formatted internal overviews can achieve 10:1 compression rates without noticeable image quality reductions, and as an added bonus can be decompressed a heck of a lot faster than wavelet-based formats. The wavelet stuff is k00l, in that there is no need for an overview pyramid (it's implicit in the compression math) and much higher compression rates can be achieved. But operationally, you can go a long way with the more primitive (open image format + open compression algorithm) approach. P. ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss Warning: Information provided via electronic media is not guaranteed against defects including translation and transmission errors. If the reader is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this information in error, please notify the sender immediately. ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss Warning: Information provided via electronic media is not guaranteed against defects including translation and transmission errors. If the reader is not the intended recipient, you are hereby notified that any
Re: [OSGeo-Discuss] Open File Formats
All, I agree, very good conversation, I've already pointed a few folks at it as the same topic has come up here at work on two different projects, as well as a Survey I just completed yesterday for a Federal data provider. I pointed the survey taker at this thread as well. bobb Landon Blake lbl...@ksninc.com wrote: I think we were talking about the easiest way to implement saving space. :] I was trying to point out (and maybe Paul was also) that ease of implementation (now or in the future) should be a factor the government chooses when selection a file format/technology for data storage and distribution to the public. There are a number of other factors, as MPG mentioned. This has been a really good discussion, and I will try to get some content about open file formats on the wiki as Arnulf requested. Maybe next week? Landon Office Phone Number: (209) 946-0268 Cell Phone Number: (209) 992-0658 From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Bob Basques Sent: Friday, August 21, 2009 11:28 AM To: OSGeo Discussions; Ivan Lucena Subject: RE: [OSGeo-Discuss] OpenFile FormatsandProprietaryAlgorithms[SEC=UNCLASSIFIED] All, Can someone remind me again, are we talking about saving space, or making it easier to implement something . . . :c) I personally prefer nice simple internal pyramided tiles with indexing, about 10% extra space, and very good performance. Someone earlier in this thread spoke about some of these technologies being somewhat obsolete what with the new network and bandwidth speeds available for publishing. bobb Lucena, Ivan ivan.luc...@pmldnet.com wrote: But you can't compress data types other than byte in JPG. Can you do that in JP2K? ---Original Message--- From: Landon Blake lbl...@ksninc.com Subject: RE: [OSGeo-Discuss] Open FileFormatsandProprietaryAlgorithms[SEC=UNCLASSIFIED] Sent: Aug 21 '09 12:42 Paul, I was wondering the same thing. It seems a little like choosing to drive a Honda Accord, or a Ferrari. The Ferrari is a lot faster and comes with a better looking trophy wife (or husband), but the Honda is a lot easier to fix. (Try finding an affordable Ferrari mechanic in Stockton, California.) To tie this back into our original discussion, it seems like the government should be choosing to drive a Honda Accord when it can, instead of the Ferrari. I guess you'd really have to crunch the numbers and see if the savings in bandwidth/disk space costs were really worth the compression savings that result from a proprietary compression scheme (wavelet black magic). The problem with this is a lot of the benefits that come from the Honda Accord (open image format + open compression algorithm) aren't easily calculated in dollars and cents. Still, this speaks to an important truth I have discovered in open source development: Simple is better, even when it isn't necessarily faster and smaller. I'd rather have code that I can understand, or a file format that a programmer in 20 years will understand, than a Ferrari you can't drive unless you have a PHD and did a thesis on wavelet compression. :] Landon Office Phone Number: (209) 946-0268 Cell Phone Number: (209) 992-0658 -Original Message- From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Paul Ramsey Sent: Friday, August 21, 2009 10:36 AM To: OSGeo Discussions Subject: Re: [OSGeo-Discuss] Open File FormatsandProprietaryAlgorithms[SEC=UNCLASSIFIED] So hung up on wavelets, we are. Internally tiled TIFF with JPEG compression and similarly formatted internal overviews can achieve 10:1 compression rates without noticeable image quality reductions, and as an added bonus can be decompressed a heck of a lot faster than wavelet-based formats. The wavelet stuff is k00l, in that there is no need for an overview pyramid (it's implicit in the compression math) and much higher compression rates can be achieved. But operationally, you can go a long way with the more primitive (open image format + open compression algorithm) approach. P. ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss Warning: Information provided via electronic media is not guaranteed against defects including translation and transmission errors. If the reader is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this information in error, please notify the sender immediately. ___ Discuss mailing list Discuss@lists.osgeo.org
[OSGeo-Discuss] Open File Formats and Proprietary Algorithms
I realized that publishing a spec for a file format like MRSID isn't as clear cut as I had at first thought. If the MRSID software uses a fancy top-secret compression/decompression algorithm to move data to and from the file format knowing only the structure of the format would do no good. You'd have to release the details of the algorithm as well. I still don't think proprietary file formats are a good idea for government data released to the public, but I admit that having a company like LizardTech publish a spec for something like MRSID is not necessarily a simple task. No doubt a lot of time and money goes into developing those algorithms. This makes me wonder about algorithms used to purposefully encrypt binary file formats. That is another can of worms. It looks like the easiest thing to do is to start with a file format that was designed to be open from the very beginning. Landon Warning: Information provided via electronic media is not guaranteed against defects including translation and transmission errors. If the reader is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this information in error, please notify the sender immediately.___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
Re: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms
The MRSID format is a very special case - and perhaps an opportunity for a new FOSS file format. MRSID is a lossless, fractal-based, multi-scale raster compression format. LizardTech has the algorithms to encode and decode MRSID locked up in copyrights, and I believe, patents. Even companies like ESRI shell out big bucks to LizardTech to be able to read and write the MRSID format. I guess I missed the context of the discussion. Is the government releasing certain data exclusively in this format? If so, I think the argument can be made against this practice. The different in compression between MRSID and gziped TIFFs isn't really that great in this day of cheap disks and fat pipes. -Eric -=--=---===---=--=-=--=---==---=--=-=- Eric B. WolfNew! 720-334-7734 USGS Geographer Center of Excellence in GIScience PhD Student CU-Boulder - Geography On Thu, Aug 20, 2009 at 12:27 PM, Landon Blake lbl...@ksninc.com wrote: I realized that publishing a spec for a file format like MRSID isn’t as clear cut as I had at first thought. If the MRSID software uses a fancy top-secret compression/decompression algorithm to move data to and from the file format knowing only the structure of the format would do no good. You’d have to release the details of the algorithm as well. I still don’t think proprietary file formats are a good idea for government data released to the public, but I admit that having a company like LizardTech publish a spec for something like MRSID is not necessarily a simple task. No doubt a lot of time and money goes into developing those algorithms. This makes me wonder about algorithms used to purposefully encrypt binary file formats. That is another can of worms. It looks like the easiest thing to do is to start with a file format that was designed to be open from the very beginning. Landon *Warning: *Information provided via electronic media is not guaranteed against defects including translation and transmission errors. If the reader is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this information in error, please notify the sender immediately. ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
RE: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms
Eric, The imagery I am talking about is from the USDA APFO: This FAQ contains a snippet about the format: http://www.fsa.usda.gov/FSA/apfoapp?area=homesubject=progtopic=nai In an interesting turn of events I note that as of 2008, the USDA is releasing the county mosaics in JP2 format, not in MRSID. I am not sure what brought about this change, and I wasn't aware that it had been made. The same web page indicates that there is a shapefile index for the individual image tiles. It appears that you can also download the county mosaics online. A lot of this has changed (improved) in the last couple of years. I'm glad I checked again. That being said, the principles from our discussion still apply. :] Landon Office Phone Number: (209) 946-0268 Cell Phone Number: (209) 992-0658 From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Eric Wolf Sent: Thursday, August 20, 2009 1:15 PM To: OSGeo Discussions Subject: Re: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms The MRSID format is a very special case - and perhaps an opportunity for a new FOSS file format. MRSID is a lossless, fractal-based, multi-scale raster compression format. LizardTech has the algorithms to encode and decode MRSID locked up in copyrights, and I believe, patents. Even companies like ESRI shell out big bucks to LizardTech to be able to read and write the MRSID format. I guess I missed the context of the discussion. Is the government releasing certain data exclusively in this format? If so, I think the argument can be made against this practice. The different in compression between MRSID and gziped TIFFs isn't really that great in this day of cheap disks and fat pipes. -Eric -=--=---===---=--=-=--=---==---=--=-=- Eric B. WolfNew! 720-334-7734 USGS Geographer Center of Excellence in GIScience PhD Student CU-Boulder - Geography On Thu, Aug 20, 2009 at 12:27 PM, Landon Blake lbl...@ksninc.com wrote: I realized that publishing a spec for a file format like MRSID isn't as clear cut as I had at first thought. If the MRSID software uses a fancy top-secret compression/decompression algorithm to move data to and from the file format knowing only the structure of the format would do no good. You'd have to release the details of the algorithm as well. I still don't think proprietary file formats are a good idea for government data released to the public, but I admit that having a company like LizardTech publish a spec for something like MRSID is not necessarily a simple task. No doubt a lot of time and money goes into developing those algorithms. This makes me wonder about algorithms used to purposefully encrypt binary file formats. That is another can of worms. It looks like the easiest thing to do is to start with a file format that was designed to be open from the very beginning. Landon Warning: Information provided via electronic media is not guaranteed against defects including translation and transmission errors. If the reader is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this information in error, please notify the sender immediately. ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
Re: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms
Interesting... I can understand why NAIP was in MRSID. It's a pretty large dataset - and I think .SID was more widely supported than JP2 until recently. The USDA site does provide links to PCI Geomatics FreeView, which can read .SID format but not save it. IrfanView, with a plugin, can read SID format and convert. So it's not a dead-end format. And it sure beats SDTS! I think data interchange and real interoperability has only recently been possible for large raster datasets. It's still a chore if you have to re-project large raster datasets. This may add some content to a research paper I'm working on. -Eric -=--=---===---=--=-=--=---==---=--=-=- Eric B. WolfNew! 720-334-7734 USGS Geographer Center of Excellence in GIScience PhD Student CU-Boulder - Geography On Thu, Aug 20, 2009 at 2:22 PM, Landon Blake lbl...@ksninc.com wrote: Eric, The imagery I am talking about is from the USDA APFO: This FAQ contains a snippet about the format: http://www.fsa.usda.gov/FSA/apfoapp?area=homesubject=progtopic=nai In an interesting turn of events I note that as of 2008, the USDA is releasing the county mosaics in JP2 format, not in MRSID. I am not sure what brought about this change, and I wasn’t aware that it had been made. The same web page indicates that there is a shapefile index for the individual image tiles. It appears that you can also download the county mosaics online. A lot of this has changed (improved) in the last couple of years. I’m glad I checked again. That being said, the principles from our discussion still apply. :] *Landon* Office Phone Number: (209) 946-0268 Cell Phone Number: (209) 992-0658 -- *From:* discuss-boun...@lists.osgeo.org [mailto: discuss-boun...@lists.osgeo.org] *On Behalf Of *Eric Wolf *Sent:* Thursday, August 20, 2009 1:15 PM *To:* OSGeo Discussions *Subject:* Re: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms The MRSID format is a very special case - and perhaps an opportunity for a new FOSS file format. MRSID is a lossless, fractal-based, multi-scale raster compression format. LizardTech has the algorithms to encode and decode MRSID locked up in copyrights, and I believe, patents. Even companies like ESRI shell out big bucks to LizardTech to be able to read and write the MRSID format. I guess I missed the context of the discussion. Is the government releasing certain data exclusively in this format? If so, I think the argument can be made against this practice. The different in compression between MRSID and gziped TIFFs isn't really that great in this day of cheap disks and fat pipes. -Eric -=--=---===---=--=-=--=---==---=--=-=- Eric B. WolfNew! 720-334-7734 USGS Geographer Center of Excellence in GIScience PhD Student CU-Boulder - Geography On Thu, Aug 20, 2009 at 12:27 PM, Landon Blake lbl...@ksninc.com wrote: I realized that publishing a spec for a file format like MRSID isn’t as clear cut as I had at first thought. If the MRSID software uses a fancy top-secret compression/decompression algorithm to move data to and from the file format knowing only the structure of the format would do no good. You’d have to release the details of the algorithm as well. I still don’t think proprietary file formats are a good idea for government data released to the public, but I admit that having a company like LizardTech publish a spec for something like MRSID is not necessarily a simple task. No doubt a lot of time and money goes into developing those algorithms. This makes me wonder about algorithms used to purposefully encrypt binary file formats. That is another can of worms. It looks like the easiest thing to do is to start with a file format that was designed to be open from the very beginning. Landon *Warning:** *Information provided via electronic media is not guaranteed against defects including translation and transmission errors. If the reader is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this information in error, please notify the sender immediately. ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
RE: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms
Some clarifications: - MrSID has both lossy and lossless modes - MrSID is not fractal based; it uses wavelets (and arithmetic encoding) - you can't copyright algorithms; the MrSID source code certainly is, however - MrSID relies on a number of patents, not all of which are owned by LizardTech - reading MrSID does not require any fees; we have libraries you can download, although they are not open source That said, some editorial comments (although I'm now wishing I hadn't been so quick to rise to Landon's bait :-) - Some of you know the history of trying to open source MrSID; I won't go into that here, except to say that LizardTech doesn't own all of the required IP needed to make that happen. - If we are speaking of the NAIP data, then no, it is not exclusively available in MrSID format; it is also shipped as GeoTIFFs. - JPEG 2000 is a very robust open standard alternative to MrSID, and a number of players already support it (including LizardTech), but not enough to make it viable for certain domains like NAIP. - some of you also know the history on open JP2 support: there is today no open source implementation of JP2 that is suitable for geo work. Alas. -mpg From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Eric Wolf Sent: Thursday, August 20, 2009 2:15 PM To: OSGeo Discussions Subject: Re: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms The MRSID format is a very special case - and perhaps an opportunity for a new FOSS file format. MRSID is a lossless, fractal-based, multi-scale raster compression format. LizardTech has the algorithms to encode and decode MRSID locked up in copyrights, and I believe, patents. Even companies like ESRI shell out big bucks to LizardTech to be able to read and write the MRSID format. I guess I missed the context of the discussion. Is the government releasing certain data exclusively in this format? If so, I think the argument can be made against this practice. The different in compression between MRSID and gziped TIFFs isn't really that great in this day of cheap disks and fat pipes. -Eric -=--=---===---=--=-=--=---==---=--=-=- Eric B. WolfNew! 720-334-7734 USGS Geographer Center of Excellence in GIScience PhD Student CU-Boulder - Geography ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
Re: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms
Thanks for the clarification, Michael! And your comments about IP may also add to the paper I am developing (or another). I have a theme I plan to develop at some point - mostly dealing with the inherent limitations of copyrighted software in an era of cloud computing... -Eric -=--=---===---=--=-=--=---==---=--=-=- Eric B. WolfNew! 720-334-7734 USGS Geographer Center of Excellence in GIScience PhD Student CU-Boulder - Geography On Thu, Aug 20, 2009 at 2:54 PM, Michael P. Gerlek m...@lizardtech.comwrote: Some clarifications: - MrSID has both lossy and lossless modes - MrSID is not fractal based; it uses wavelets (and arithmetic encoding) - you can’t copyright algorithms; the MrSID source code certainly is, however - MrSID relies on a number of patents, not all of which are owned by LizardTech - reading MrSID does not require any fees; we have libraries you can download, although they are not open source That said, some editorial comments (although I’m now wishing I hadn’t been so quick to rise to Landon’s bait :-) - Some of you know the history of trying to open source MrSID; I won’t go into that here, except to say that LizardTech doesn’t own all of the required IP needed to make that happen. - If we are speaking of the NAIP data, then no, it is not exclusively available in MrSID format; it is also shipped as GeoTIFFs. - JPEG 2000 is a very robust open standard alternative to MrSID, and a number of players already support it (including LizardTech), but not enough to make it viable for certain domains like NAIP. - some of you also know the history on open JP2 support: there is today no open source implementation of JP2 that is suitable for geo work. Alas. -mpg *From:* discuss-boun...@lists.osgeo.org [mailto: discuss-boun...@lists.osgeo.org] *On Behalf Of *Eric Wolf *Sent:* Thursday, August 20, 2009 2:15 PM *To:* OSGeo Discussions *Subject:* Re: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms The MRSID format is a very special case - and perhaps an opportunity for a new FOSS file format. MRSID is a lossless, fractal-based, multi-scale raster compression format. LizardTech has the algorithms to encode and decode MRSID locked up in copyrights, and I believe, patents. Even companies like ESRI shell out big bucks to LizardTech to be able to read and write the MRSID format. I guess I missed the context of the discussion. Is the government releasing certain data exclusively in this format? If so, I think the argument can be made against this practice. The different in compression between MRSID and gziped TIFFs isn't really that great in this day of cheap disks and fat pipes. -Eric -=--=---===---=--=-=--=---==---=--=-=- Eric B. WolfNew! 720-334-7734 USGS Geographer Center of Excellence in GIScience PhD Student CU-Boulder - Geography ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
RE: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms
MPG: Thanks for the clarification. When you said there is today no open source implementation of JP2 that is suitable for geo work do you mean that there is no open source library that can read and write JP2? If so, who is using the format? Do you know why there hasn't been a broader adoption of JP2? (I should also add the MPG helped me publish a short article in support for open file formats, so I know he is on our side.) :] Landon Office Phone Number: (209) 946-0268 Cell Phone Number: (209) 992-0658 From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Michael P. Gerlek Sent: Thursday, August 20, 2009 1:55 PM To: OSGeo Discussions Subject: RE: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms Some clarifications: - MrSID has both lossy and lossless modes - MrSID is not fractal based; it uses wavelets (and arithmetic encoding) - you can't copyright algorithms; the MrSID source code certainly is, however - MrSID relies on a number of patents, not all of which are owned by LizardTech - reading MrSID does not require any fees; we have libraries you can download, although they are not open source That said, some editorial comments (although I'm now wishing I hadn't been so quick to rise to Landon's bait :-) - Some of you know the history of trying to open source MrSID; I won't go into that here, except to say that LizardTech doesn't own all of the required IP needed to make that happen. - If we are speaking of the NAIP data, then no, it is not exclusively available in MrSID format; it is also shipped as GeoTIFFs. - JPEG 2000 is a very robust open standard alternative to MrSID, and a number of players already support it (including LizardTech), but not enough to make it viable for certain domains like NAIP. - some of you also know the history on open JP2 support: there is today no open source implementation of JP2 that is suitable for geo work. Alas. -mpg From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Eric Wolf Sent: Thursday, August 20, 2009 2:15 PM To: OSGeo Discussions Subject: Re: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms The MRSID format is a very special case - and perhaps an opportunity for a new FOSS file format. MRSID is a lossless, fractal-based, multi-scale raster compression format. LizardTech has the algorithms to encode and decode MRSID locked up in copyrights, and I believe, patents. Even companies like ESRI shell out big bucks to LizardTech to be able to read and write the MRSID format. I guess I missed the context of the discussion. Is the government releasing certain data exclusively in this format? If so, I think the argument can be made against this practice. The different in compression between MRSID and gziped TIFFs isn't really that great in this day of cheap disks and fat pipes. -Eric -=--=---===---=--=-=--=---==---=--=-=- Eric B. WolfNew! 720-334-7734 USGS Geographer Center of Excellence in GIScience PhD Student CU-Boulder - Geography Warning: Information provided via electronic media is not guaranteed against defects including translation and transmission errors. If the reader is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this information in error, please notify the sender immediately.___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
RE: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms
Landon asked: When you said there is today no open source implementation of JP2 that is suitable for geo work do you mean that there is no open source library that can read and write JP2? If so, who is using the format? There are a few implementations of JP2 around. The Kakadu library, which is extremely compliant and featureful and robust (and correspondingly extremely big and complicated and scary) is the best-known package: it is available only via licensing fees. LizardTech uses Kakadu, in fact, and a number of geo vendors use either Kakadu directly or LizardTech's packaging of it. The ER Mapper folks had a JP2 solution at one time, but I never understood their licensing terms to be OSI compliant -- and since they got bought out by Leica I've sort of stopped tracking that issue. If anyone has any current info, I'd like to hear it. There are a couple truly open source libraries, but none have been written in such a way as to be able to support geo-sized imagery (500MB, say). Doing the wavelet algorithms efficiently for large data sets requires rocket science. Do you know why there hasn't been a broader adoption of JP2? Not through lack of trying on my part :-) I think the two biggest reasons are: (1) The algorithms for handling large images in memory really are rocket science, and no one in the FOSS community has gotten the itch sufficiently bad enough to go and do the work needed inside the existing open source packages. Hopefully someday someone will. (2) MrSID (and, perhaps, ECW) are widely used and supported. Philosophical motivations aside, MrSID and ECW have historically gotten the job done and so the need for JP2 just isn't as high as it otherwise might be. That said, NGA is a good counter-example. They support JP2 in a number of areas already and have mandates to broaden that support. -mpg ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
RE: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms
I'll mention too the question of patents and JP2, since this thread is bound to get into THAT issue too before long :-) Some of the algorithms within the JP2 standard (from ISO) are patented. However, the companies in question have agreed to not exercise their rights on those patents for any implementation of the standard. That is, if you write a ISO-compliant JP2 encoder, Company X won't come after you. This is a good thing, and is not uncommon practice for some standards groups. It's better for us than the RAND (reasonable and non-discriminatory) clauses that get used by some groups. However, there is an interesting philosophical consideration for the open source community here. Let's say I write a nice, compliant MpgJp2 library on Monday and open source it. Landon looks at my code and, smart cookie that he is, realizes that he could improve the overall compression ratio by tweaking one of the core algorithms. He forks my code, makes the change, and posts the SunburnedJp2 library to the web on Tuesday night. Cool. We like that. Open source in action. But wait -- Wednesday morning, he finds an email from Company X's lawyers in his inbox: he is now in violation of X's patent, because he is not using the patent within the bounds of a compliant JP2 encoder. He broke the file format. [You break it, you buy it?] It's not a JPEG 2000 library anymore. Some open source partisans may therefore consider the JP2 standard to not be truly open enough. I'm sure there are other standards with this same problem, although I don't know of any offhand. -mpg From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Landon Blake Sent: Thursday, August 20, 2009 2:57 PM To: OSGeo Discussions Subject: RE: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms MPG: Thanks for the clarification. When you said there is today no open source implementation of JP2 that is suitable for geo work do you mean that there is no open source library that can read and write JP2? If so, who is using the format? Do you know why there hasn't been a broader adoption of JP2? (I should also add the MPG helped me publish a short article in support for open file formats, so I know he is on our side.) :] Landon Office Phone Number: (209) 946-0268 Cell Phone Number: (209) 992-0658 From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Michael P. Gerlek Sent: Thursday, August 20, 2009 1:55 PM To: OSGeo Discussions Subject: RE: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms Some clarifications: - MrSID has both lossy and lossless modes - MrSID is not fractal based; it uses wavelets (and arithmetic encoding) - you can't copyright algorithms; the MrSID source code certainly is, however - MrSID relies on a number of patents, not all of which are owned by LizardTech - reading MrSID does not require any fees; we have libraries you can download, although they are not open source That said, some editorial comments (although I'm now wishing I hadn't been so quick to rise to Landon's bait :-) - Some of you know the history of trying to open source MrSID; I won't go into that here, except to say that LizardTech doesn't own all of the required IP needed to make that happen. - If we are speaking of the NAIP data, then no, it is not exclusively available in MrSID format; it is also shipped as GeoTIFFs. - JPEG 2000 is a very robust open standard alternative to MrSID, and a number of players already support it (including LizardTech), but not enough to make it viable for certain domains like NAIP. - some of you also know the history on open JP2 support: there is today no open source implementation of JP2 that is suitable for geo work. Alas. -mpg From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Eric Wolf Sent: Thursday, August 20, 2009 2:15 PM To: OSGeo Discussions Subject: Re: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms The MRSID format is a very special case - and perhaps an opportunity for a new FOSS file format. MRSID is a lossless, fractal-based, multi-scale raster compression format. LizardTech has the algorithms to encode and decode MRSID locked up in copyrights, and I believe, patents. Even companies like ESRI shell out big bucks to LizardTech to be able to read and write the MRSID format. I guess I missed the context of the discussion. Is the government releasing certain data exclusively in this format? If so, I think the argument can be made against this practice. The different in compression between MRSID and gziped TIFFs isn't really that great in this day of cheap disks and fat pipes. -Eric -=--=---===---=--=-=--=---==---=--=-=- Eric B. WolfNew! 720-334-7734 USGS Geographer Center of Excellence in GIScience PhD Student CU-Boulder - Geography
Re: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms
On Thu, Aug 20, 2009 at 01:57:16PM -0700, Landon Blake wrote: MPG: Thanks for the clarification. When you said there is today no open source implementation of JP2 that is suitable for geo work do you mean that there is no open source library that can read and write JP2? If so, who is using the format? There are: 1. Several non-open source implementations (most of which cost money) which work at geo-sized JP2 images. 2. Many use cases of JPEG2000 which involve imagery at sizes that are less than geo. (This is the much more common case, in my research.) Do you know why there hasn't been a broader adoption of JP2? I'm not sure what your definition is of broader adoption; many of the datasources I worked with for OAM were provided in either JP2 or MrSID formats. I would almost always go with MrSID, because I could: * Work with it easily, and for free * It was typically significantly smaller. Perhaps you're asking why there hasn't been more open source software written to handle large, highly compressed JP2 images better -- to which I would point out that there isn't *any* format that has good open source support for large, highly compressed images. (gzipped TIFFs work to some extent, but don't compare to the benefits gained by JP2 or MrSID in many cases.) It's a hard problem, and -- given that the major players see the costs to 'pay to play' as being trivial (and they typically are, in the big scheme of things), not in a situation where it's likely that the people with ots of money ar ein a position to spend it on open source, rather than simply paying a smaller amount for existing non-opensource solutions. Despite the claims that 'disks are cheap and bandwidth is free', many providers *are* limited by bandwidth: MassGIS, for example, had to put in cash for a costly upgrade to their badnwidth solely due to the demand put on their servers by people downloading aerial imagery. Those funds could have gone to funding more open geodata, but instead were used to maek the data that already existed more readily available. These things *do* matter, and MrSID offers, by far, the best 'bang for the buck' for amount of data per byte of download. This applies even more at the consumer end; when you talk about consuming data, MrSID is even *more* user-friendly, because the users (who have limited bandwidth) are able to open it more easily. Additionally, many viewers which include MrSID support are able to display larger images -- due to the MrSID library -- than they would be by opening the entire image in RAM or something similar. Many of my friends have used MrSID for looking at thigns like Shakespear's Folios, because tools like IfranView include it by default, and the tool Just Works better than anything else. I believe that the important things in terms of delivering public content to users are: * License -- Are they allowed to do what they want with it? * Ease of use -- Is it *possible* For them to do what they want with it, including downloading it in the first place? * Openness -- Can they do what htey want with it with free/open tools? If the formwer two are true, then the latter -- openness -- can be handled by third parties. Imagine that you have two options: * Data provided online, for users to download, in MrSID * Data provided on CDs, for users to have shipped to them, in GeoTIFF (The latter will almost always have a non-trivial fee, because it involves person time, but ignore that for the time being.) If these are your options -- and this *is* the case for a non-zero number of imagery providers -- which one would you prefer to use? Best Regards, -- Christopher Schmidt Web Developer ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
RE: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms
Good post Christopher. I will think about what you have said. In the meantime, I won't be using any big images. :] Landon Office Phone Number: (209) 946-0268 Cell Phone Number: (209) 992-0658 -Original Message- From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Christopher Schmidt Sent: Thursday, August 20, 2009 2:50 PM To: OSGeo Discussions Subject: Re: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms On Thu, Aug 20, 2009 at 01:57:16PM -0700, Landon Blake wrote: MPG: Thanks for the clarification. When you said there is today no open source implementation of JP2 that is suitable for geo work do you mean that there is no open source library that can read and write JP2? If so, who is using the format? There are: 1. Several non-open source implementations (most of which cost money) which work at geo-sized JP2 images. 2. Many use cases of JPEG2000 which involve imagery at sizes that are less than geo. (This is the much more common case, in my research.) Do you know why there hasn't been a broader adoption of JP2? I'm not sure what your definition is of broader adoption; many of the datasources I worked with for OAM were provided in either JP2 or MrSID formats. I would almost always go with MrSID, because I could: * Work with it easily, and for free * It was typically significantly smaller. Perhaps you're asking why there hasn't been more open source software written to handle large, highly compressed JP2 images better -- to which I would point out that there isn't *any* format that has good open source support for large, highly compressed images. (gzipped TIFFs work to some extent, but don't compare to the benefits gained by JP2 or MrSID in many cases.) It's a hard problem, and -- given that the major players see the costs to 'pay to play' as being trivial (and they typically are, in the big scheme of things), not in a situation where it's likely that the people with ots of money ar ein a position to spend it on open source, rather than simply paying a smaller amount for existing non-opensource solutions. Despite the claims that 'disks are cheap and bandwidth is free', many providers *are* limited by bandwidth: MassGIS, for example, had to put in cash for a costly upgrade to their badnwidth solely due to the demand put on their servers by people downloading aerial imagery. Those funds could have gone to funding more open geodata, but instead were used to maek the data that already existed more readily available. These things *do* matter, and MrSID offers, by far, the best 'bang for the buck' for amount of data per byte of download. This applies even more at the consumer end; when you talk about consuming data, MrSID is even *more* user-friendly, because the users (who have limited bandwidth) are able to open it more easily. Additionally, many viewers which include MrSID support are able to display larger images -- due to the MrSID library -- than they would be by opening the entire image in RAM or something similar. Many of my friends have used MrSID for looking at thigns like Shakespear's Folios, because tools like IfranView include it by default, and the tool Just Works better than anything else. I believe that the important things in terms of delivering public content to users are: * License -- Are they allowed to do what they want with it? * Ease of use -- Is it *possible* For them to do what they want with it, including downloading it in the first place? * Openness -- Can they do what htey want with it with free/open tools? If the formwer two are true, then the latter -- openness -- can be handled by third parties. Imagine that you have two options: * Data provided online, for users to download, in MrSID * Data provided on CDs, for users to have shipped to them, in GeoTIFF (The latter will almost always have a non-trivial fee, because it involves person time, but ignore that for the time being.) If these are your options -- and this *is* the case for a non-zero number of imagery providers -- which one would you prefer to use? Best Regards, -- Christopher Schmidt Web Developer ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss Warning: Information provided via electronic media is not guaranteed against defects including translation and transmission errors. If the reader is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this information in error, please notify the sender immediately. ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
RE: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms [SEC=UNCLASSIFIED]
IMO: Just another thought on this issue (though we do seem to be recycling arguments over the years...): Assuming that I have a very large archive of spatial data, be it imagery or any other spatial format and that I store my data in a variety of proprietary formats: In ten years from now, can I be sure that: - the company that created, understands, and holds the IP in the data format will still be around? - there will still be software that runs on the then current operating environment, that can read and 'fully exploit' the data in the proprietary standard? - that this future software will work seamlessly with my then current spatial environment? - if all of the above risks prove to eventuate, can I be sure that I'll be able to salvage my data into another format, retaining its complete semantic context? IMO, it is a high risk proposition to lock public (or private) archives away in proprietary data formats. It makes more sense to use open standards and formats that are publically available. Bruce Bannerman -Original Message- From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Michael P. Gerlek Sent: Friday, 21 August 2009 6:55 AM To: OSGeo Discussions Subject: RE: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms Some clarifications: - MrSID has both lossy and lossless modes - MrSID is not fractal based; it uses wavelets (and arithmetic encoding) - you can't copyright algorithms; the MrSID source code certainly is, however - MrSID relies on a number of patents, not all of which are owned by LizardTech - reading MrSID does not require any fees; we have libraries you can download, although they are not open source That said, some editorial comments (although I'm now wishing I hadn't been so quick to rise to Landon's bait :-) - Some of you know the history of trying to open source MrSID; I won't go into that here, except to say that LizardTech doesn't own all of the required IP needed to make that happen. - If we are speaking of the NAIP data, then no, it is not exclusively available in MrSID format; it is also shipped as GeoTIFFs. - JPEG 2000 is a very robust open standard alternative to MrSID, and a number of players already support it (including LizardTech), but not enough to make it viable for certain domains like NAIP. - some of you also know the history on open JP2 support: there is today no open source implementation of JP2 that is suitable for geo work. Alas. -mpg From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Eric Wolf Sent: Thursday, August 20, 2009 2:15 PM To: OSGeo Discussions Subject: Re: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms The MRSID format is a very special case - and perhaps an opportunity for a new FOSS file format. MRSID is a lossless, fractal-based, multi-scale raster compression format. LizardTech has the algorithms to encode and decode MRSID locked up in copyrights, and I believe, patents. Even companies like ESRI shell out big bucks to LizardTech to be able to read and write the MRSID format. I guess I missed the context of the discussion. Is the government releasing certain data exclusively in this format? If so, I think the argument can be made against this practice. The different in compression between MRSID and gziped TIFFs isn't really that great in this day of cheap disks and fat pipes. -Eric -=--=---===---=--=-=--=---==---=--=-=- Eric B. WolfNew! 720-334-7734 USGS Geographer Center of Excellence in GIScience PhD Student CU-Boulder - Geography ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
