Simone answered a number of my questions about compression formats. I'm forwarding to the geotools list in case there are others interested in following Simon's progress.
[7:00:16 AM] Cameron Shorter says: Hello Simone [7:04:29 AM] simboss18830 says: Ciao Cameron [7:04:46 AM] simboss18830 says: I didn't notice you were online, sorry! [7:05:21 AM] Cameron Shorter says: Got a minute to talk about compression? [7:05:29 AM] simboss18830 says: sure [7:06:17 AM] Cameron Shorter says: I want to get my head around the level of effort required to get compression into UDig. [7:06:44 AM] Cameron Shorter says: And the level of quality that it is currently at. [7:08:19 AM] simboss18830 says: I have a couple of questions for you in order to get a more clear idea of what you really need [7:08:24 AM] Cameron Shorter says: I assume you plan to move your code into the Geotools baseline sometime? When are you planning to do that? [7:08:27 AM] Cameron Shorter says: yes? [7:08:49 AM] simboss18830 says: 1>do you need only reading capabilities at first [7:08:57 AM] Cameron Shorter says: Yes [7:09:44 AM] simboss18830 says: 2>do you havein mind about a specific format/library, like ECW or MRSID or KAKADU or maybe something else? [7:09:59 AM] simboss18830 says: (remove *about* :) ) [7:10:20 AM] Cameron Shorter says: ECW or JP2000 were mentioned by the customer. [7:10:38 AM] Cameron Shorter says: But if another format is easier, we could consider that. [7:10:55 AM] simboss18830 says: ECW sdk [7:11:04 AM] simboss18830 says: allows to access ECW and JP2 format [7:11:21 AM] simboss18830 says: MRSID sdk allows to access MrSID and JP2 [7:11:40 AM] simboss18830 says: kakadu allows to access only JP2 WITHOUT any geoinfo [7:11:56 AM] simboss18830 says: also the cost might be a factor to consider [7:12:10 AM] Cameron Shorter says: Cost of development? [7:12:19 AM] Cameron Shorter says: Or license? [7:12:20 AM] simboss18830 says: cost of the license for the sdk [7:12:31 AM] simboss18830 says: kakadu for instance can be quite expensive [7:12:36 AM] simboss18830 says: for a big organizzation [7:12:59 AM] simboss18830 says: if you want tomorrow I can do a little survey [7:13:08 AM] simboss18830 says: costs/capabilities [7:13:15 AM] simboss18830 says: for the various libs [7:13:25 AM] simboss18830 says: and put it on the geotools wiki [7:13:27 AM] Cameron Shorter says: We have time. It is not needed until OWS5.2 which is in a month or 2. [7:13:38 AM] Cameron Shorter says: That would be great. [7:14:15 AM] Cameron Shorter says: As you might have guessed, I know very little about compression formats at the moment. [7:14:43 AM] Cameron Shorter says: I'm guessing that we will be wanting ECW/JP2000 SDK. [7:14:48 AM] simboss18830 says: well basically all of them work in a pretty similar way [7:14:53 AM] simboss18830 says: using wavelet compression [7:15:00 AM] simboss18830 says: the sdk itself usuallly makes the difference [7:15:10 AM] simboss18830 says: I think ecw is a good balance [7:15:28 AM] simboss18830 says: I have to check but I think that for reading [7:15:31 AM] simboss18830 says: ECW is free [7:15:35 AM] simboss18830 says: and for compressing [7:15:45 AM] simboss18830 says: is free for images under 500 mb [7:15:56 AM] Cameron Shorter says: ECW is produced by ERMapper isn't it? [7:16:00 AM] simboss18830 says: yeah [7:16:08 AM] simboss18830 says: they are australian btw [7:16:17 AM] simboss18830 says: but they have been bought by Leica recently [7:16:17 AM] Cameron Shorter says: ERMapper is an Australian company - which is of benefit to us. [7:16:28 AM] simboss18830 says: yeah :) [7:16:48 AM] simboss18830 says: now the second part of quick poll :) [7:16:58 AM] Cameron Shorter says: (We will be funded by Australian Research agenency CRC-SI which favours sponsoring Australian industry) [7:17:20 AM] simboss18830 says: The easiest, fastest way to use ECW is by leveraging on GDAL SWIG bindings [7:17:49 AM] simboss18830 says: but this introduces an additional indirection level between us and the ECW sdk [7:17:54 AM] simboss18830 says: hiding some capabilities [7:18:35 AM] simboss18830 says: hence you might loose some of the power of the underlying SDK [7:18:50 AM] simboss18830 says: and also [7:18:58 AM] simboss18830 says: you alwayd depend on gdal [7:19:08 AM] Cameron Shorter says: ok [7:19:09 AM] simboss18830 says: which addsa more burden on the maintainer [7:19:48 AM] Cameron Shorter says: When bunding, how many packages would a user need to install and what licenses are involved? [7:20:04 AM] Cameron Shorter says: Is there a proprietary license inserted into one of the libraries? [7:20:59 AM] simboss18830 says: mmmmhh, I am not sure, I would have to check [7:21:02 AM] simboss18830 says: but I think [7:21:08 AM] simboss18830 says: that you could freely redistribuite [7:21:22 AM] simboss18830 says: the ERMapper DLLs or SOs [7:21:37 AM] simboss18830 says: anyway [7:21:45 AM] simboss18830 says: anyone can downloade the sdk [7:21:47 AM] simboss18830 says: and install it [7:21:50 AM] Cameron Shorter says: Is the ECW code in C or java? [7:22:19 AM] simboss18830 says: some time ago [7:22:30 AM] simboss18830 says: a java wrapper ONLY for windows [7:22:35 AM] Cameron Shorter says: Setting up an easy install process will be important for the client as they need to roll out across non-technical people. [7:22:37 AM] simboss18830 says: was available [7:22:51 AM] simboss18830 says: but it isn't anymore [7:22:53 AM] simboss18830 says: I have a copy [7:23:02 AM] simboss18830 says: but anyway it is pretty bad [7:23:04 AM] simboss18830 says: hence I never used it [7:23:29 AM] simboss18830 says: the code actually should be mixed [7:23:31 AM] simboss18830 says: c/c++ [7:23:41 AM] Cameron Shorter says: Does the GDAL code use the ECW SDK or is it independant? [7:23:53 AM] simboss18830 says: it uses the ECW sdk [7:24:07 AM] simboss18830 says: the ECW code is open, which means you can look at it [7:24:14 AM] simboss18830 says: but it is proprietary [7:24:30 AM] Cameron Shorter says: "Look but don't touch"? [7:24:30 AM] simboss18830 says: *the algorithm* is proprietary [7:24:38 AM] simboss18830 says: for the ECW compression [7:24:39 AM] simboss18830 says: yeah [7:24:58 AM] Cameron Shorter says: ok [7:25:30 AM] Cameron Shorter says: Why will it be easier to use GDAL. Is it because there is already a Geotools interface to GDAL? [7:25:43 AM] simboss18830 says: it would be easier to use gdal [7:25:44 AM] simboss18830 says: because [7:25:54 AM] simboss18830 says: the bindings to access gdal from java [7:26:03 AM] simboss18830 says: area available [7:26:11 AM] simboss18830 says: and we have already played quite a bit with tme [7:26:14 AM] simboss18830 says: *them* [7:26:20 AM] simboss18830 says: anyway using SWIG [7:26:36 AM] Cameron Shorter says: ok, makes sense. [7:26:41 AM] simboss18830 says: it should be that hard to build the java bindings for the ECW package directly [7:26:55 AM] simboss18830 says: it pretty much depends in the capabilities you are interested in [7:27:03 AM] simboss18830 says: and in the level of effort you are willing to take [7:28:19 AM] Cameron Shorter says: At the moment, I'd like to know minimum effort required. But also understand the long term goal. [7:28:39 AM] Cameron Shorter says: Ie, I [7:29:07 AM] Cameron Shorter says: I'd like to ensure that the code we write can be built upon later to extend the UDig functionality. [7:30:30 AM] simboss18830 says: I am 99% sure we can import whater we would do in udig with no problems [7:30:47 AM] Cameron Shorter says: Is the code you have written in geotools yet? [7:30:56 AM] simboss18830 says: nope [7:31:10 AM] simboss18830 says: we are not ready to release it yet [7:31:17 AM] simboss18830 says: if we find time [7:31:25 AM] simboss18830 says: we should start releasing part it [7:31:30 AM] simboss18830 says: before the end of the month [7:31:40 AM] simboss18830 says: we are already using it in a couple of project [7:31:44 AM] simboss18830 says: *projects* [7:31:55 AM] Cameron Shorter says: Ok. [7:31:55 AM] simboss18830 says: commercial projects [7:32:28 AM] Cameron Shorter says: How many people are working on the codebase? Is it just you? [7:32:40 AM] simboss18830 says: we are 3 [7:32:48 AM] simboss18830 says: working on it [7:32:59 AM] simboss18830 says: as I was talking about other geotools guys [7:33:02 AM] simboss18830 says: to share the effort [7:33:05 AM] simboss18830 says: once we go public [7:33:32 AM] Cameron Shorter says: What is holding you back from release early/often? [7:33:35 AM] simboss18830 says: I would like top open up a project unde java.net for it [7:33:50 AM] simboss18830 says: there is some refactoring that I need to do [7:33:53 AM] simboss18830 says: first of all [7:33:55 AM] simboss18830 says: and the fact [7:34:42 AM] simboss18830 says: that for the moment most part of the work was done as part of other project [7:34:46 AM] simboss18830 says: *project* [7:34:59 AM] simboss18830 says: I want to spend some weeks on it fufll time [7:35:02 AM] simboss18830 says: before releasing it [7:35:03 AM] Cameron Shorter says: Will you be setting up a seperate project to geotools project. [7:35:23 AM] simboss18830 says: that's my vision [7:35:29 AM] simboss18830 says: but we have been discussing it [7:35:38 AM] Cameron Shorter says: ok. [7:35:44 AM] simboss18830 says: you have to understand that [7:35:48 AM] simboss18830 says: this work in 2 levels [7:35:54 AM] simboss18830 says: first level [7:36:01 AM] simboss18830 says: is done at the ImageIO level [7:36:08 AM] simboss18830 says: and is pretty independent from Geotools [7:36:14 AM] simboss18830 says: is the code to read RenderedImage [7:36:19 AM] simboss18830 says: back from the original format [7:36:25 AM] simboss18830 says: above this [7:36:29 AM] simboss18830 says: we have the geotools plugins [7:36:40 AM] simboss18830 says: which then wrap the renderedimages [7:36:43 AM] simboss18830 says: into coverages [7:37:01 AM] simboss18830 says: Most part of the work we are doing is at ImageIO level [7:37:06 AM] simboss18830 says: so it is independent from geotools [7:37:19 AM] Cameron Shorter says: okWhen you release, what will be the quality of ECW reading? Should I expect that all ECW files will be readable, or is there likely to be the occasional file that doesn't read? [7:37:19 AM] simboss18830 says: our goal is to use it from geotools [7:37:49 AM] Cameron Shorter says: Ditto for JP2000. [7:38:10 AM] simboss18830 says: Talking about limitations [7:38:20 AM] simboss18830 says: and not considering bugs :) [7:38:32 AM] simboss18830 says: the only problem I see may be [7:38:38 AM] simboss18830 says: with really huge images [7:38:47 AM] simboss18830 says: like 64k*64K [7:39:04 AM] simboss18830 says: but this is related to a java limitation [7:39:13 AM] simboss18830 says: that I have some ideas on how to overcome [7:39:15 AM] simboss18830 says: basically [7:39:22 AM] simboss18830 says: it is impossible to build an image [7:39:28 AM] simboss18830 says: even using the deferred mode [7:39:34 AM] simboss18830 says: for which [7:39:49 AM] simboss18830 says: width*height >Integer.MAXIMUM [7:40:02 AM] simboss18830 says: as I said this is a java limitation [7:40:16 AM] simboss18830 says: I have found a workaround [7:40:25 AM] simboss18830 says: but I have had no time ot test it so far [7:40:45 AM] Cameron Shorter says: From a user point of view, how big is Too Big? What sort of area/resolution would that be equivaltent to? [7:41:15 AM] simboss18830 says: let's put it this way [7:41:26 AM] simboss18830 says: I have been able to easil read (and also encode) [7:41:36 AM] simboss18830 says: jp2 iamges that were as big as 3 gb [7:41:54 AM] simboss18830 says: for the moment I have not gone beyond that [7:42:05 AM] simboss18830 says: for ECW [7:42:19 AM] simboss18830 says: I have not been able to go over 500mb since don't have a license right now [7:42:28 AM] simboss18830 says: I have one for kakadu but not for ECW [7:42:48 AM] Cameron Shorter says: Will I need to buy a license to read files bigger than 500mb? [7:43:31 AM] simboss18830 says: nope [7:43:34 AM] simboss18830 says: just to encode them [7:43:40 AM] Cameron Shorter says: ok [7:43:43 AM] simboss18830 says: if you have a geotiff [7:43:48 AM] simboss18830 says: that is [7:43:49 AM] simboss18830 says: 600 mb [7:43:53 AM] simboss18830 says: and you want to encode it [7:43:55 AM] simboss18830 says: as ecw [7:44:00 AM] simboss18830 says: you need a valid license [7:44:06 AM] simboss18830 says: but if some does that for you [7:44:15 AM] simboss18830 says: you can erad it with the free version [7:44:16 AM] Cameron Shorter says: ah - ok. I can see that would be desirable. [7:44:57 AM] Cameron Shorter says: What is performance like? [7:45:14 AM] Cameron Shorter says: How long would it take to read your 3Gig file? [7:45:46 AM] simboss18830 says: well [7:46:04 AM] simboss18830 says: we usually encode in a tiled fafshion [7:46:16 AM] simboss18830 says: so you can read tile by tile [7:46:38 AM] simboss18830 says: if you are interested in playing with ecw inside udig [7:46:41 AM] simboss18830 says: that the way to go [7:46:44 AM] Cameron Shorter says: ok, and tiled reading is fairly fast? [7:47:04 AM] simboss18830 says: tiling+on-the-fly resolution level choice [7:47:29 AM] simboss18830 says: you select the correct resolutions level depending on the viewing resolution [7:47:37 AM] simboss18830 says: and then you load tile by tile [7:47:47 AM] simboss18830 says: to have more responsiveness [7:47:47 AM] Cameron Shorter says: So to implement ECW in UDig will require some customisation in order to set up parameters like tiling. [7:47:52 AM] Cameron Shorter says: ? [7:48:06 AM] simboss18830 says: unfortunately I have a imited experience with udig [7:48:19 AM] simboss18830 says: but I know they use some code I wrote to render coverage [7:48:26 AM] simboss18830 says: it was made for geoserver [7:48:37 AM] simboss18830 says: so I think I would have to tweak a bit [7:48:53 AM] simboss18830 says: but the basic architecture [7:48:57 AM] simboss18830 says: should be alreay in place [7:49:02 AM] simboss18830 says: I know jesse and jody [7:49:05 AM] simboss18830 says: have a version of udig [7:49:12 AM] simboss18830 says: that uses my work to read geotiff file [7:49:20 AM] simboss18830 says: you can ask them to show you [7:49:30 AM] simboss18830 says: how it performs on big geotiff files [7:49:38 AM] simboss18830 says: you will have to preprocess them a bit [7:49:48 AM] Cameron Shorter says: ah - great. [7:49:49 AM] simboss18830 says: as people usually do [7:49:53 AM] simboss18830 says: add overviews [7:50:00 AM] simboss18830 says: tile the image on disk [7:50:06 AM] simboss18830 says: but then performances get pretty good [7:50:14 AM] simboss18830 says: or you could do the same with the geoserver [7:50:32 AM] Cameron Shorter says: Geoserver already has the pre-tiling? [7:51:42 AM] simboss18830 says: geoserver plugins [7:51:52 AM] simboss18830 says: that I have developed [7:52:05 AM] simboss18830 says: look for prebuilt overviews [7:52:10 AM] simboss18830 says: and when they exists [7:52:19 AM] simboss18830 says: the use them as I explained before [7:52:32 AM] simboss18830 says: otherwsei they try to do decimation on the fly [7:52:52 AM] simboss18830 says: it also uses tiling to reduce the amount of data loaded [7:53:51 AM] Cameron Shorter says: Lastly, I'm keen to get some approximate estimates from you with regards to level of effort required to: 1. Get you code released. 2. Get reading ECW working. 3. Get writing EWC working. Answers for alpha level code and release quality code. [7:54:34 AM] simboss18830 says: k [7:54:48 AM] Cameron Shorter says: You are welcome to get back to me on that if you prefer. [7:54:58 AM] simboss18830 says: I was about to write that [7:55:21 AM] simboss18830 says: even because it is pretty late here now :) [7:55:29 AM] simboss18830 says: I'l talk to other guys [7:55:32 AM] Cameron Shorter says: Yes, I understand. [7:55:37 AM] simboss18830 says: and I 'll get back to you [7:55:46 AM] simboss18830 says: within tuesday [7:55:48 AM] simboss18830 says: ok? [7:55:50 AM] Cameron Shorter says: Should we email the logs of this discussion to geotools-devel? [7:56:12 AM] Cameron Shorter says: I suspect Jody and co might be interested in some of your answers. [7:56:49 AM] simboss18830 says: it should be ok [7:56:54 AM] simboss18830 says: go ahead :) [7:57:02 AM] Cameron Shorter says: I need to get the kids ready for school. Thankyou very much for your time. Good night. [7:57:11 AM] simboss18830 says: you are welcome [7:57:18 AM] simboss18830 says: ciao! -- Cameron Shorter Systems Architect, http://lisasoft.com.au Tel: +61 (0)2 8570 5050 Mob: +61 (0)419 142 254 ------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ _______________________________________________ Geotools-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/geotools-devel
