Re: [Geotools-devel] GeoTools / GeoServer PMC meeting - 2018-11-13
On Wed, Nov 14, 2018 at 10:19 AM Andrea Aime wrote: > On Tue, Nov 13, 2018 at 9:52 PM Torben Barsballe < > tbarsba...@boundlessgeo.com> wrote: > >> GeoServer / GeoWebCache doc migration >> > Doc migration started during code sprint. >> >> GS Doc builds fixed last week. >> >> Swagger docs are currently broken. Planning on investigating if the >> swagger docs can be hosted statically. >> > > I keep on not understanding this "statically" bit. The swagger UI > javascript/html combo just needs a URL to the YAML that > defines the API and does everything else on its own, the current issue > afaik is just that the link it's getting is the wrong one > > I think there might be a little more JS magic in there somewhere, given the swagger path uses # - I have not had a chance to look into the exact issue yet. But yes, it is something that *should* work fine statically (i.e. from the filesystem or from the current S3 doc hosting solution); the issue is it doesn't right now. It's entirely possible it is as simple as an incorrect path. Cheers, Torben ___ GeoTools-Devel mailing list GeoTools-Devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/geotools-devel
Re: [Geotools-devel] GeoTools / GeoServer PMC meeting - 2018-11-13
> The idea seems cool, but at the same time, I'm wondering if we should be > investing the money in the project instead, there > are a variety of things that could be done, e.g. start pushing more > aggressively towards code quality (PMD, findbugs and the like in the > builds), > refreshing CITE tests, or even getting more people to help on the user > list somehow. > I am open to ideas here, and have one of my own. For CITE tests I am thinking of making it an OSGeo priority (it says interoperability in the goals but funding such would be action rather than words). I get t-shirts help morale and pride, but will this turn into more > participation? Cause that's what we actually need :-) > I would like to try this t-shirt idea in that it is something I think I can get done (even just using sprint funds) in calendar year 2018. As for recruiting more, the Geoserver developers workshop Ian and myself did was not half bad. Perhaps we could set that up as a better use of these "foss4g sprints" (use it explicitly for promoting participation rather than bug fixing). Check out the AWS OpenJDK long term support move... it's on Java 8: > https://aws.amazon.com/it/corretto/ > > So be mindful of that, Java on AWS seems like it will mean Java 8 for > quite some time. > (at least, they are not mentioning a java 11 version of it) > Actually it is on more than AWS, they have bundles for macOS and even an MSI for windows (allowing it to by managed windows shops). Wow this is a great response, and yeah Java 8 forever apparently. Can you ping me before publishing it, I'd like to help, there is a lot of > work that did not make into twitter and mails, > mostly because the european team passed the current status in an oral way > as opposed to mails. > Blog post is stuck no the previous items :P > > ___ GeoTools-Devel mailing list GeoTools-Devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/geotools-devel
Re: [Geotools-devel] Raster classifier operation for large inputs
Ahhh sorry, totally missed that, my apologies, too much multitasking on my part :) Anyways, +1 from me fwiw. On Wed, Nov 14, 2018 at 11:59 AM Andrea Aime wrote: > On Wed, Nov 14, 2018 at 7:54 PM Justin Deoliveira > wrote: > >> Hey Andrea, >> All of your changes sound good to me. Only question I have is whether >> your proposed change will replace what is there? Or is your thought to add >> some config parameter that would trigger the histogram / approximation >> based method? >> > > New entry in the methods enum, to be used from the caller when approximate > calcuation is desirable. > Citing from my initial mail (yes, it was a bit of a wall of text): > > " Ideally, these would be new entries in the ClassificationMethod > enumeration, say > QUANTILE_HISTOGRAM and NATURAL_BREAKS_HISTOGRAM, and ClassBreaksOpImage > would have an > extra optional parameter to decide the bucket count (with some reasonable > defaults, e.g. 256 for byte data, > 1000 for any other type)." > > >> As for moving the code to jai-text definitely makes sense to me. >> > > Great, thanks for following up! > > Cheers > Andrea > > == > > GeoServer Professional Services from the experts! Visit > http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf > Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa > (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 > http://www.geo-solutions.it http://twitter.com/geosolutions_it > --- *Con riferimento > alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - > Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni > circostanza inerente alla presente email (il suo contenuto, gli eventuali > allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i > destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per > errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le > sarei comunque grato se potesse darmene notizia. This email is intended > only for the person or entity to which it is addressed and may contain > information that is privileged, confidential or otherwise protected from > disclosure. We remind that - as provided by European Regulation 2016/679 > “GDPR” - copying, dissemination or use of this e-mail or the information > herein by anyone other than the intended recipient is prohibited. If you > have received this email by mistake, please notify us immediately by > telephone or e-mail.* > ___ GeoTools-Devel mailing list GeoTools-Devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/geotools-devel
Re: [Geotools-devel] Raster classifier operation for large inputs
On Wed, Nov 14, 2018 at 7:54 PM Justin Deoliveira wrote: > Hey Andrea, > All of your changes sound good to me. Only question I have is whether your > proposed change will replace what is there? Or is your thought to add some > config parameter that would trigger the histogram / approximation based > method? > New entry in the methods enum, to be used from the caller when approximate calcuation is desirable. Citing from my initial mail (yes, it was a bit of a wall of text): " Ideally, these would be new entries in the ClassificationMethod enumeration, say QUANTILE_HISTOGRAM and NATURAL_BREAKS_HISTOGRAM, and ClassBreaksOpImage would have an extra optional parameter to decide the bucket count (with some reasonable defaults, e.g. 256 for byte data, 1000 for any other type)." > As for moving the code to jai-text definitely makes sense to me. > Great, thanks for following up! Cheers Andrea == GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it --- *Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.* ___ GeoTools-Devel mailing list GeoTools-Devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/geotools-devel
Re: [Geotools-devel] Raster classifier operation for large inputs
For what it's worth, I like the plan. Improving current code, making it shareable between multiple parts of the codebase by pushing back to JAI-Ext It will be nice to have in the SLDService. Regards, Simone Giannecchini == GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Simone Giannecchini @simogeo Founder/Director GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) Italy phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 333 8128928 http://www.geo-solutions.it http://twitter.com/geosolutions_it --- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail. On Wed, Nov 14, 2018 at 5:01 PM Andrea Aime wrote: > Hi, > I'm looking into extending the GeoServer SLDService API to work against > raster data too. > The current code in that module works off the vector classification > functions for equal intervals, natural breaks and quantiles. > > When looking at extending it for rasters, I stumbled into > the ClassBreaksOpImage and its subclasses, which > does more or less what I need... with a hitch though: the rasters that I'm > playing with can be large and have floats/doubles > > Looking at the implementation for quantilies and natural breaks I've > noticed that all input values get collected > either in a List or a Map, where the doubles are > the values and the integer is a pixel count. > Mind, this is the same as vector code is doing, but getting to a million > of those in raster space only requires a 1000x1000 > image... and millions of double values (or map entries) take up a lot of > space. I could look into using non boxed > variants, but the issue is not really that one, it's just that keeping > track of all values requires too much space. > > So I'd like to add an approximate calculator instead that collects > histograms, and the works off the result applying the > same logic as today. Ideally, these would be new entries in > the ClassificationMethod enumeration, say > QUANTILE_HISTOGRAM and NATURAL_BREAKS_HISTOGRAM, and ClassBreaksOpImage > would have an > extra optional parameter to decide the bucket count (with some reasonable > defaults, e.g. 256 for byte data, > 1000 for any other type). > Working off histograms has a clear benefit, the size of the working memory > is fixed at the start, and it's possible > to use primitives in the data structure, of course it also means the > resulting classification won't be exact, but > should be close enough. > The downside is that the min/max values need to be known in advance to > build the buckets, so for the histogram > based methods the "extrema" parameter in the ClassBreaksOpImage will be > mandatory (exception thrown if not provided). > > How does that sound? > > Cheers > Andrea > > PS: most operations are in jai-ext, mind if the ClassBreaksOpImage gets > moved there, in its own module? > > -- > > Regards, Andrea Aime == GeoServer Professional Services from the experts! > Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime > @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 > Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 > 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it > --- *Con riferimento > alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - > Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni > circostanza inerente alla presente email (il suo contenuto, gli eventuali > allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i > destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per > errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le > sarei comunque grato se potesse darmene notizia. This email is intended > only for the person or entity to which it is addressed and may contain > information that is privileged, confidential or otherwise protected from > disclosure. We remind that
Re: [Geotools-devel] Raster classifier operation for large inputs
Hey Andrea, All of your changes sound good to me. Only question I have is whether your proposed change will replace what is there? Or is your thought to add some config parameter that would trigger the histogram / approximation based method? As for moving the code to jai-text definitely makes sense to me. -Justin On Wed, Nov 14, 2018 at 10:00 AM Andrea Aime wrote: > Hi, > I'm looking into extending the GeoServer SLDService API to work against > raster data too. > The current code in that module works off the vector classification > functions for equal intervals, natural breaks and quantiles. > > When looking at extending it for rasters, I stumbled into > the ClassBreaksOpImage and its subclasses, which > does more or less what I need... with a hitch though: the rasters that I'm > playing with can be large and have floats/doubles > > Looking at the implementation for quantilies and natural breaks I've > noticed that all input values get collected > either in a List or a Map, where the doubles are > the values and the integer is a pixel count. > Mind, this is the same as vector code is doing, but getting to a million > of those in raster space only requires a 1000x1000 > image... and millions of double values (or map entries) take up a lot of > space. I could look into using non boxed > variants, but the issue is not really that one, it's just that keeping > track of all values requires too much space. > > So I'd like to add an approximate calculator instead that collects > histograms, and the works off the result applying the > same logic as today. Ideally, these would be new entries in > the ClassificationMethod enumeration, say > QUANTILE_HISTOGRAM and NATURAL_BREAKS_HISTOGRAM, and ClassBreaksOpImage > would have an > extra optional parameter to decide the bucket count (with some reasonable > defaults, e.g. 256 for byte data, > 1000 for any other type). > Working off histograms has a clear benefit, the size of the working memory > is fixed at the start, and it's possible > to use primitives in the data structure, of course it also means the > resulting classification won't be exact, but > should be close enough. > The downside is that the min/max values need to be known in advance to > build the buckets, so for the histogram > based methods the "extrema" parameter in the ClassBreaksOpImage will be > mandatory (exception thrown if not provided). > > How does that sound? > > Cheers > Andrea > > PS: most operations are in jai-ext, mind if the ClassBreaksOpImage gets > moved there, in its own module? > > -- > > Regards, Andrea Aime == GeoServer Professional Services from the experts! > Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime > @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 > Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 > 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it > --- *Con riferimento > alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - > Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni > circostanza inerente alla presente email (il suo contenuto, gli eventuali > allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i > destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per > errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le > sarei comunque grato se potesse darmene notizia. This email is intended > only for the person or entity to which it is addressed and may contain > information that is privileged, confidential or otherwise protected from > disclosure. We remind that - as provided by European Regulation 2016/679 > “GDPR” - copying, dissemination or use of this e-mail or the information > herein by anyone other than the intended recipient is prohibited. If you > have received this email by mistake, please notify us immediately by > telephone or e-mail.* > ___ GeoTools-Devel mailing list GeoTools-Devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/geotools-devel
Re: [Geotools-devel] GeoTools / GeoServer PMC meeting - 2018-11-13
On Tue, Nov 13, 2018 at 9:52 PM Torben Barsballe < tbarsba...@boundlessgeo.com> wrote: > Ideas for remained of 2018: > >- > >"code-sprint" t-shirts >- > > Kevin: Maybe logos of participating projects on jigsaw puzzle > pieces? > - > >"project specific shirts" to send out to contributors >- > > single color shirt two-color logo > - > > "I am a GeoServer Contributor" on the front > - > > OSGeo Project log on the back (very small) > - > > Jody proposes picking up $1k and taking to marketing committee for > the above > > The idea seems cool, but at the same time, I'm wondering if we should be investing the money in the project instead, there are a variety of things that could be done, e.g. start pushing more aggressively towards code quality (PMD, findbugs and the like in the builds), refreshing CITE tests, or even getting more people to help on the user list somehow. I get t-shirts help morale and pride, but will this turn into more participation? Cause that's what we actually need :-) > Start thinking about 2019 budget, need to make the request in December :) > >- > >JAI if it happens in 2019 it will be a large activity >- > >java roadmap will be an ongoing cost >- > > Do we want to focus on the LTS? probably > > Check out the AWS OpenJDK long term support move... it's on Java 8: https://aws.amazon.com/it/corretto/ So be mindful of that, Java on AWS seems like it will mean Java 8 for quite some time. (at least, they are not mentioning a java 11 version of it) > >- > > Running on last LTS and current "master" would be least risk > - > >Send officers to AGM? >- > >OSGeo code sprint participation? > > This would work, if we can find a topic that drives people there (something actionable with immediate benefit) > JDK 11 Sprint Follow-up > >- > >Jody provided an update to the board with respect to sponsorship, >budget and osgeo hosting the event and what it means to the projects. >- > >Aside: reporting back positive feedback from customers (impressed this >was done at all, and that it was "done" so quickly) >- > >Finish up sprint split-package goal and unfreezing master [DONE] >- > > https://github.com/geotools/geotools/pull/2154, > https://github.com/geoserver/geoserver/pull/3217 > - > > What else is left - see > > https://docs.google.com/spreadsheets/d/1oE6mU4jp-ZL5PebgXf-fuhtf7MY5dzSwPqpMtrzdZ94/edit#gid=2055024842&range=A513 > - > > App-schema split packages > - > > ArcSDE internal split package > - > > sqlite/spatialite split package > - > > JAIExt scale split package > - > > anything else > - > > jody has a split of gt-metadata into gt-metadata and gt-util > - > >Running up t-shirts for sprint participants >- > > jody has taken an action to ask marketing committee for design and > quote > - > >Documentation >- > > We need migration instructions for GeoTools > - > > Turn the spreadsheet into sed script? > - > > For GeoServer we to update the Production Considerations section > for Java 11 > - > > https://docs.geoserver.org/latest/en/user/production/java.html > - > >Blog post: >- > > Publish once upgrade and production consideration docs are ready to > link to > - > > We have a responsibility to thank sponsors in the blog post, and > release announcements. > > Can you ping me before publishing it, I'd like to help, there is a lot of work that did not make into twitter and mails, mostly because the european team passed the current status in an oral way as opposed to mails. > 2.14.1 GeoServer Release - Call for volunteers > >- > >Jody has volunteered >- > >Torben as backup / GWC > > Thank you! > > Action: > >- > >sent "release train" email for friday (since this falls on the weekend) > > > GeoServer / GeoWebCache doc migration > > Doc migration started during code sprint. > > GS Doc builds fixed last week. > > Swagger docs are currently broken. Planning on investigating if the > swagger docs can be hosted statically. > I keep on not understanding this "statically" bit. The swagger UI javascript/html combo just needs a URL to the YAML that defines the API and does everything else on its own, the current issue afaik is just that the link it's getting is the wrong one Cheers Andrea == GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it -
[Geotools-devel] Raster classifier operation for large inputs
Hi, I'm looking into extending the GeoServer SLDService API to work against raster data too. The current code in that module works off the vector classification functions for equal intervals, natural breaks and quantiles. When looking at extending it for rasters, I stumbled into the ClassBreaksOpImage and its subclasses, which does more or less what I need... with a hitch though: the rasters that I'm playing with can be large and have floats/doubles Looking at the implementation for quantilies and natural breaks I've noticed that all input values get collected either in a List or a Map, where the doubles are the values and the integer is a pixel count. Mind, this is the same as vector code is doing, but getting to a million of those in raster space only requires a 1000x1000 image... and millions of double values (or map entries) take up a lot of space. I could look into using non boxed variants, but the issue is not really that one, it's just that keeping track of all values requires too much space. So I'd like to add an approximate calculator instead that collects histograms, and the works off the result applying the same logic as today. Ideally, these would be new entries in the ClassificationMethod enumeration, say QUANTILE_HISTOGRAM and NATURAL_BREAKS_HISTOGRAM, and ClassBreaksOpImage would have an extra optional parameter to decide the bucket count (with some reasonable defaults, e.g. 256 for byte data, 1000 for any other type). Working off histograms has a clear benefit, the size of the working memory is fixed at the start, and it's possible to use primitives in the data structure, of course it also means the resulting classification won't be exact, but should be close enough. The downside is that the min/max values need to be known in advance to build the buckets, so for the histogram based methods the "extrema" parameter in the ClassBreaksOpImage will be mandatory (exception thrown if not provided). How does that sound? Cheers Andrea PS: most operations are in jai-ext, mind if the ClassBreaksOpImage gets moved there, in its own module? -- Regards, Andrea Aime == GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it --- *Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.* ___ GeoTools-Devel mailing list GeoTools-Devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/geotools-devel
