Re: [Geotools-devel] Solr heatmap as a grid coverage

2016-04-14 Thread Justin Deoliveira 
Thanks Simone. See below.

On Thu, Apr 14, 2016 at 10:55 AM Simone Giannecchini <
simone.giannecch...@geo-solutions.it> wrote:

> Ciao Justing,
> please, read below
>
>
>
> On Thu, Apr 14, 2016 at 6:18 PM, Justin Deoliveira 
> wrote:
> > Thanks for the input Simone. Comments inline below.
> >
> > On Thu, Apr 14, 2016 at 10:07 AM Simone Giannecchini
> >  wrote:
> >>
> >> See below..
> >>
> >> Regards,
> >> Simone Giannecchini
> >> ==
> >> GeoServer Professional Services from the experts!
> >> Visit http://goo.gl/it488V for more information.
> >> ==
> >> Ing. Simone Giannecchini
> >> @simogeo
> >> Founder/Director
> >>
> >> GeoSolutions S.A.S.
> >> Via di Montramito 3/A
> >> 55054  Massarosa (LU)
> >> Italy
> >> phone: +39 0584 962313
> >> fax: +39 0584 1660272
> >> mob:   +39  333 8128928
> >>
> >> http://www.geo-solutions.it
> >> http://twitter.com/geosolutions_it
> >>
> >> ---
> >> AVVERTENZE AI SENSI DEL D.Lgs. 196/2003
> >> Le informazioni contenute in questo messaggio di posta elettronica e/o
> >> nel/i file/s allegato/i sono da considerarsi strettamente riservate.
> >> Il loro utilizzo è consentito esclusivamente al destinatario del
> >> messaggio, per le finalità indicate nel messaggio stesso. Qualora
> >> riceviate questo messaggio senza esserne il destinatario, Vi preghiamo
> >> cortesemente di darcene notizia via e-mail e di procedere alla
> >> distruzione del messaggio stesso, cancellandolo dal Vostro sistema.
> >> Conservare il messaggio stesso, divulgarlo anche in parte,
> >> distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità
> >> diverse, costituisce comportamento contrario ai principi dettati dal
> >> D.Lgs. 196/2003.
> >>
> >> The information in this message and/or attachments, is intended solely
> >> for the attention and use of the named addressee(s) and may be
> >> confidential or proprietary in nature or covered by the provisions of
> >> privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New
> >> Data Protection Code).Any use not in accord with its purpose, any
> >> disclosure, reproduction, copying, distribution, or either
> >> dissemination, either whole or partial, is strictly forbidden except
> >> previous formal approval of the named addressee(s). If you are not the
> >> intended recipient, please contact immediately the sender by
> >> telephone, fax or e-mail and delete the information in this message
> >> that has been received in error. The sender does not give any warranty
> >> or accept liability as the content, accuracy or completeness of sent
> >> messages and accepts no responsibility  for changes made after they
> >> were sent or for other risks which arise as a result of e-mail
> >> transmission, viruses, etc.
> >>
> >>
> >> On Thu, Apr 14, 2016 at 4:10 PM, Justin Deoliveira 
> >> wrote:
> >> > Hey folks.
> >> >
> >> > I just pushed a branch into my repo that contains a coverage reader
> for
> >> > solr
> >> > heatmaps:
> >> >
> >> >
> >> >
> >> >
> https://github.com/geotools/geotools/compare/master...jdeolive:solr-heatmap
> >> >
> >> > A few notes.
> >> >
> >> > First, I tried playing around with using simple interpolation for
> >> > smoothing.
> >> > While it provided something nicer looking than the raw heatmap it
> didn’t
> >> > seem to go far enough. Aside from the fact that it doesn’t seem the
> >> > scale
> >> > operations do anything when translate and scale parameters are
> identity?
> >> > Perhaps that was just user error on my part.
> >> >
> >>
> >> Strange, it should do the simple interpolation. It might be worth
> >> investingating further.
> >> (investigation ongoing)
> >>
> >> Ok, had a quick look at the scale class in JAI-EXT ( I am assuming you
> >> are using geoserver master and have this activated)
> >> and apparently there is an overoptimistic optimization that bypasses
> >> the interpolation:
> >>
> >>
> https://github.com/geosolutions-it/jai-ext/blob/master/jt-scale/src/main/java/it/geosolutions/jaiext/scale/ScaleCRIF.java#L110
> >>
> >> I'll talk to daniele to improve this.
> >>
> > Much appreciated!
>
> I have put this on his table.
> Let's see what he says.
>
> >>
> >>
> >> > That said I didn’t quite like adding any decisions regarding
> >> > visualization
> >> > into the coverage reader itself, so I decided to take all of that out
> >> > and
> >> > instead do it via rendering transformations. I think this gives us and
> >> > the
> >> > user the most flexibility. And also allows us to try out different
> >> > visualization techniques quite easily.
> >> >
> >> > In support of this idea on that branch there are two new processes:
> >> >
> >> > 1. ConvolveCoverageProcess - Exposes the Convolve operation with a few
> >> > of
> >> > it’s parameters
> >>
> >> Nice idea.
> >> There should ne no convolve preocess in GT, but there convolve in JAI
> >> and JAI-Ext
> >> (https://github.com/geosolutions-it/jai-ext/tree/master/jt-convolve).
> >>
> >> > 2. NormalizeC

Re: [Geotools-devel] Solr heatmap as a grid coverage

2016-04-14 Thread Simone Giannecchini 
Ciao Justing,
please, read below

Regards,
Simone Giannecchini
==
GeoServer Professional Services from the experts!
Visit http://goo.gl/it488V for more information.
==
Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via di Montramito 3/A
55054  Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob:   +39  333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---
AVVERTENZE AI SENSI DEL D.Lgs. 196/2003
Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate.
Il loro utilizzo è consentito esclusivamente al destinatario del
messaggio, per le finalità indicate nel messaggio stesso. Qualora
riceviate questo messaggio senza esserne il destinatario, Vi preghiamo
cortesemente di darcene notizia via e-mail e di procedere alla
distruzione del messaggio stesso, cancellandolo dal Vostro sistema.
Conservare il messaggio stesso, divulgarlo anche in parte,
distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità
diverse, costituisce comportamento contrario ai principi dettati dal
D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely
for the attention and use of the named addressee(s) and may be
confidential or proprietary in nature or covered by the provisions of
privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New
Data Protection Code).Any use not in accord with its purpose, any
disclosure, reproduction, copying, distribution, or either
dissemination, either whole or partial, is strictly forbidden except
previous formal approval of the named addressee(s). If you are not the
intended recipient, please contact immediately the sender by
telephone, fax or e-mail and delete the information in this message
that has been received in error. The sender does not give any warranty
or accept liability as the content, accuracy or completeness of sent
messages and accepts no responsibility  for changes made after they
were sent or for other risks which arise as a result of e-mail
transmission, viruses, etc.


On Thu, Apr 14, 2016 at 6:18 PM, Justin Deoliveira  wrote:
> Thanks for the input Simone. Comments inline below.
>
> On Thu, Apr 14, 2016 at 10:07 AM Simone Giannecchini
>  wrote:
>>
>> See below..
>>
>> Regards,
>> Simone Giannecchini
>> ==
>> GeoServer Professional Services from the experts!
>> Visit http://goo.gl/it488V for more information.
>> ==
>> Ing. Simone Giannecchini
>> @simogeo
>> Founder/Director
>>
>> GeoSolutions S.A.S.
>> Via di Montramito 3/A
>> 55054  Massarosa (LU)
>> Italy
>> phone: +39 0584 962313
>> fax: +39 0584 1660272
>> mob:   +39  333 8128928
>>
>> http://www.geo-solutions.it
>> http://twitter.com/geosolutions_it
>>
>> ---
>> AVVERTENZE AI SENSI DEL D.Lgs. 196/2003
>> Le informazioni contenute in questo messaggio di posta elettronica e/o
>> nel/i file/s allegato/i sono da considerarsi strettamente riservate.
>> Il loro utilizzo è consentito esclusivamente al destinatario del
>> messaggio, per le finalità indicate nel messaggio stesso. Qualora
>> riceviate questo messaggio senza esserne il destinatario, Vi preghiamo
>> cortesemente di darcene notizia via e-mail e di procedere alla
>> distruzione del messaggio stesso, cancellandolo dal Vostro sistema.
>> Conservare il messaggio stesso, divulgarlo anche in parte,
>> distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità
>> diverse, costituisce comportamento contrario ai principi dettati dal
>> D.Lgs. 196/2003.
>>
>> The information in this message and/or attachments, is intended solely
>> for the attention and use of the named addressee(s) and may be
>> confidential or proprietary in nature or covered by the provisions of
>> privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New
>> Data Protection Code).Any use not in accord with its purpose, any
>> disclosure, reproduction, copying, distribution, or either
>> dissemination, either whole or partial, is strictly forbidden except
>> previous formal approval of the named addressee(s). If you are not the
>> intended recipient, please contact immediately the sender by
>> telephone, fax or e-mail and delete the information in this message
>> that has been received in error. The sender does not give any warranty
>> or accept liability as the content, accuracy or completeness of sent
>> messages and accepts no responsibility  for changes made after they
>> were sent or for other risks which arise as a result of e-mail
>> transmission, viruses, etc.
>>
>>
>> On Thu, Apr 14, 2016 at 4:10 PM, Justin Deoliveira 
>> wrote:
>> > Hey folks.
>> >
>> > I just pushed a branch into my repo that contains a coverage reader for
>> > solr
>> > heatmaps:
>> >
>> >
>> >
>> > https://github.com/geotools/geotools/compare/master...jdeolive:solr-heatmap
>> >
>> > A few notes.
>> >
>> > F

Re: [Geotools-devel] Solr heatmap as a grid coverage

2016-04-14 Thread Justin Deoliveira 
Thanks for the input Simone. Comments inline below.

On Thu, Apr 14, 2016 at 10:07 AM Simone Giannecchini <
simone.giannecch...@geo-solutions.it> wrote:

> See below..
>
> Regards,
> Simone Giannecchini
> ==
> GeoServer Professional Services from the experts!
> Visit http://goo.gl/it488V for more information.
> ==
> Ing. Simone Giannecchini
> @simogeo
> Founder/Director
>
> GeoSolutions S.A.S.
> Via di Montramito 3/A
> 55054  Massarosa (LU)
> Italy
> phone: +39 0584 962313
> fax: +39 0584 1660272
> mob:   +39  333 8128928
>
> http://www.geo-solutions.it
> http://twitter.com/geosolutions_it
>
> ---
> AVVERTENZE AI SENSI DEL D.Lgs. 196/2003
> Le informazioni contenute in questo messaggio di posta elettronica e/o
> nel/i file/s allegato/i sono da considerarsi strettamente riservate.
> Il loro utilizzo è consentito esclusivamente al destinatario del
> messaggio, per le finalità indicate nel messaggio stesso. Qualora
> riceviate questo messaggio senza esserne il destinatario, Vi preghiamo
> cortesemente di darcene notizia via e-mail e di procedere alla
> distruzione del messaggio stesso, cancellandolo dal Vostro sistema.
> Conservare il messaggio stesso, divulgarlo anche in parte,
> distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità
> diverse, costituisce comportamento contrario ai principi dettati dal
> D.Lgs. 196/2003.
>
> The information in this message and/or attachments, is intended solely
> for the attention and use of the named addressee(s) and may be
> confidential or proprietary in nature or covered by the provisions of
> privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New
> Data Protection Code).Any use not in accord with its purpose, any
> disclosure, reproduction, copying, distribution, or either
> dissemination, either whole or partial, is strictly forbidden except
> previous formal approval of the named addressee(s). If you are not the
> intended recipient, please contact immediately the sender by
> telephone, fax or e-mail and delete the information in this message
> that has been received in error. The sender does not give any warranty
> or accept liability as the content, accuracy or completeness of sent
> messages and accepts no responsibility  for changes made after they
> were sent or for other risks which arise as a result of e-mail
> transmission, viruses, etc.
>
>
> On Thu, Apr 14, 2016 at 4:10 PM, Justin Deoliveira 
> wrote:
> > Hey folks.
> >
> > I just pushed a branch into my repo that contains a coverage reader for
> solr
> > heatmaps:
> >
> >
> >
> https://github.com/geotools/geotools/compare/master...jdeolive:solr-heatmap
> >
> > A few notes.
> >
> > First, I tried playing around with using simple interpolation for
> smoothing.
> > While it provided something nicer looking than the raw heatmap it didn’t
> > seem to go far enough. Aside from the fact that it doesn’t seem the scale
> > operations do anything when translate and scale parameters are identity?
> > Perhaps that was just user error on my part.
> >
>
> Strange, it should do the simple interpolation. It might be worth
> investingating further.
> (investigation ongoing)
>
> Ok, had a quick look at the scale class in JAI-EXT ( I am assuming you
> are using geoserver master and have this activated)
> and apparently there is an overoptimistic optimization that bypasses
> the interpolation:
>
> https://github.com/geosolutions-it/jai-ext/blob/master/jt-scale/src/main/java/it/geosolutions/jaiext/scale/ScaleCRIF.java#L110
>
> I'll talk to daniele to improve this.
>
> Much appreciated!

>
> > That said I didn’t quite like adding any decisions regarding
> visualization
> > into the coverage reader itself, so I decided to take all of that out and
> > instead do it via rendering transformations. I think this gives us and
> the
> > user the most flexibility. And also allows us to try out different
> > visualization techniques quite easily.
> >
> > In support of this idea on that branch there are two new processes:
> >
> > 1. ConvolveCoverageProcess - Exposes the Convolve operation with a few of
> > it’s parameters
>
> Nice idea.
> There should ne no convolve preocess in GT, but there convolve in JAI
> and JAI-Ext (
> https://github.com/geosolutions-it/jai-ext/tree/master/jt-convolve).
>
> > 2. NormalizeCoverageProcess - “Normalizes” (please correct my terminology
> > here) a raster by first calculating the max value, and then running it
> > through DivideByConst to provide a raster whose values fall between the
> > range [0,1].
>
> This approach might interact badly with tiled requests since you would
> normalize the values using the Min/Max values in the local tile.
> Aside, I don't see how this per se would help with making the raster
> look smoother, no? Correct me if I am wrong.
>
>
> Right. Tiled maps is not something I’ve really considered yet… mostly
because my experience with heatmaps is that they in general don’t work well
will tiled m

Re: [Geotools-devel] Solr heatmap as a grid coverage

2016-04-14 Thread Simone Giannecchini 
See below..

Regards,
Simone Giannecchini
==
GeoServer Professional Services from the experts!
Visit http://goo.gl/it488V for more information.
==
Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via di Montramito 3/A
55054  Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob:   +39  333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---
AVVERTENZE AI SENSI DEL D.Lgs. 196/2003
Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate.
Il loro utilizzo è consentito esclusivamente al destinatario del
messaggio, per le finalità indicate nel messaggio stesso. Qualora
riceviate questo messaggio senza esserne il destinatario, Vi preghiamo
cortesemente di darcene notizia via e-mail e di procedere alla
distruzione del messaggio stesso, cancellandolo dal Vostro sistema.
Conservare il messaggio stesso, divulgarlo anche in parte,
distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità
diverse, costituisce comportamento contrario ai principi dettati dal
D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely
for the attention and use of the named addressee(s) and may be
confidential or proprietary in nature or covered by the provisions of
privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New
Data Protection Code).Any use not in accord with its purpose, any
disclosure, reproduction, copying, distribution, or either
dissemination, either whole or partial, is strictly forbidden except
previous formal approval of the named addressee(s). If you are not the
intended recipient, please contact immediately the sender by
telephone, fax or e-mail and delete the information in this message
that has been received in error. The sender does not give any warranty
or accept liability as the content, accuracy or completeness of sent
messages and accepts no responsibility  for changes made after they
were sent or for other risks which arise as a result of e-mail
transmission, viruses, etc.


On Thu, Apr 14, 2016 at 4:10 PM, Justin Deoliveira  wrote:
> Hey folks.
>
> I just pushed a branch into my repo that contains a coverage reader for solr
> heatmaps:
>
>
> https://github.com/geotools/geotools/compare/master...jdeolive:solr-heatmap
>
> A few notes.
>
> First, I tried playing around with using simple interpolation for smoothing.
> While it provided something nicer looking than the raw heatmap it didn’t
> seem to go far enough. Aside from the fact that it doesn’t seem the scale
> operations do anything when translate and scale parameters are identity?
> Perhaps that was just user error on my part.
>

Strange, it should do the simple interpolation. It might be worth
investingating further.
(investigation ongoing)

Ok, had a quick look at the scale class in JAI-EXT ( I am assuming you
are using geoserver master and have this activated)
and apparently there is an overoptimistic optimization that bypasses
the interpolation:
https://github.com/geosolutions-it/jai-ext/blob/master/jt-scale/src/main/java/it/geosolutions/jaiext/scale/ScaleCRIF.java#L110

I'll talk to daniele to improve this.


> That said I didn’t quite like adding any decisions regarding visualization
> into the coverage reader itself, so I decided to take all of that out and
> instead do it via rendering transformations. I think this gives us and the
> user the most flexibility. And also allows us to try out different
> visualization techniques quite easily.
>
> In support of this idea on that branch there are two new processes:
>
> 1. ConvolveCoverageProcess - Exposes the Convolve operation with a few of
> it’s parameters

Nice idea.
There should ne no convolve preocess in GT, but there convolve in JAI
and JAI-Ext 
(https://github.com/geosolutions-it/jai-ext/tree/master/jt-convolve).

> 2. NormalizeCoverageProcess - “Normalizes” (please correct my terminology
> here) a raster by first calculating the max value, and then running it
> through DivideByConst to provide a raster whose values fall between the
> range [0,1].

This approach might interact badly with tiled requests since you would
normalize the values using the Min/Max values in the local tile.
Aside, I don't see how this per se would help with making the raster
look smoother, no? Correct me if I am wrong.



> The rationale for (2) is to have a “stable” value space for the color map,
> otherwise you don’t really know what the min / max values are going to be.
> If there some existing code in the code base to handle this situation that
> doens’t involve adding any new code please let me know.
>
> Feedback welcome!

Gave this feedback quickly, hope this helps.


>
> Thanks.
>
> -Justin
>
>
> On Fri, Apr 8, 2016 at 8:31 AM Simone Giannecchini
>  wrote:
>>
>> Let me know how it goes,
>> we have some potential work in the pipeline for Solr and this is
>> _ve

Re: [Geotools-devel] Solr heatmap as a grid coverage

2016-04-14 Thread Justin Deoliveira 
Hey folks.

I just pushed a branch into my repo that contains a coverage reader for
solr heatmaps:


https://github.com/geotools/geotools/compare/master...jdeolive:solr-heatmap

A few notes.

First, I tried playing around with using simple interpolation for
smoothing. While it provided something nicer looking than the raw heatmap
it didn’t seem to go far enough. Aside from the fact that it doesn’t seem
the scale operations do anything when translate and scale parameters are
identity? Perhaps that was just user error on my part.

That said I didn’t quite like adding any decisions regarding visualization
into the coverage reader itself, so I decided to take all of that out and
instead do it via rendering transformations. I think this gives us and the
user the most flexibility. And also allows us to try out different
visualization techniques quite easily.

In support of this idea on that branch there are two new processes:

1. ConvolveCoverageProcess - Exposes the Convolve operation with a few of
it’s parameters
2. NormalizeCoverageProcess - “Normalizes” (please correct my terminology
here) a raster by first calculating the max value, and then running it
through DivideByConst to provide a raster whose values fall between the
range [0,1].

The rationale for (2) is to have a “stable” value space for the color map,
otherwise you don’t really know what the min / max values are going to be.
If there some existing code in the code base to handle this situation that
doens’t involve adding any new code please let me know.

Feedback welcome!

Thanks.

-Justin


On Fri, Apr 8, 2016 at 8:31 AM Simone Giannecchini <
simone.giannecch...@geo-solutions.it> wrote:

> Let me know how it goes,
> we have some potential work in the pipeline for Solr and this is
> _very_ interesting for us.
>
> Regards,
> Simone Giannecchini
> ==
> GeoServer Professional Services from the experts!
> Visit http://goo.gl/it488V for more information.
> ==
> Ing. Simone Giannecchini
> @simogeo
> Founder/Director
>
> GeoSolutions S.A.S.
> Via di Montramito 3/A
> 55054  Massarosa (LU)
> Italy
> phone: +39 0584 962313
> fax: +39 0584 1660272
> mob:   +39  333 8128928
>
> http://www.geo-solutions.it
> http://twitter.com/geosolutions_it
>
> ---
> AVVERTENZE AI SENSI DEL D.Lgs. 196/2003
> Le informazioni contenute in questo messaggio di posta elettronica e/o
> nel/i file/s allegato/i sono da considerarsi strettamente riservate.
> Il loro utilizzo è consentito esclusivamente al destinatario del
> messaggio, per le finalità indicate nel messaggio stesso. Qualora
> riceviate questo messaggio senza esserne il destinatario, Vi preghiamo
> cortesemente di darcene notizia via e-mail e di procedere alla
> distruzione del messaggio stesso, cancellandolo dal Vostro sistema.
> Conservare il messaggio stesso, divulgarlo anche in parte,
> distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità
> diverse, costituisce comportamento contrario ai principi dettati dal
> D.Lgs. 196/2003.
>
> The information in this message and/or attachments, is intended solely
> for the attention and use of the named addressee(s) and may be
> confidential or proprietary in nature or covered by the provisions of
> privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New
> Data Protection Code).Any use not in accord with its purpose, any
> disclosure, reproduction, copying, distribution, or either
> dissemination, either whole or partial, is strictly forbidden except
> previous formal approval of the named addressee(s). If you are not the
> intended recipient, please contact immediately the sender by
> telephone, fax or e-mail and delete the information in this message
> that has been received in error. The sender does not give any warranty
> or accept liability as the content, accuracy or completeness of sent
> messages and accepts no responsibility  for changes made after they
> were sent or for other risks which arise as a result of e-mail
> transmission, viruses, etc.
>
>
> On Fri, Apr 8, 2016 at 4:25 PM, Justin Deoliveira 
> wrote:
> > Thanks a lot for the feedback guys. Simply using interpolation to do the
> > smoothing makes a lot of sense! Thanks Simone. I am going to experiment a
> > bit more and I’ll come back with my findings.
> >
> > On Fri, Apr 8, 2016 at 7:09 AM Simone Giannecchini
> >  wrote:
> >>
> >> Ciao Justin,
> >> adding to Andrea's, please, read below...
> >>
> >> Regards,
> >> Simone Giannecchini
> >> ==
> >> GeoServer Professional Services from the experts!
> >> Visit http://goo.gl/it488V for more information.
> >> ==
> >> Ing. Simone Giannecchini
> >> @simogeo
> >> Founder/Director
> >>
> >> GeoSolutions S.A.S.
> >> Via di Montramito 3/A
> >> 55054  Massarosa (LU)
> >> Italy
> >> phone: +39 0584 962313
> >> fax: +39 0584 1660272
> >> mob:   +39  333 8128928
> >>
> >> http://www.geo-solutions.it
> >> http://twitter.com/geosolutions_it
> >>
> >> -

Re: [Geotools-devel] Solr heatmap as a grid coverage

2016-04-08 Thread Simone Giannecchini 
Let me know how it goes,
we have some potential work in the pipeline for Solr and this is
_very_ interesting for us.

Regards,
Simone Giannecchini
==
GeoServer Professional Services from the experts!
Visit http://goo.gl/it488V for more information.
==
Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via di Montramito 3/A
55054  Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob:   +39  333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---
AVVERTENZE AI SENSI DEL D.Lgs. 196/2003
Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate.
Il loro utilizzo è consentito esclusivamente al destinatario del
messaggio, per le finalità indicate nel messaggio stesso. Qualora
riceviate questo messaggio senza esserne il destinatario, Vi preghiamo
cortesemente di darcene notizia via e-mail e di procedere alla
distruzione del messaggio stesso, cancellandolo dal Vostro sistema.
Conservare il messaggio stesso, divulgarlo anche in parte,
distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità
diverse, costituisce comportamento contrario ai principi dettati dal
D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely
for the attention and use of the named addressee(s) and may be
confidential or proprietary in nature or covered by the provisions of
privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New
Data Protection Code).Any use not in accord with its purpose, any
disclosure, reproduction, copying, distribution, or either
dissemination, either whole or partial, is strictly forbidden except
previous formal approval of the named addressee(s). If you are not the
intended recipient, please contact immediately the sender by
telephone, fax or e-mail and delete the information in this message
that has been received in error. The sender does not give any warranty
or accept liability as the content, accuracy or completeness of sent
messages and accepts no responsibility  for changes made after they
were sent or for other risks which arise as a result of e-mail
transmission, viruses, etc.


On Fri, Apr 8, 2016 at 4:25 PM, Justin Deoliveira  wrote:
> Thanks a lot for the feedback guys. Simply using interpolation to do the
> smoothing makes a lot of sense! Thanks Simone. I am going to experiment a
> bit more and I’ll come back with my findings.
>
> On Fri, Apr 8, 2016 at 7:09 AM Simone Giannecchini
>  wrote:
>>
>> Ciao Justin,
>> adding to Andrea's, please, read below...
>>
>> Regards,
>> Simone Giannecchini
>> ==
>> GeoServer Professional Services from the experts!
>> Visit http://goo.gl/it488V for more information.
>> ==
>> Ing. Simone Giannecchini
>> @simogeo
>> Founder/Director
>>
>> GeoSolutions S.A.S.
>> Via di Montramito 3/A
>> 55054  Massarosa (LU)
>> Italy
>> phone: +39 0584 962313
>> fax: +39 0584 1660272
>> mob:   +39  333 8128928
>>
>> http://www.geo-solutions.it
>> http://twitter.com/geosolutions_it
>>
>> ---
>> AVVERTENZE AI SENSI DEL D.Lgs. 196/2003
>> Le informazioni contenute in questo messaggio di posta elettronica e/o
>> nel/i file/s allegato/i sono da considerarsi strettamente riservate.
>> Il loro utilizzo è consentito esclusivamente al destinatario del
>> messaggio, per le finalità indicate nel messaggio stesso. Qualora
>> riceviate questo messaggio senza esserne il destinatario, Vi preghiamo
>> cortesemente di darcene notizia via e-mail e di procedere alla
>> distruzione del messaggio stesso, cancellandolo dal Vostro sistema.
>> Conservare il messaggio stesso, divulgarlo anche in parte,
>> distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità
>> diverse, costituisce comportamento contrario ai principi dettati dal
>> D.Lgs. 196/2003.
>>
>> The information in this message and/or attachments, is intended solely
>> for the attention and use of the named addressee(s) and may be
>> confidential or proprietary in nature or covered by the provisions of
>> privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New
>> Data Protection Code).Any use not in accord with its purpose, any
>> disclosure, reproduction, copying, distribution, or either
>> dissemination, either whole or partial, is strictly forbidden except
>> previous formal approval of the named addressee(s). If you are not the
>> intended recipient, please contact immediately the sender by
>> telephone, fax or e-mail and delete the information in this message
>> that has been received in error. The sender does not give any warranty
>> or accept liability as the content, accuracy or completeness of sent
>> messages and accepts no responsibility  for changes made after they
>> were sent or for other risks which arise as a result of e-mail
>> transmission, viruses, etc.
>>
>>
>> On Thu, Apr 7, 2016 at 7:56 PM, Andrea Aime
>>

Re: [Geotools-devel] Solr heatmap as a grid coverage

2016-04-08 Thread Justin Deoliveira 
Thanks a lot for the feedback guys. Simply using interpolation to do the
smoothing makes a lot of sense! Thanks Simone. I am going to experiment a
bit more and I’ll come back with my findings.

On Fri, Apr 8, 2016 at 7:09 AM Simone Giannecchini <
simone.giannecch...@geo-solutions.it> wrote:

> Ciao Justin,
> adding to Andrea's, please, read below...
>
> Regards,
> Simone Giannecchini
> ==
> GeoServer Professional Services from the experts!
> Visit http://goo.gl/it488V for more information.
> ==
> Ing. Simone Giannecchini
> @simogeo
> Founder/Director
>
> GeoSolutions S.A.S.
> Via di Montramito 3/A
> 55054  Massarosa (LU)
> Italy
> phone: +39 0584 962313
> fax: +39 0584 1660272
> mob:   +39  333 8128928
>
> http://www.geo-solutions.it
> http://twitter.com/geosolutions_it
>
> ---
> AVVERTENZE AI SENSI DEL D.Lgs. 196/2003
> Le informazioni contenute in questo messaggio di posta elettronica e/o
> nel/i file/s allegato/i sono da considerarsi strettamente riservate.
> Il loro utilizzo è consentito esclusivamente al destinatario del
> messaggio, per le finalità indicate nel messaggio stesso. Qualora
> riceviate questo messaggio senza esserne il destinatario, Vi preghiamo
> cortesemente di darcene notizia via e-mail e di procedere alla
> distruzione del messaggio stesso, cancellandolo dal Vostro sistema.
> Conservare il messaggio stesso, divulgarlo anche in parte,
> distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità
> diverse, costituisce comportamento contrario ai principi dettati dal
> D.Lgs. 196/2003.
>
> The information in this message and/or attachments, is intended solely
> for the attention and use of the named addressee(s) and may be
> confidential or proprietary in nature or covered by the provisions of
> privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New
> Data Protection Code).Any use not in accord with its purpose, any
> disclosure, reproduction, copying, distribution, or either
> dissemination, either whole or partial, is strictly forbidden except
> previous formal approval of the named addressee(s). If you are not the
> intended recipient, please contact immediately the sender by
> telephone, fax or e-mail and delete the information in this message
> that has been received in error. The sender does not give any warranty
> or accept liability as the content, accuracy or completeness of sent
> messages and accepts no responsibility  for changes made after they
> were sent or for other risks which arise as a result of e-mail
> transmission, viruses, etc.
>
>
> On Thu, Apr 7, 2016 at 7:56 PM, Andrea Aime
>  wrote:
> > On Wed, Apr 6, 2016 at 10:02 PM, Justin Deoliveira 
> > wrote:
> >>
> >> Hi folks,
> >>
> >> I’m working on a project to expose Solr’s heatmap capability through
> >> GeoServer. You can find details about Solr heatmaps here:
> >>
> >> https://cwiki.apache.org/confluence/display/solr/Spatial+Search
> >> (search for “Heatmap Faceting”.
> >>
> >> But the gist of it is this: If you have a spatial field that uses the
> >> recursive prefix tree type (ie. geohash) for indexing then it’s easy
> using
> >> Solr’s facetting infrastructure to generate a heatmap grid. What you get
> >> back from Solr is a 2D array representing the geohash grid, where each
> value
> >> is a count of documents that intersect that grid cell. Applying some
> >> symbolization you can get something that looks like this:
> >>
> >>
> >>
> https://raw.githubusercontent.com/voyagersearch/leaflet-solr-heatmap/master/sample.png
> >>
> >> The above screen shot comes from a leaflet plugin that visualizes the
> >> heatmap directly in the browser. I would like to add a similar looking
> >> visualization for GeoTools/GeoServer.
> >>
> >> My first thought was to expose this as a new type of coverage reader,
> >> since the data is simple grid it falls into the model quite easily. The
> >> major benefit of this approach is that becomes trivial to configure in
> >> GeoServer and easy to style using all of the existing raster symbology
> >> support. I’m interested to hear if others think this is a good approach.
> >
> >
> > I believe it is. You might want have a look at how the sde coverage
> reader
> > was setup, to share the same connection
> > info as a vector solr store (downside, you first have to setup a vector
> > store).
> >
> >>
> >>
> >> If that sounds good my plan was to add this to the existing solr module.
> >> It won’t add any new dependencies aside from a dependency on the
> coverage
> >> module.
> >>
> >> @Andrea: you’re listed as the module maintainer… although if I recall
> >> correctly we agreed to co-maintain the module?
> >
> >
> > We did, just add yourself in the pom :-)
> >
> >>
> >>
> >> I have a prototype working so if that all sounds good I’ll push up a
> >> branch for folks to look at.
> >
> >
> > Great
> >
> >>
> >> One thing I am particular eager to get some feed back on is how to best
> >> achieve the blur affect tha

Re: [Geotools-devel] Solr heatmap as a grid coverage

2016-04-08 Thread Simone Giannecchini 
Ciao Justin,
adding to Andrea's, please, read below...

Regards,
Simone Giannecchini
==
GeoServer Professional Services from the experts!
Visit http://goo.gl/it488V for more information.
==
Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via di Montramito 3/A
55054  Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob:   +39  333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---
AVVERTENZE AI SENSI DEL D.Lgs. 196/2003
Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate.
Il loro utilizzo è consentito esclusivamente al destinatario del
messaggio, per le finalità indicate nel messaggio stesso. Qualora
riceviate questo messaggio senza esserne il destinatario, Vi preghiamo
cortesemente di darcene notizia via e-mail e di procedere alla
distruzione del messaggio stesso, cancellandolo dal Vostro sistema.
Conservare il messaggio stesso, divulgarlo anche in parte,
distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità
diverse, costituisce comportamento contrario ai principi dettati dal
D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely
for the attention and use of the named addressee(s) and may be
confidential or proprietary in nature or covered by the provisions of
privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New
Data Protection Code).Any use not in accord with its purpose, any
disclosure, reproduction, copying, distribution, or either
dissemination, either whole or partial, is strictly forbidden except
previous formal approval of the named addressee(s). If you are not the
intended recipient, please contact immediately the sender by
telephone, fax or e-mail and delete the information in this message
that has been received in error. The sender does not give any warranty
or accept liability as the content, accuracy or completeness of sent
messages and accepts no responsibility  for changes made after they
were sent or for other risks which arise as a result of e-mail
transmission, viruses, etc.


On Thu, Apr 7, 2016 at 7:56 PM, Andrea Aime
 wrote:
> On Wed, Apr 6, 2016 at 10:02 PM, Justin Deoliveira 
> wrote:
>>
>> Hi folks,
>>
>> I’m working on a project to expose Solr’s heatmap capability through
>> GeoServer. You can find details about Solr heatmaps here:
>>
>> https://cwiki.apache.org/confluence/display/solr/Spatial+Search
>> (search for “Heatmap Faceting”.
>>
>> But the gist of it is this: If you have a spatial field that uses the
>> recursive prefix tree type (ie. geohash) for indexing then it’s easy using
>> Solr’s facetting infrastructure to generate a heatmap grid. What you get
>> back from Solr is a 2D array representing the geohash grid, where each value
>> is a count of documents that intersect that grid cell. Applying some
>> symbolization you can get something that looks like this:
>>
>>
>> https://raw.githubusercontent.com/voyagersearch/leaflet-solr-heatmap/master/sample.png
>>
>> The above screen shot comes from a leaflet plugin that visualizes the
>> heatmap directly in the browser. I would like to add a similar looking
>> visualization for GeoTools/GeoServer.
>>
>> My first thought was to expose this as a new type of coverage reader,
>> since the data is simple grid it falls into the model quite easily. The
>> major benefit of this approach is that becomes trivial to configure in
>> GeoServer and easy to style using all of the existing raster symbology
>> support. I’m interested to hear if others think this is a good approach.
>
>
> I believe it is. You might want have a look at how the sde coverage reader
> was setup, to share the same connection
> info as a vector solr store (downside, you first have to setup a vector
> store).
>
>>
>>
>> If that sounds good my plan was to add this to the existing solr module.
>> It won’t add any new dependencies aside from a dependency on the coverage
>> module.
>>
>> @Andrea: you’re listed as the module maintainer… although if I recall
>> correctly we agreed to co-maintain the module?
>
>
> We did, just add yourself in the pom :-)
>
>>
>>
>> I have a prototype working so if that all sounds good I’ll push up a
>> branch for folks to look at.
>
>
> Great
>
>>
>> One thing I am particular eager to get some feed back on is how to best
>> achieve the blur affect that makes heatmaps look “
>> pretty”. At the moment what I have done is baked in a parameter to the
>> coverage format that specifies a blur radius and then when reading the
>> coverage I run it through the Convolve operation to achieve the desired
>> affect. It would be ideal if this could be done at symbolization time. I’m
>> wondering if we currently have any way to define a blur or some similar
>> effect at rendering time with sld? Would a rendering transform work?
>
>
> I don't think we have a blur operation, a rendering

Re: [Geotools-devel] Solr heatmap as a grid coverage

2016-04-07 Thread Jody Garnett 
I like the approach of using a new kind of coverage reader, the limitation
is of course providing a suitable grid size (even though as an actual
coverage your mathematical surface is not limited to a grid).

--
Jody Garnett

On 6 April 2016 at 13:02, Justin Deoliveira  wrote:

> Hi folks,
>
> I’m working on a project to expose Solr’s heatmap capability through
> GeoServer. You can find details about Solr heatmaps here:
>
> https://cwiki.apache.org/confluence/display/solr/Spatial+Search (search
> for “Heatmap Faceting”.
>
> But the gist of it is this: If you have a spatial field that uses the
> recursive prefix tree type (ie. geohash) for indexing then it’s easy using
> Solr’s facetting infrastructure to generate a heatmap grid. What you get
> back from Solr is a 2D array representing the geohash grid, where each
> value is a count of documents that intersect that grid cell. Applying some
> symbolization you can get something that looks like this:
>
>
> https://raw.githubusercontent.com/voyagersearch/leaflet-solr-heatmap/master/sample.png
>
> The above screen shot comes from a leaflet plugin that visualizes the
> heatmap directly in the browser. I would like to add a similar looking
> visualization for GeoTools/GeoServer.
>
> My first thought was to expose this as a new type of coverage reader,
> since the data is simple grid it falls into the model quite easily. The
> major benefit of this approach is that becomes trivial to configure in
> GeoServer and easy to style using all of the existing raster symbology
> support. I’m interested to hear if others think this is a good approach.
>
> If that sounds good my plan was to add this to the existing solr module.
> It won’t add any new dependencies aside from a dependency on the coverage
> module.
>
> @Andrea: you’re listed as the module maintainer… although if I recall
> correctly we agreed to co-maintain the module?
>
> I have a prototype working so if that all sounds good I’ll push up a
> branch for folks to look at. One thing I am particular eager to get some
> feed back on is how to best achieve the blur affect that makes heatmaps
> look “
> pretty”. At the moment what I have done is baked in a parameter to the
> coverage format that specifies a blur radius and then when reading the
> coverage I run it through the Convolve operation to achieve the desired
> affect. It would be ideal if this could be done at symbolization time. I’m
> wondering if we currently have any way to define a blur or some similar
> effect at rendering time with sld? Would a rendering transform work?
>
> Thanks folks.
>
> -Justin
>
>
>
> --
>
> ___
> GeoTools-Devel mailing list
> GeoTools-Devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/geotools-devel
>
>
--
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] Solr heatmap as a grid coverage

2016-04-07 Thread Andrea Aime 
On Wed, Apr 6, 2016 at 10:02 PM, Justin Deoliveira 
wrote:

> Hi folks,
>
> I’m working on a project to expose Solr’s heatmap capability through
> GeoServer. You can find details about Solr heatmaps here:
>
> https://cwiki.apache.org/confluence/display/solr/Spatial+Search (search
> for “Heatmap Faceting”.
>
> But the gist of it is this: If you have a spatial field that uses the
> recursive prefix tree type (ie. geohash) for indexing then it’s easy using
> Solr’s facetting infrastructure to generate a heatmap grid. What you get
> back from Solr is a 2D array representing the geohash grid, where each
> value is a count of documents that intersect that grid cell. Applying some
> symbolization you can get something that looks like this:
>
>
> https://raw.githubusercontent.com/voyagersearch/leaflet-solr-heatmap/master/sample.png
>
> The above screen shot comes from a leaflet plugin that visualizes the
> heatmap directly in the browser. I would like to add a similar looking
> visualization for GeoTools/GeoServer.
>
> My first thought was to expose this as a new type of coverage reader,
> since the data is simple grid it falls into the model quite easily. The
> major benefit of this approach is that becomes trivial to configure in
> GeoServer and easy to style using all of the existing raster symbology
> support. I’m interested to hear if others think this is a good approach.
>

I believe it is. You might want have a look at how the sde coverage reader
was setup, to share the same connection
info as a vector solr store (downside, you first have to setup a vector
store).


>
> If that sounds good my plan was to add this to the existing solr module.
> It won’t add any new dependencies aside from a dependency on the coverage
> module.
>
> @Andrea: you’re listed as the module maintainer… although if I recall
> correctly we agreed to co-maintain the module?
>

We did, just add yourself in the pom :-)


>
> I have a prototype working so if that all sounds good I’ll push up a
> branch for folks to look at.
>

Great


> One thing I am particular eager to get some feed back on is how to best
> achieve the blur affect that makes heatmaps look “
> pretty”. At the moment what I have done is baked in a parameter to the
> coverage format that specifies a blur radius and then when reading the
> coverage I run it through the Convolve operation to achieve the desired
> affect. It would be ideal if this could be done at symbolization time. I’m
> wondering if we currently have any way to define a blur or some similar
> effect at rendering time with sld? Would a rendering transform work?
>

I don't think we have a blur operation, a rendering transform would likely
do the trick with the smallest effort.
In an ideal world, it would be nice to have a FTS level vendor parameter to
perform blurring just like we do color composition
but I guess that might be more complex to implement.

Cheers
Andrea

-- 
==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.
==

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054  Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39  339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.



The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility  for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

--

Re: [Geotools-devel] Solr heatmap as a grid coverage

2016-04-06 Thread Jim Hughes 

Justin,

Ayup, the HeatMap Process already have a notion that you have to mess 
with the query.


In a method like invertQuery:
https://github.com/locationtech/geomesa/blob/master/geomesa-process/src/main/scala/org/locationtech/geomesa/process/DensityProcess.scala#L98-L125

one can munge the query as necessary.

In the process bits:
https://github.com/locationtech/geomesa/blob/master/geomesa-process/src/main/scala/org/locationtech/geomesa/process/DensityProcess.scala#L59-L74

one can then decode whatever different-non-features mojo came back from 
the request.  In the GeoMesa example, we are return a map of (x, y) -> 
weight.  That bit of code handles the yoga between the aggregated result 
and the HeatMap process.


When we've used heatmaps as rasters, our general approach has been to 
'import' the output of the DensityProcess as a new layer.


Hope that helps a bit; cheers!

Jim

On 04/06/2016 04:27 PM, Justin Deoliveira wrote:
Thanks for the info Jim. Indeed that was my first inclination as well 
but once I thought about it I started to struggle with how I could fit 
it into the vector pipeline cleanly. HeatmapProcess didn’t seem to 
really apply since the aggregation needs to be done on the Solr side 
and not by the rendering transform. Is that how the GeoMesa heatmaps 
work?


To make that approach I would need to hack the request to include all 
the required facet parameters. I couldn’t really think of a clean way 
to do that although thinking about it again I could probably just 
throw a bunch of hints into the query that would trigger the Solr 
datastore to engage all of the facetting stuff… I think i’ll explore 
that approach as well, thanks Jim!


My other thought was that the heatmap coverage could be useful for 
reasons other than purely visualizing areas of density. Like for 
instance using it as a mask against another coverage to do some 
processing… I don’t have a concrete use case there but thought it had 
potential.



On Wed, Apr 6, 2016 at 2:14 PM Jim Hughes > wrote:


Hi Justin,

Since it is somewhat similar, I wanted to share how GeoMesa
creates heatmaps using GeoServer.  We have a small WPS which riffs
on the HeatMapProcess (1).  That process is called via an SLD.  By
doing that, we can have a regular vector layer, but then generate
heatmaps for it without fiddling with a second coverage, etc.

Potentially, an approach like that could line up more of the
rendering pipeline to achieve blur, etc.

Cheers,

Jim

1.

https://github.com/geotools/geotools/blob/master/modules/unsupported/process-feature/src/main/java/org/geotools/process/vector/HeatmapProcess.java


On 04/06/2016 04:02 PM, Justin Deoliveira wrote:

Hi folks,

I’m working on a project to expose Solr’s heatmap capability
through GeoServer. You can find details about Solr heatmaps here:

https://cwiki.apache.org/confluence/display/solr/Spatial+Search (search
for “Heatmap Faceting”.

But the gist of it is this: If you have a spatial field that uses
the recursive prefix tree type (ie. geohash) for indexing then
it’s easy using Solr’s facetting infrastructure to generate a
heatmap grid. What you get back from Solr is a 2D array
representing the geohash grid, where each value is a count of
documents that intersect that grid cell. Applying some
symbolization you can get something that looks like this:


https://raw.githubusercontent.com/voyagersearch/leaflet-solr-heatmap/master/sample.png

The above screen shot comes from a leaflet plugin that visualizes
the heatmap directly in the browser. I would like to add a
similar looking visualization for GeoTools/GeoServer.

My first thought was to expose this as a new type of coverage
reader, since the data is simple grid it falls into the model
quite easily. The major benefit of this approach is that becomes
trivial to configure in GeoServer and easy to style using all of
the existing raster symbology support. I’m interested to hear if
others think this is a good approach.

If that sounds good my plan was to add this to the existing solr
module. It won’t add any new dependencies aside from a dependency
on the coverage module.

@Andrea: you’re listed as the module maintainer… although if I
recall correctly we agreed to co-maintain the module?

I have a prototype working so if that all sounds good I’ll push
up a branch for folks to look at. One thing I am particular eager
to get some feed back on is how to best achieve the blur affect
that makes heatmaps look “
pretty”. At the moment what I have done is baked in a parameter
to the coverage format that specifies a blur radius and then when
reading the coverage I run it through the Convolve operation to
achieve the desired affect. It would be ideal if this could be
done at symbolization time. I’m wondering if we currently have

Re: [Geotools-devel] Solr heatmap as a grid coverage

2016-04-06 Thread Justin Deoliveira 
Thanks for the info Jim. Indeed that was my first inclination as well but
once I thought about it I started to struggle with how I could fit it into
the vector pipeline cleanly. HeatmapProcess didn’t seem to really apply
since the aggregation needs to be done on the Solr side and not by the
rendering transform. Is that how the GeoMesa heatmaps work?

To make that approach I would need to hack the request to include all the
required facet parameters. I couldn’t really think of a clean way to do
that although thinking about it again I could probably just throw a bunch
of hints into the query that would trigger the Solr datastore to engage all
of the facetting stuff… I think i’ll explore that approach as well, thanks
Jim!

My other thought was that the heatmap coverage could be useful for reasons
other than purely visualizing areas of density. Like for instance using it
as a mask against another coverage to do some processing… I don’t have a
concrete use case there but thought it had potential.


On Wed, Apr 6, 2016 at 2:14 PM Jim Hughes  wrote:

> Hi Justin,
>
> Since it is somewhat similar, I wanted to share how GeoMesa creates
> heatmaps using GeoServer.  We have a small WPS which riffs on the
> HeatMapProcess (1).  That process is called via an SLD.  By doing that, we
> can have a regular vector layer, but then generate heatmaps for it without
> fiddling with a second coverage, etc.
>
> Potentially, an approach like that could line up more of the rendering
> pipeline to achieve blur, etc.
>
> Cheers,
>
> Jim
>
> 1.
> https://github.com/geotools/geotools/blob/master/modules/unsupported/process-feature/src/main/java/org/geotools/process/vector/HeatmapProcess.java
>
>
> On 04/06/2016 04:02 PM, Justin Deoliveira wrote:
>
> Hi folks,
>
> I’m working on a project to expose Solr’s heatmap capability through
> GeoServer. You can find details about Solr heatmaps here:
>
> https://cwiki.apache.org/confluence/display/solr/Spatial+Search (search
> for “Heatmap Faceting”.
>
> But the gist of it is this: If you have a spatial field that uses the
> recursive prefix tree type (ie. geohash) for indexing then it’s easy using
> Solr’s facetting infrastructure to generate a heatmap grid. What you get
> back from Solr is a 2D array representing the geohash grid, where each
> value is a count of documents that intersect that grid cell. Applying some
> symbolization you can get something that looks like this:
>
>
> https://raw.githubusercontent.com/voyagersearch/leaflet-solr-heatmap/master/sample.png
>
> The above screen shot comes from a leaflet plugin that visualizes the
> heatmap directly in the browser. I would like to add a similar looking
> visualization for GeoTools/GeoServer.
>
> My first thought was to expose this as a new type of coverage reader,
> since the data is simple grid it falls into the model quite easily. The
> major benefit of this approach is that becomes trivial to configure in
> GeoServer and easy to style using all of the existing raster symbology
> support. I’m interested to hear if others think this is a good approach.
>
> If that sounds good my plan was to add this to the existing solr module.
> It won’t add any new dependencies aside from a dependency on the coverage
> module.
>
> @Andrea: you’re listed as the module maintainer… although if I recall
> correctly we agreed to co-maintain the module?
>
> I have a prototype working so if that all sounds good I’ll push up a
> branch for folks to look at. One thing I am particular eager to get some
> feed back on is how to best achieve the blur affect that makes heatmaps
> look “
> pretty”. At the moment what I have done is baked in a parameter to the
> coverage format that specifies a blur radius and then when reading the
> coverage I run it through the Convolve operation to achieve the desired
> affect. It would be ideal if this could be done at symbolization time. I’m
> wondering if we currently have any way to define a blur or some similar
> effect at rendering time with sld? Would a rendering transform work?
>
> Thanks folks.
>
> -Justin
>
>
>
> --
>
>
>
> ___
> GeoTools-Devel mailing 
> listGeoTools-Devel@lists.sourceforge.nethttps://lists.sourceforge.net/lists/listinfo/geotools-devel
>
>
>
> --
> ___
> GeoTools-Devel mailing list
> GeoTools-Devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/geotools-devel
>
--
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] Solr heatmap as a grid coverage

2016-04-06 Thread Jim Hughes 

Hi Justin,

Since it is somewhat similar, I wanted to share how GeoMesa creates 
heatmaps using GeoServer.  We have a small WPS which riffs on the 
HeatMapProcess (1).  That process is called via an SLD.  By doing that, 
we can have a regular vector layer, but then generate heatmaps for it 
without fiddling with a second coverage, etc.


Potentially, an approach like that could line up more of the rendering 
pipeline to achieve blur, etc.


Cheers,

Jim

1. 
https://github.com/geotools/geotools/blob/master/modules/unsupported/process-feature/src/main/java/org/geotools/process/vector/HeatmapProcess.java


On 04/06/2016 04:02 PM, Justin Deoliveira wrote:

Hi folks,

I’m working on a project to expose Solr’s heatmap capability through 
GeoServer. You can find details about Solr heatmaps here:


https://cwiki.apache.org/confluence/display/solr/Spatial+Search (search for 
“Heatmap Faceting”.


But the gist of it is this: If you have a spatial field that uses the 
recursive prefix tree type (ie. geohash) for indexing then it’s easy 
using Solr’s facetting infrastructure to generate a heatmap grid. What 
you get back from Solr is a 2D array representing the geohash grid, 
where each value is a count of documents that intersect that grid 
cell. Applying some symbolization you can get something that looks 
like this:


https://raw.githubusercontent.com/voyagersearch/leaflet-solr-heatmap/master/sample.png

The above screen shot comes from a leaflet plugin that visualizes the 
heatmap directly in the browser. I would like to add a similar looking 
visualization for GeoTools/GeoServer.


My first thought was to expose this as a new type of coverage reader, 
since the data is simple grid it falls into the model quite easily. 
The major benefit of this approach is that becomes trivial to 
configure in GeoServer and easy to style using all of the existing 
raster symbology support. I’m interested to hear if others think this 
is a good approach.


If that sounds good my plan was to add this to the existing solr 
module. It won’t add any new dependencies aside from a dependency on 
the coverage module.


@Andrea: you’re listed as the module maintainer… although if I recall 
correctly we agreed to co-maintain the module?


I have a prototype working so if that all sounds good I’ll push up a 
branch for folks to look at. One thing I am particular eager to get 
some feed back on is how to best achieve the blur affect that makes 
heatmaps look “
pretty”. At the moment what I have done is baked in a parameter to the 
coverage format that specifies a blur radius and then when reading the 
coverage I run it through the Convolve operation to achieve the 
desired affect. It would be ideal if this could be done at 
symbolization time. I’m wondering if we currently have any way to 
define a blur or some similar effect at rendering time with sld? Would 
a rendering transform work?


Thanks folks.

-Justin



--


___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel


--
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

[Geotools-devel] Solr heatmap as a grid coverage

2016-04-06 Thread Justin Deoliveira 
Hi folks,

I’m working on a project to expose Solr’s heatmap capability through
GeoServer. You can find details about Solr heatmaps here:

https://cwiki.apache.org/confluence/display/solr/Spatial+Search (search
for “Heatmap Faceting”.

But the gist of it is this: If you have a spatial field that uses the
recursive prefix tree type (ie. geohash) for indexing then it’s easy using
Solr’s facetting infrastructure to generate a heatmap grid. What you get
back from Solr is a 2D array representing the geohash grid, where each
value is a count of documents that intersect that grid cell. Applying some
symbolization you can get something that looks like this:


https://raw.githubusercontent.com/voyagersearch/leaflet-solr-heatmap/master/sample.png

The above screen shot comes from a leaflet plugin that visualizes the
heatmap directly in the browser. I would like to add a similar looking
visualization for GeoTools/GeoServer.

My first thought was to expose this as a new type of coverage reader, since
the data is simple grid it falls into the model quite easily. The major
benefit of this approach is that becomes trivial to configure in GeoServer
and easy to style using all of the existing raster symbology support. I’m
interested to hear if others think this is a good approach.

If that sounds good my plan was to add this to the existing solr module. It
won’t add any new dependencies aside from a dependency on the coverage
module.

@Andrea: you’re listed as the module maintainer… although if I recall
correctly we agreed to co-maintain the module?

I have a prototype working so if that all sounds good I’ll push up a branch
for folks to look at. One thing I am particular eager to get some feed back
on is how to best achieve the blur affect that makes heatmaps look “
pretty”. At the moment what I have done is baked in a parameter to the
coverage format that specifies a blur radius and then when reading the
coverage I run it through the Convolve operation to achieve the desired
affect. It would be ideal if this could be done at symbolization time. I’m
wondering if we currently have any way to define a blur or some similar
effect at rendering time with sld? Would a rendering transform work?

Thanks folks.

-Justin
--
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel