Re: [Geotools-devel] Copyright header policy

2016-04-22 Thread Torben Barsballe 
My thoughts on the @author tag:
I feel like the git commit log gives a clearer and more usefull picture of
this sort of information - You can see the initial author, as well as the
author of any major changes, and can usually get a good idea of who is
currently doing the most work in that area of the codebase (as compared to
someone who authored a class 5 years ago and hasn't touched it since).
The one exception is when stuff gets transitioned between repositories or
other similar bulk import cases.
Conversely, the @author tag is much less maintainable, less enforcable, and
prone to getting out-of-date.

Torben

On Fri, Apr 22, 2016 at 2:39 PM, Ben Caradoc-Davies 
wrote:

> I used to share Andrea's view but, as my confidence grew, I became more
> comfortable seeing @author tags as historical information not an
> assertion of ownership.
>
> On 23/04/16 09:24, Jody Garnett wrote:
> > Andrea had a very reasoned argument for not using the @author tag and
> > started discouraging its use.
> > It amounted to team ownership of the codebase empowering everyone to work
> > on a class, rather than slowing down to seek permission (or even worse
> wait
> > for someone else) to fix a problem.
>
> --
> Ben Caradoc-Davies 
> Director
> Transient Software Limited 
> New Zealand
>
>
> --
> Find and fix application performance issues faster with Applications
> Manager
> Applications Manager provides deep performance insights into multiple
> tiers of
> your business applications. It resolves application problems quickly and
> reduces your MTTR. Get your free trial!
> https://ad.doubleclick.net/ddm/clk/302982198;130105516;z
> ___
> GeoTools-Devel mailing list
> GeoTools-Devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/geotools-devel
>
--
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] wfs-ng vs wfs datastore fight!

2016-04-22 Thread Niels Charlier 

Andre,

The reason I closed it as "not a bug" was because I knew it was done by 
intentional design (not by me, but by the initial developer of wfs-ng) 
for a particular purpose and therefore I did not consider it a bug. You 
do make good points to criticise this choice, which does affect 
backwards compatibility. I do not personally oppose your alternative 
suggestion.


Regards
Niels

On 04/22/2016 07:48 PM, Andrea Aime wrote:
On Thu, Apr 21, 2016 at 7:52 PM, Jody Garnett > wrote:



> Hum... wfs-ng was supposed to be a drop-in replacement,
not having a upgrade
> path seems to be a serious issue to me.
> Where is it documented?

Quick search got us this:
https://osgeo-org.atlassian.net/browse/GEOT-4883
and we googled some more messages on it. 


A backwards incompable change blocking upgrades presented as a
GeoServer requirement and resolved as "not a bug"...


Reading this issue, and I agree it is not a bug - the new
datastore is behaving as designed in order to preserve valid names
for GML generation.


Jody I believe we have a problem here, in terms of responsibilities. 
Is it really the job of the stores to decide what is valid, and what 
is not?
Not all usage is WFS cascading, stores are used for a number of other 
cases, which might have different naming limitations.


Also from a consistency point of view, did you know that you can 
create a typename like "a:b" using a postgis store?


> create table "a:b" (id serial, geom geometry(POLYGON, 4326));
CREATE TABLE

And then the datastore is happily returning a:b as the typename,
And of course we also have shapefiles, on linux a file name like 
a:b.shp is also completely valid.


With you and Niels claiming that is not a bug, we have a difficult 
situation, as nobody else can just jump in and make
the upgrade path easier, since according to your evaluation, we'd be 
introducing a bug... (and from that same point of view,

then jdbc store and shapefile store are buggy too...).

Unless of course your position is a bit more relaxed, as in, you don't 
consider that a bug, but the behavior returning the native

name is valid too.


For this specific case the old datastore would of required a
similar fix (to prevent type names being of the form
"local_namespace:remote_namespace:typename").

As far as I know GeoServer was already handling the ":" in the
name, and the admin is able to rename layers as they see fit.
Yep, there is code stripping the prefix:

https://github.com/geoserver/geoserver/blob/master/src/main/src/main/java/org/geoserver/catalog/CatalogBuilder.java#L1212


GeoServer may have work arounds, but I do not mind if the new
DataStore is taking more care to produce consistent type names.


See Mark's situation though. I'm also aware of other situations like 
this one that will be broken by an upgrade (systems configured

to work against well known type names in remote servers).

Mind, I checked, GeoServer is not affected, because internally we have 
this little thing that hides from GeoServer the columns and

guess what, replaces them with underscores. See here:

https://github.com/geoserver/geoserver/blob/master/src/main/src/main/java/org/vfny/geoserver/util/DataStoreUtils.java#L74

and then the retyper does this:

https://github.com/geoserver/geoserver/blob/master/src/main/src/main/java/org/geoserver/feature/retype/RetypingDataStore.java#L252

So luckily for GeoServer even if the store did change its behavior, 
nothing changed config wise (we end up seeing a_b as the native
type name regardless of whether the store returns a:b or a_b because 
of the above wrapper).


Given this, I'd still make it possible for GeoTools users to upgrade 
from wfs to wfs-ng without this much hassle by exposing the

actual native type name.
If the worry is about valid type names for GML, I'm more than happy to 
contribute a little utility wrapping store based on the GeoServer one
that does the same thing as RetypingDataStore, but centralized in one 
place, and optional, only for those that actually need to do GML encoding.

A DataUtilities.makeGmlCompatible(DataStore) -> DataStore of sorts.

Cheers
Andrea


--
==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.
==

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39  339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o 
nel/i file/s allegato/i sono da considerarsi strettamente riservate. 
Il loro utilizzo è consentito esclusivamente al destinatario del 
messaggio, per le finalità indicate nel messaggio

[Geotools-devel] [JIRA] (GEOT-5406) Ignore NetCDF grid_mapping_name that is present but unsupported

2016-04-22 Thread Ben Caradoc-Davies [Administrator] (JIRA) 
Title: Message Title


 
 
 
 

 
 
 

 
   
 Ben Caradoc-Davies [Administrator] created an issue  
 

  
 
 
 
 

 
 
  
 
 
 
 

 
 GeoTools /  GEOT-5406  
 
 
  Ignore NetCDF grid_mapping_name that is present but unsupported   
 

  
 
 
 
 

 
Issue Type: 
  Improvement  
 
 
Assignee: 
 Ben Caradoc-Davies [Administrator]  
 
 
Components: 
 coverage-multidim  
 
 
Created: 
 23/Apr/16 12:01 AM  
 
 
Priority: 
  Medium  
 
 
Reporter: 
 Ben Caradoc-Davies [Administrator]  
 

  
 
 
 
 

 
 The current behaviour of NetCDFProjection when encountering an unsupported NetCDF grid_mapping_name is to throw a NullPointerException. This fix changes the behaviour to instead log a message and return null, consistent with the behaviour earlier in the method for a missing grid_mapping_name.  
 

  
 
 
 
 

 
 
 

 
 
 Add Comment  
 

  
 

  
 
 
 
 

Re: [Geotools-devel] Copyright header policy

2016-04-22 Thread Ben Caradoc-Davies 
I used to share Andrea's view but, as my confidence grew, I became more 
comfortable seeing @author tags as historical information not an 
assertion of ownership.

On 23/04/16 09:24, Jody Garnett wrote:
> Andrea had a very reasoned argument for not using the @author tag and
> started discouraging its use.
> It amounted to team ownership of the codebase empowering everyone to work
> on a class, rather than slowing down to seek permission (or even worse wait
> for someone else) to fix a problem.

-- 
Ben Caradoc-Davies 
Director
Transient Software Limited 
New Zealand

--
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] Copyright header policy

2016-04-22 Thread Jody Garnett 
Andrea had a very reasoned argument for not using the @author tag and
started discouraging its use.

It amounted to team ownership of the codebase empowering everyone to work
on a class, rather than slowing down to seek permission (or even worse wait
for someone else) to fix a problem.

--
Jody Garnett

On 22 April 2016 at 14:21, Ben Caradoc-Davies  wrote:

> In fact, I think we should *require* rather than just encourage the use of
> @author.
>
> Kind regards,
> Ben.
>
>
> On 23/04/16 09:14, Ben Caradoc-Davies wrote:
>
>> I like the Javadoc @author tag. We should encourage its use on all new
>> classes, and additional @author tags for any substantial contribution to
>> an existing file. @author tags are useful not only for provenance but
>> for seeking help; successive changes can make @author tags easier to use
>> than git blame. @author tags also allow us to respect the moral right of
>> attribution.
>>
>> Kind regards,
>> Ben.
>>
>> On 23/04/16 08:42, Jody Garnett wrote:
>>
>>> I like having headers that capture the initial author - since that has
>>> been
>>> very helpful in sorting out any questions during our incubation and
>>> subsequent external code audits.
>>>
>>
>>
> --
> Ben Caradoc-Davies 
> Director
> Transient Software Limited 
> New Zealand
>
--
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] Copyright header policy

2016-04-22 Thread Ben Caradoc-Davies 
+1. This is a great practice, and required by some licences. Dates in 
these headers cannot be removed or changed.

On 23/04/16 09:19, Jody Garnett wrote:
> The other clarification I have gotten from external review is to leave
> headers intact when copying code into our codebase (we only have a small
> number of examples of this).

-- 
Ben Caradoc-Davies 
Director
Transient Software Limited 
New Zealand

--
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] Copyright header policy

2016-04-22 Thread Ben Caradoc-Davies 
In fact, I think we should *require* rather than just encourage the use 
of @author.

Kind regards,
Ben.

On 23/04/16 09:14, Ben Caradoc-Davies wrote:
> I like the Javadoc @author tag. We should encourage its use on all new
> classes, and additional @author tags for any substantial contribution to
> an existing file. @author tags are useful not only for provenance but
> for seeking help; successive changes can make @author tags easier to use
> than git blame. @author tags also allow us to respect the moral right of
> attribution.
>
> Kind regards,
> Ben.
>
> On 23/04/16 08:42, Jody Garnett wrote:
>> I like having headers that capture the initial author - since that has been
>> very helpful in sorting out any questions during our incubation and
>> subsequent external code audits.
>

-- 
Ben Caradoc-Davies 
Director
Transient Software Limited 
New Zealand

--
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] Copyright header policy

2016-04-22 Thread Jody Garnett 
The other clarification I have gotten from external review is to leave
headers intact when copying code into our codebase (we only have a small
number of examples of this).

--
Jody Garnett

On 22 April 2016 at 14:14, Ben Caradoc-Davies  wrote:

> I like the Javadoc @author tag. We should encourage its use on all new
> classes, and additional @author tags for any substantial contribution to an
> existing file. @author tags are useful not only for provenance but for
> seeking help; successive changes can make @author tags easier to use than
> git blame. @author tags also allow us to respect the moral right of
> attribution.
>
> Kind regards,
> Ben.
>
> On 23/04/16 08:42, Jody Garnett wrote:
>
>> I like having headers that capture the initial author - since that has
>> been
>> very helpful in sorting out any questions during our incubation and
>> subsequent external code audits.
>>
>
> --
> Ben Caradoc-Davies 
> Director
> Transient Software Limited 
> New Zealand
>
--
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] Copyright header policy

2016-04-22 Thread Ben Caradoc-Davies 
I like the Javadoc @author tag. We should encourage its use on all new 
classes, and additional @author tags for any substantial contribution to 
an existing file. @author tags are useful not only for provenance but 
for seeking help; successive changes can make @author tags easier to use 
than git blame. @author tags also allow us to respect the moral right of 
attribution.

Kind regards,
Ben.

On 23/04/16 08:42, Jody Garnett wrote:
> I like having headers that capture the initial author - since that has been
> very helpful in sorting out any questions during our incubation and
> subsequent external code audits.

-- 
Ben Caradoc-Davies 
Director
Transient Software Limited 
New Zealand

--
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] Copyright header policy

2016-04-22 Thread Jody Garnett 
OSGeo is governed by projects such as ours setting a precedent.

I like having headers that capture the initial author - since that has been
very helpful in sorting out any questions during our incubation and
subsequent external code audits.

I really cannot see song away with headers as they are being used.
On Fri, Apr 22, 2016 at 1:39 PM Ben Caradoc-Davies  wrote:

> My understanding is that, under the Berne Convention, there is no
> requirement at all for any copyright notice, "Copyright", "(C)", "All
> Rights Reserved", or dates. The WTO requires compliance to TRIPS rules
> which thus apply these rules to practically every country in the world.
> Even the United States adopted the Berne Convention in 1989:
> https://en.wikipedia.org/wiki/Berne_Convention
>
> In my view, and I am not a lawyer, copyright headers are copyright
> theatre without any legal effect, and serve only to permit authors to
> appear serious and knowledgeable about copyright to people who know
> little about it. They are based in obsolete pre-Berne practices.
> Remember that the original GPL boilerplate was written just as Berne
> came to the US and was not widely understood.
>
> I would like to see headers that are more useful to readers and more
> maintainable. I would love to see a static header like this:
>
> /*
>   * GeoTools, The Open Source Java GIS Toolkit 
>   */
>
> All the copyright ownership and licensing information is here:
> http://geotools.org/about.html
>
> And we are done.
>
> So, what should we do? I think we are governed by OSGeo policy. Does
> OSGeo have a policy on headers? Has OSGeo obtained a legal opinion on
> this matter?
>
> Kind regards,
> Ben.
>
> On 23/04/16 01:01, Justin Deoliveira wrote:
> > Hi folks,
> >
> > Something I’ve been meaning to ask about is a clarification of why we
> have
> > to update copyright header dates to the current year for every file
> change.
> > As opposed to just having a static copyright header that signifies a
> > “copyright event” (like the change over to osgeo) and then stands for
> that
> > time forward.
> >
> > The reason I ask is because I think this policy has some negative things
> in
> > my opinion. For one it’s a pain for both contributors and reviewers to
> > enforce. I myself never remember as you probably well know and I would
> say
> > I am a pretty regular contributor. It also adds noise to patches and pull
> > requests, which also makes things harder to review than necessary.
> >
> >
> > Anyways, just curious as to the rationale behind this decision. My
> > apologies if this has been covered in previous email.
> >
> > -Justin
> >
> >
> >
> >
> --
> > Find and fix application performance issues faster with Applications
> Manager
> > Applications Manager provides deep performance insights into multiple
> tiers of
> > your business applications. It resolves application problems quickly and
> > reduces your MTTR. Get your free trial!
> > https://ad.doubleclick.net/ddm/clk/302982198;130105516;z
> >
> >
> >
> > ___
> > GeoTools-Devel mailing list
> > GeoTools-Devel@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/geotools-devel
> >
>
> --
> Ben Caradoc-Davies 
> Director
> Transient Software Limited 
> New Zealand
>
>
> --
> Find and fix application performance issues faster with Applications
> Manager
> Applications Manager provides deep performance insights into multiple
> tiers of
> your business applications. It resolves application problems quickly and
> reduces your MTTR. Get your free trial!
> https://ad.doubleclick.net/ddm/clk/302982198;130105516;z
> ___
> GeoTools-Devel mailing list
> GeoTools-Devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/geotools-devel
>
-- 
--
Jody Garnett
--
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] Copyright header policy

2016-04-22 Thread Ben Caradoc-Davies 
My understanding is that, under the Berne Convention, there is no 
requirement at all for any copyright notice, "Copyright", "(C)", "All 
Rights Reserved", or dates. The WTO requires compliance to TRIPS rules 
which thus apply these rules to practically every country in the world. 
Even the United States adopted the Berne Convention in 1989:
https://en.wikipedia.org/wiki/Berne_Convention

In my view, and I am not a lawyer, copyright headers are copyright 
theatre without any legal effect, and serve only to permit authors to 
appear serious and knowledgeable about copyright to people who know 
little about it. They are based in obsolete pre-Berne practices. 
Remember that the original GPL boilerplate was written just as Berne 
came to the US and was not widely understood.

I would like to see headers that are more useful to readers and more 
maintainable. I would love to see a static header like this:

/*
  * GeoTools, The Open Source Java GIS Toolkit 
  */

All the copyright ownership and licensing information is here: 
http://geotools.org/about.html

And we are done.

So, what should we do? I think we are governed by OSGeo policy. Does 
OSGeo have a policy on headers? Has OSGeo obtained a legal opinion on 
this matter?

Kind regards,
Ben.

On 23/04/16 01:01, Justin Deoliveira wrote:
> Hi folks,
>
> Something I’ve been meaning to ask about is a clarification of why we have
> to update copyright header dates to the current year for every file change.
> As opposed to just having a static copyright header that signifies a
> “copyright event” (like the change over to osgeo) and then stands for that
> time forward.
>
> The reason I ask is because I think this policy has some negative things in
> my opinion. For one it’s a pain for both contributors and reviewers to
> enforce. I myself never remember as you probably well know and I would say
> I am a pretty regular contributor. It also adds noise to patches and pull
> requests, which also makes things harder to review than necessary.
>
>
> Anyways, just curious as to the rationale behind this decision. My
> apologies if this has been covered in previous email.
>
> -Justin
>
>
>
> --
> Find and fix application performance issues faster with Applications Manager
> Applications Manager provides deep performance insights into multiple tiers of
> your business applications. It resolves application problems quickly and
> reduces your MTTR. Get your free trial!
> https://ad.doubleclick.net/ddm/clk/302982198;130105516;z
>
>
>
> ___
> GeoTools-Devel mailing list
> GeoTools-Devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/geotools-devel
>

-- 
Ben Caradoc-Davies 
Director
Transient Software Limited 
New Zealand

--
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] [Geoserver-devel] Reading GeoTiffs from HDFS

2016-04-22 Thread Jim Hughes 
Hi Chris,

Nice!  That's a fun find.

Generally, I do like the idea of using Map/Reduce or Spark to 
pre-generate tiles or an image pyramid.  We've kicked around the idea of 
GWC + M/R a few times in passing.  If one has Hadoop infrastructure 
hanging around, it might make sense to use GeoTrellis, SpatialHadoop 
(GeoJini), etc. for some of that processing.

Either way, being able to read the odd raster file straight from hdfs:// 
or s3:// and have it cached in memory seems like an amusing/useful 
project.  I'm hopeful we can nail down the details.

Cheers,

Jim

On 04/22/2016 02:27 PM, Chris Snider wrote:
> I did find this reference (helpful ?):
> https://github.com/openreserach/bin2seq/blob/master/src/main/java/com/openresearchinc/hadoop/sequencefile/GeoTiff.java
>
> " /@formatter:off
> /**
>   *
>   * A program to demo retrive attributes from Geotiff images as Hadoop 
> SequenceFile stored on hdfs:// or s3://
>   *
>   *
>   * @author heq
>   */
> // @formatter:on"
>
> Chris Snider
> Senior Software Engineer
> Intelligent Software Solutions, Inc.
>
>
>
> -Original Message-
> From: Chris Snider [mailto:chris.sni...@issinc.com]
> Sent: Friday, April 22, 2016 12:11 PM
> To: Jim Hughes ; Simone Giannecchini 
> 
> Cc: geoserver-de...@lists.sourceforge.net; GeoTools Developers list 
> 
> Subject: Re: [Geoserver-devel] [Geotools-devel] Reading GeoTiffs from HDFS
>
> Hi,
>
> I don't know that much about HDFS, but is there something that can be setup 
> like a map/reduce function directly in the HDFS servers that can do some of 
> the restriction of byte level data returned? Yarn/Sparql, some other acronym? 
>  I assume it would have to be administrator responsibility to add said 
> process to the server stack if it is even possible.
>
> Chris Snider
> Senior Software Engineer
> Intelligent Software Solutions, Inc.
>
>
> -Original Message-
> From: Jim Hughes [mailto:jn...@ccri.com]
> Sent: Friday, April 22, 2016 12:05 PM
> To: Simone Giannecchini 
> Cc: geoserver-de...@lists.sourceforge.net; GeoTools Developers list 
> 
> Subject: Re: [Geoserver-devel] [Geotools-devel] Reading GeoTiffs from HDFS
>
> Hi Simone,
>
> Thanks for the feedback!
>
> As quick response, for #1, I agree that using mosaicing / an image
> pyramid would be a great option.  I was mainly working at the prototype
> phase, and I wanted to have a discussion on the mailing lists
> (especially since changes are required in ImageIO-Ext or GeoTools and
> GeoServer.)
>
> For #2, I do like the idea of having a cahce in the ImageInputStream.
>   From that suggestion, I take it that you'd be willing to entertain
> changes to the current ImageInputStreams and the additional of some way
> to cache data.
>
> In terms of caching, do you have any suggestions?  Also, I'd be
> interested in any advice for how we can configure that cache and make
> those options available to a GeoServer admin appropriately.
>
> Further, at a high-level, should the goal for this work be a community
> module?
>
> Cheers,
>
> Jim
>
> On 04/22/2016 01:49 PM, Simone Giannecchini wrote:
>> Dear Jim,
>> quick feedback.
>>
>> First of all congratulation on making this work. As I suspected the
>> bottleneck is getting the data out of HDFS.
>> I can think about two things (which we are not mutually exclusive):
>>
>> -1- Maybe complex, put smaller bits into HFDS and use the mosaic to
>> serve or even develop a light(er)weight layer that can pull the
>> granules.
>>
>> This would help with WMS requests over large files as you'll end up
>> use smaller chunks to satisfy them most of the time
>>
>> -2- We could build a more complex ImageInputStream that:
>>
>> - has an internal cache (file and or memory) that does not get thrown
>> away upon each request but tends to live longer for each single file
>> in HDF
>> - we would have different streams reuse the same cache. Multiple
>> requests might read data from the cache concurrently but when data is
>> not there, we would block the thread for the request, go back to HFDS,
>> pull the data, write to the cache and so on
>>
>> We could put together 1 and 2 to make things faster.
>>
>> Hope this helps, anyway, I am in favour of exploring this in order to
>> allow the GeoServer stack to support data from HDFS.
>>
>> Regards,
>> Simone Giannecchini
>> ==
>> GeoServer Professional Services from the experts!
>> Visit http://goo.gl/it488V for more information.
>> ==
>> Ing. Simone Giannecchini
>> @simogeo
>> Founder/Director
>>
>> GeoSolutions S.A.S.
>> Via di Montramito 3/A
>> 55054  Massarosa (LU)
>> Italy
>> phone: +39 0584 962313
>> fax: +39 0584 1660272
>> mob:   +39  333 8128928
>>
>> http://www.geo-solutions.it
>> http://twitter.com/geosolutions_it
>>
>> ---
>> AVVERTENZE AI SENSI DEL D.Lgs. 196/2003
>> Le

Re: [Geotools-devel] Copyright header policy

2016-04-22 Thread Justin Deoliveira 
Cool. I'll write up a quick one.
On Fri, Apr 22, 2016 at 1:01 PM Jody Garnett  wrote:

> If you would like to make that proposal we can update the developers guide
> and get it done ...
>
> --
> Jody Garnett
>
> On 22 April 2016 at 11:38, Justin Deoliveira  wrote:
>
>> Thanks Jody. That helps me understand.
>>
>> My thought is that the day to day overhead is not worth being able to
>> rest assured that in 75 years GeoTools will still be in good standing with
>> regard to copyright ;)
>>
>> My vote would be to change to a process where the copyright is set to the
>> current year that the file is created or undergoes a major
>> overhaul/rewrite. I
>>
>> $0.02
>>
>> On Fri, Apr 22, 2016 at 12:18 PM Andrea Aime <
>> andrea.a...@geo-solutions.it> wrote:
>>
>>> On Fri, Apr 22, 2016 at 8:06 PM, Jody Garnett 
>>> wrote:
>>>
 If the PMC wants to make this decision I would be -0, I understand that
 it would assist with github pull requests, but I feel we would do so by
 cutting a corner.

>>>
>>> I don't feel strongly either way, copyright updates can be automated
>>> (see Kevin's GeoServer script), but yeah, it's really annoying to require
>>> every single time a pull request comes in to update the copyright
>>> headers...
>>>
>>> Cheers
>>> Andrea
>>>
>>> --
>>> ==
>>> GeoServer Professional Services from the experts! Visit
>>> http://goo.gl/it488V for more information.
>>> ==
>>>
>>> Ing. Andrea Aime
>>> @geowolf
>>> Technical Lead
>>>
>>> GeoSolutions S.A.S.
>>> Via di Montramito 3/A
>>> 55054  Massarosa (LU)
>>> phone: +39 0584 962313
>>> fax: +39 0584 1660272
>>> mob: +39  339 8844549
>>>
>>> http://www.geo-solutions.it
>>> http://twitter.com/geosolutions_it
>>>
>>> *AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*
>>>
>>> Le informazioni contenute in questo messaggio di posta elettronica e/o
>>> nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
>>> loro utilizzo è consentito esclusivamente al destinatario del messaggio,
>>> per le finalità indicate nel messaggio stesso. Qualora riceviate questo
>>> messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
>>> darcene notizia via e-mail e di procedere alla distruzione del messaggio
>>> stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
>>> divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
>>> utilizzarlo per finalità diverse, costituisce comportamento contrario ai
>>> principi dettati dal D.Lgs. 196/2003.
>>>
>>>
>>>
>>> The information in this message and/or attachments, is intended solely
>>> for the attention and use of the named addressee(s) and may be confidential
>>> or proprietary in nature or covered by the provisions of privacy act
>>> (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
>>> Code).Any use not in accord with its purpose, any disclosure, reproduction,
>>> copying, distribution, or either dissemination, either whole or partial, is
>>> strictly forbidden except previous formal approval of the named
>>> addressee(s). If you are not the intended recipient, please contact
>>> immediately the sender by telephone, fax or e-mail and delete the
>>> information in this message that has been received in error. The sender
>>> does not give any warranty or accept liability as the content, accuracy or
>>> completeness of sent messages and accepts no responsibility  for changes
>>> made after they were sent or for other risks which arise as a result of
>>> e-mail transmission, viruses, etc.
>>>
>>> ---
>>>
>>
>
--
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] Copyright header policy

2016-04-22 Thread Jody Garnett 
If you would like to make that proposal we can update the developers guide
and get it done ...

--
Jody Garnett

On 22 April 2016 at 11:38, Justin Deoliveira  wrote:

> Thanks Jody. That helps me understand.
>
> My thought is that the day to day overhead is not worth being able to rest
> assured that in 75 years GeoTools will still be in good standing with
> regard to copyright ;)
>
> My vote would be to change to a process where the copyright is set to the
> current year that the file is created or undergoes a major
> overhaul/rewrite. I
>
> $0.02
>
> On Fri, Apr 22, 2016 at 12:18 PM Andrea Aime 
> wrote:
>
>> On Fri, Apr 22, 2016 at 8:06 PM, Jody Garnett 
>> wrote:
>>
>>> If the PMC wants to make this decision I would be -0, I understand that
>>> it would assist with github pull requests, but I feel we would do so by
>>> cutting a corner.
>>>
>>
>> I don't feel strongly either way, copyright updates can be automated (see
>> Kevin's GeoServer script), but yeah, it's really annoying to require
>> every single time a pull request comes in to update the copyright
>> headers...
>>
>> Cheers
>> Andrea
>>
>> --
>> ==
>> GeoServer Professional Services from the experts! Visit
>> http://goo.gl/it488V for more information.
>> ==
>>
>> Ing. Andrea Aime
>> @geowolf
>> Technical Lead
>>
>> GeoSolutions S.A.S.
>> Via di Montramito 3/A
>> 55054  Massarosa (LU)
>> phone: +39 0584 962313
>> fax: +39 0584 1660272
>> mob: +39  339 8844549
>>
>> http://www.geo-solutions.it
>> http://twitter.com/geosolutions_it
>>
>> *AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*
>>
>> Le informazioni contenute in questo messaggio di posta elettronica e/o
>> nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
>> loro utilizzo è consentito esclusivamente al destinatario del messaggio,
>> per le finalità indicate nel messaggio stesso. Qualora riceviate questo
>> messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
>> darcene notizia via e-mail e di procedere alla distruzione del messaggio
>> stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
>> divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
>> utilizzarlo per finalità diverse, costituisce comportamento contrario ai
>> principi dettati dal D.Lgs. 196/2003.
>>
>>
>>
>> The information in this message and/or attachments, is intended solely
>> for the attention and use of the named addressee(s) and may be confidential
>> or proprietary in nature or covered by the provisions of privacy act
>> (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
>> Code).Any use not in accord with its purpose, any disclosure, reproduction,
>> copying, distribution, or either dissemination, either whole or partial, is
>> strictly forbidden except previous formal approval of the named
>> addressee(s). If you are not the intended recipient, please contact
>> immediately the sender by telephone, fax or e-mail and delete the
>> information in this message that has been received in error. The sender
>> does not give any warranty or accept liability as the content, accuracy or
>> completeness of sent messages and accepts no responsibility  for changes
>> made after they were sent or for other risks which arise as a result of
>> e-mail transmission, viruses, etc.
>>
>> ---
>>
>
--
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] Copyright header policy

2016-04-22 Thread Justin Deoliveira 
Thanks Jody. That helps me understand.

My thought is that the day to day overhead is not worth being able to rest
assured that in 75 years GeoTools will still be in good standing with
regard to copyright ;)

My vote would be to change to a process where the copyright is set to the
current year that the file is created or undergoes a major
overhaul/rewrite. I

$0.02

On Fri, Apr 22, 2016 at 12:18 PM Andrea Aime 
wrote:

> On Fri, Apr 22, 2016 at 8:06 PM, Jody Garnett 
> wrote:
>
>> If the PMC wants to make this decision I would be -0, I understand that
>> it would assist with github pull requests, but I feel we would do so by
>> cutting a corner.
>>
>
> I don't feel strongly either way, copyright updates can be automated (see
> Kevin's GeoServer script), but yeah, it's really annoying to require
> every single time a pull request comes in to update the copyright
> headers...
>
> Cheers
> Andrea
>
> --
> ==
> GeoServer Professional Services from the experts! Visit
> http://goo.gl/it488V for more information.
> ==
>
> Ing. Andrea Aime
> @geowolf
> Technical Lead
>
> GeoSolutions S.A.S.
> Via di Montramito 3/A
> 55054  Massarosa (LU)
> phone: +39 0584 962313
> fax: +39 0584 1660272
> mob: +39  339 8844549
>
> http://www.geo-solutions.it
> http://twitter.com/geosolutions_it
>
> *AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*
>
> Le informazioni contenute in questo messaggio di posta elettronica e/o
> nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
> loro utilizzo è consentito esclusivamente al destinatario del messaggio,
> per le finalità indicate nel messaggio stesso. Qualora riceviate questo
> messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
> darcene notizia via e-mail e di procedere alla distruzione del messaggio
> stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
> divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
> utilizzarlo per finalità diverse, costituisce comportamento contrario ai
> principi dettati dal D.Lgs. 196/2003.
>
>
>
> The information in this message and/or attachments, is intended solely for
> the attention and use of the named addressee(s) and may be confidential or
> proprietary in nature or covered by the provisions of privacy act
> (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
> Code).Any use not in accord with its purpose, any disclosure, reproduction,
> copying, distribution, or either dissemination, either whole or partial, is
> strictly forbidden except previous formal approval of the named
> addressee(s). If you are not the intended recipient, please contact
> immediately the sender by telephone, fax or e-mail and delete the
> information in this message that has been received in error. The sender
> does not give any warranty or accept liability as the content, accuracy or
> completeness of sent messages and accepts no responsibility  for changes
> made after they were sent or for other risks which arise as a result of
> e-mail transmission, viruses, etc.
>
> ---
>
--
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] Copyright header policy

2016-04-22 Thread Andrea Aime 
On Fri, Apr 22, 2016 at 8:06 PM, Jody Garnett 
wrote:

> If the PMC wants to make this decision I would be -0, I understand that it
> would assist with github pull requests, but I feel we would do so by
> cutting a corner.
>

I don't feel strongly either way, copyright updates can be automated (see
Kevin's GeoServer script), but yeah, it's really annoying to require
every single time a pull request comes in to update the copyright headers...

Cheers
Andrea

-- 
==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.
==

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054  Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39  339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.



The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility  for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

---
--
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] Copyright header policy

2016-04-22 Thread Jody Garnett 
There are two schools of thought Justin. I had this discussion in detail on
the geowebache-devel
 with Arne
Kepp (and did some research at that time which was unsatisfying to both of
us - and probably scared away a contributor). See #355
 for epic heated
discussion - during which I learned two things:
- there is a difference of between publication vs the contents of the file
changing
- when transferring ownership from OpenPlans --> OSGeo we could of done a
search and replace (removing reference to OpenPlans)

As you say some projects write down the (c) of the initial file
contribution and call it a day ... reasoning being that the length of a
copyright is now 75+ years and by that time we will have diamond lattice
grid chips running a combination of javascript wrappers around q-bit
cores

The headers are supposed to be maintained each year the content changes in
a substantial way (the actual code stuff, not javadocs and comments).
Because we change our files often we usually get away with a range - "(C)
2003-2016". For files that change less often we could have gaps "(C) 2003,
2005, 2006-2009, 2016"

I understand that we could, as a PMC, decide to just go with initial file
creation date (which is admittedly often forms the bulk of the work).
However subsequent contributions can be very important, as someone who's
work has been rewritten on occasion can attest. We can make the argument
that we should be able to recover this information from github if needed,
and I guess we would need to if it ever comes up.

If the PMC wants to make this decision I would be -0, I understand that it
would assist with github pull requests, but I feel we would do so by
cutting a corner.



--
Jody Garnett

On 22 April 2016 at 06:01, Justin Deoliveira  wrote:

> Hi folks,
>
> Something I’ve been meaning to ask about is a clarification of why we have
> to update copyright header dates to the current year for every file change.
> As opposed to just having a static copyright header that signifies a
> “copyright event” (like the change over to osgeo) and then stands for that
> time forward.
>
> The reason I ask is because I think this policy has some negative things
> in my opinion. For one it’s a pain for both contributors and reviewers to
> enforce. I myself never remember as you probably well know and I would say
> I am a pretty regular contributor. It also adds noise to patches and pull
> requests, which also makes things harder to review than necessary.
>
>
> Anyways, just curious as to the rationale behind this decision. My
> apologies if this has been covered in previous email.
>
> -Justin
>
>
>
>
>
>
> --
> Find and fix application performance issues faster with Applications
> Manager
> Applications Manager provides deep performance insights into multiple
> tiers of
> your business applications. It resolves application problems quickly and
> reduces your MTTR. Get your free trial!
> https://ad.doubleclick.net/ddm/clk/302982198;130105516;z
> ___
> GeoTools-Devel mailing list
> GeoTools-Devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/geotools-devel
>
>
--
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] Reading GeoTiffs from HDFS

2016-04-22 Thread Jim Hughes 
Hi Simone,

Thanks for the feedback!

As quick response, for #1, I agree that using mosaicing / an image 
pyramid would be a great option.  I was mainly working at the prototype 
phase, and I wanted to have a discussion on the mailing lists 
(especially since changes are required in ImageIO-Ext or GeoTools and 
GeoServer.)

For #2, I do like the idea of having a cahce in the ImageInputStream.  
 From that suggestion, I take it that you'd be willing to entertain 
changes to the current ImageInputStreams and the additional of some way 
to cache data.

In terms of caching, do you have any suggestions?  Also, I'd be 
interested in any advice for how we can configure that cache and make 
those options available to a GeoServer admin appropriately.

Further, at a high-level, should the goal for this work be a community 
module?

Cheers,

Jim

On 04/22/2016 01:49 PM, Simone Giannecchini wrote:
> Dear Jim,
> quick feedback.
>
> First of all congratulation on making this work. As I suspected the
> bottleneck is getting the data out of HDFS.
> I can think about two things (which we are not mutually exclusive):
>
> -1- Maybe complex, put smaller bits into HFDS and use the mosaic to
> serve or even develop a light(er)weight layer that can pull the
> granules.
>
> This would help with WMS requests over large files as you'll end up
> use smaller chunks to satisfy them most of the time
>
> -2- We could build a more complex ImageInputStream that:
>
> - has an internal cache (file and or memory) that does not get thrown
> away upon each request but tends to live longer for each single file
> in HDF
> - we would have different streams reuse the same cache. Multiple
> requests might read data from the cache concurrently but when data is
> not there, we would block the thread for the request, go back to HFDS,
> pull the data, write to the cache and so on
>
> We could put together 1 and 2 to make things faster.
>
> Hope this helps, anyway, I am in favour of exploring this in order to
> allow the GeoServer stack to support data from HDFS.
>
> Regards,
> Simone Giannecchini
> ==
> GeoServer Professional Services from the experts!
> Visit http://goo.gl/it488V for more information.
> ==
> Ing. Simone Giannecchini
> @simogeo
> Founder/Director
>
> GeoSolutions S.A.S.
> Via di Montramito 3/A
> 55054  Massarosa (LU)
> Italy
> phone: +39 0584 962313
> fax: +39 0584 1660272
> mob:   +39  333 8128928
>
> http://www.geo-solutions.it
> http://twitter.com/geosolutions_it
>
> ---
> AVVERTENZE AI SENSI DEL D.Lgs. 196/2003
> Le informazioni contenute in questo messaggio di posta elettronica e/o
> nel/i file/s allegato/i sono da considerarsi strettamente riservate.
> Il loro utilizzo è consentito esclusivamente al destinatario del
> messaggio, per le finalità indicate nel messaggio stesso. Qualora
> riceviate questo messaggio senza esserne il destinatario, Vi preghiamo
> cortesemente di darcene notizia via e-mail e di procedere alla
> distruzione del messaggio stesso, cancellandolo dal Vostro sistema.
> Conservare il messaggio stesso, divulgarlo anche in parte,
> distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità
> diverse, costituisce comportamento contrario ai principi dettati dal
> D.Lgs. 196/2003.
>
> The information in this message and/or attachments, is intended solely
> for the attention and use of the named addressee(s) and may be
> confidential or proprietary in nature or covered by the provisions of
> privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New
> Data Protection Code).Any use not in accord with its purpose, any
> disclosure, reproduction, copying, distribution, or either
> dissemination, either whole or partial, is strictly forbidden except
> previous formal approval of the named addressee(s). If you are not the
> intended recipient, please contact immediately the sender by
> telephone, fax or e-mail and delete the information in this message
> that has been received in error. The sender does not give any warranty
> or accept liability as the content, accuracy or completeness of sent
> messages and accepts no responsibility  for changes made after they
> were sent or for other risks which arise as a result of e-mail
> transmission, viruses, etc.
>
>
> On Sun, Apr 17, 2016 at 9:49 PM, Jim Hughes  wrote:
>> Hi all,
>>
>> I want to report on my success with registering and displaying GeoTiffs
>> stored on HDFS.  There are some limitations with this approach;
>> particularly, I am unsure if there's anyway to cache / memory-map the
>> data.  As such, I believe each request is re-downloading the entire file.
>>
>> Generally, I hope to document my approach well enough so that others
>> could follow it (if needed) and to solicit feedback.  In terms of
>> feedback, I'd love to hear 1) if there are improvements, and 2) if the
>> changes are reasonable enough to be considered for a proposal/merge request.
>>
>> That out of the way,

Re: [Geotools-devel] wfs-ng vs wfs datastore fight!

2016-04-22 Thread Andrea Aime 
On Fri, Apr 22, 2016 at 7:48 PM, Andrea Aime 
wrote:

> Given this, I'd still make it possible for GeoTools users to upgrade from
> wfs to wfs-ng without this much hassle by exposing the
> actual native type name.
>

Actually, given that wfs-ng has been available for a few releases now,
probably the best option is to have
a configuration parameter that allows controlling this mapping behavior

Cheers
Andrea


-- 
==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.
==

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054  Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39  339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.



The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility  for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

---
--
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] Reading GeoTiffs from HDFS

2016-04-22 Thread Simone Giannecchini 
Dear Jim,
quick feedback.

First of all congratulation on making this work. As I suspected the
bottleneck is getting the data out of HDFS.
I can think about two things (which we are not mutually exclusive):

-1- Maybe complex, put smaller bits into HFDS and use the mosaic to
serve or even develop a light(er)weight layer that can pull the
granules.

This would help with WMS requests over large files as you'll end up
use smaller chunks to satisfy them most of the time

-2- We could build a more complex ImageInputStream that:

- has an internal cache (file and or memory) that does not get thrown
away upon each request but tends to live longer for each single file
in HDF
- we would have different streams reuse the same cache. Multiple
requests might read data from the cache concurrently but when data is
not there, we would block the thread for the request, go back to HFDS,
pull the data, write to the cache and so on

We could put together 1 and 2 to make things faster.

Hope this helps, anyway, I am in favour of exploring this in order to
allow the GeoServer stack to support data from HDFS.

Regards,
Simone Giannecchini
==
GeoServer Professional Services from the experts!
Visit http://goo.gl/it488V for more information.
==
Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via di Montramito 3/A
55054  Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob:   +39  333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---
AVVERTENZE AI SENSI DEL D.Lgs. 196/2003
Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate.
Il loro utilizzo è consentito esclusivamente al destinatario del
messaggio, per le finalità indicate nel messaggio stesso. Qualora
riceviate questo messaggio senza esserne il destinatario, Vi preghiamo
cortesemente di darcene notizia via e-mail e di procedere alla
distruzione del messaggio stesso, cancellandolo dal Vostro sistema.
Conservare il messaggio stesso, divulgarlo anche in parte,
distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità
diverse, costituisce comportamento contrario ai principi dettati dal
D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely
for the attention and use of the named addressee(s) and may be
confidential or proprietary in nature or covered by the provisions of
privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New
Data Protection Code).Any use not in accord with its purpose, any
disclosure, reproduction, copying, distribution, or either
dissemination, either whole or partial, is strictly forbidden except
previous formal approval of the named addressee(s). If you are not the
intended recipient, please contact immediately the sender by
telephone, fax or e-mail and delete the information in this message
that has been received in error. The sender does not give any warranty
or accept liability as the content, accuracy or completeness of sent
messages and accepts no responsibility  for changes made after they
were sent or for other risks which arise as a result of e-mail
transmission, viruses, etc.


On Sun, Apr 17, 2016 at 9:49 PM, Jim Hughes  wrote:
> Hi all,
>
> I want to report on my success with registering and displaying GeoTiffs
> stored on HDFS.  There are some limitations with this approach;
> particularly, I am unsure if there's anyway to cache / memory-map the
> data.  As such, I believe each request is re-downloading the entire file.
>
> Generally, I hope to document my approach well enough so that others
> could follow it (if needed) and to solicit feedback.  In terms of
> feedback, I'd love to hear 1) if there are improvements, and 2) if the
> changes are reasonable enough to be considered for a proposal/merge request.
>
> That out of the way, here's the rough outline:
>
> 1.  Register additional URL handlers.
> 2.  Convince validation layers in GeoServer that 'hdfs' is an ok URL scheme.
> 3.  Get bytes out of the HDFS file.
>
> For step 1, note that Java's URL scheme is pluggable via
> java.net.URLStreamHandler.  The docs(1) point out that one can call
> URL.setURLStreamHandlerFactory to setup a Factory to provide such a
> handler.  This method can only be called once, and folks from the
> internet (2) do yoga since Tomcat already registers a factory.  They
> seem to have missed the fact that the Tomcat factory actually lets you
> add your own.  I provide a gist (3) to show a little bean which will
> instantiate a Hadoop URL handler and try to install it using both of
> those methods.
>
> There are two places I found in GeoServer which validate the URL given
> in the page for adding a GeoTiff.  The first is the GeoServer
> FileExistValidator which calls out to a Wicket UrlValidator. Telling the
> Wicket class to allow_all_schemes knocks out that issue.  For the
> second, in the

Re: [Geotools-devel] wfs-ng vs wfs datastore fight!

2016-04-22 Thread Andrea Aime 
On Thu, Apr 21, 2016 at 7:52 PM, Jody Garnett 
wrote:

>
> > Hum... wfs-ng was supposed to be a drop-in replacement, not having a
>>> upgrade
>>> > path seems to be a serious issue to me.
>>> > Where is it documented?
>>>
>>> Quick search got us this:
>>> https://osgeo-org.atlassian.net/browse/GEOT-4883
>>> and we googled some more messages on it.
>>
>> A backwards incompable change blocking upgrades presented as a GeoServer
>> requirement and resolved as "not a bug"...
>>
>
> Reading this issue, and I agree it is not a bug - the new datastore is
> behaving as designed in order to preserve valid names for GML generation.
>

Jody I believe we have a problem here, in terms of responsibilities. Is it
really the job of the stores to decide what is valid, and what is not?
Not all usage is WFS cascading, stores are used for a number of other
cases, which might have different naming limitations.

Also from a consistency point of view, did you know that you can create a
typename like "a:b" using a postgis store?

> create table "a:b" (id serial, geom geometry(POLYGON, 4326));
CREATE TABLE

And then the datastore is happily returning a:b as the typename,
And of course we also have shapefiles, on linux a file name like a:b.shp is
also completely valid.

With you and Niels claiming that is not a bug, we have a difficult
situation, as nobody else can just jump in and make
the upgrade path easier, since according to your evaluation, we'd be
introducing a bug... (and from that same point of view,
then jdbc store and shapefile store are buggy too...).

Unless of course your position is a bit more relaxed, as in, you don't
consider that a bug, but the behavior returning the native
name is valid too.


>
> For this specific case the old datastore would of required a similar fix
> (to prevent type names being of the form
> "local_namespace:remote_namespace:typename").
>
>
>> As far as I know GeoServer was already handling the ":" in the name, and
>> the admin is able to rename layers as they see fit.
>> Yep, there is code stripping the prefix:
>>
>> https://github.com/geoserver/geoserver/blob/master/src/main/src/main/java/org/geoserver/catalog/CatalogBuilder.java#L1212
>>
>
> GeoServer may have work arounds, but I do not mind if the new DataStore is
> taking more care to produce consistent type names.
>

See Mark's situation though. I'm also aware of other situations like this
one that will be broken by an upgrade (systems configured
to work against well known type names in remote servers).

Mind, I checked, GeoServer is not affected, because internally we have this
little thing that hides from GeoServer the columns and
guess what, replaces them with underscores. See here:

https://github.com/geoserver/geoserver/blob/master/src/main/src/main/java/org/vfny/geoserver/util/DataStoreUtils.java#L74

and then the retyper does this:

https://github.com/geoserver/geoserver/blob/master/src/main/src/main/java/org/geoserver/feature/retype/RetypingDataStore.java#L252

So luckily for GeoServer even if the store did change its behavior, nothing
changed config wise (we end up seeing a_b as the native
type name regardless of whether the store returns a:b or a_b because of the
above wrapper).

Given this, I'd still make it possible for GeoTools users to upgrade from
wfs to wfs-ng without this much hassle by exposing the
actual native type name.
If the worry is about valid type names for GML, I'm more than happy to
contribute a little utility wrapping store based on the GeoServer one
that does the same thing as RetypingDataStore, but centralized in one
place, and optional, only for those that actually need to do GML encoding.
A DataUtilities.makeGmlCompatible(DataStore) -> DataStore of sorts.

Cheers
Andrea


-- 
==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.
==

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054  Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39  339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.



The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or

[Geotools-devel] Copyright header policy

2016-04-22 Thread Justin Deoliveira 
Hi folks,

Something I’ve been meaning to ask about is a clarification of why we have
to update copyright header dates to the current year for every file change.
As opposed to just having a static copyright header that signifies a
“copyright event” (like the change over to osgeo) and then stands for that
time forward.

The reason I ask is because I think this policy has some negative things in
my opinion. For one it’s a pain for both contributors and reviewers to
enforce. I myself never remember as you probably well know and I would say
I am a pretty regular contributor. It also adds noise to patches and pull
requests, which also makes things harder to review than necessary.


Anyways, just curious as to the rationale behind this decision. My
apologies if this has been covered in previous email.

-Justin
--
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] wfs-ng vs wfs datastore fight!

2016-04-22 Thread Mark Prins 
2016-04-22 9:18 GMT+02:00 Andrea Aime :

> On Thu, Apr 21, 2016 at 2:27 PM, Mark Prins  wrote:
>
>> > Where is it documented?
>>
>> Quick search got us this:
>> https://osgeo-org.atlassian.net/browse/GEOT-4883
>> and we googled some more messages on it.
>>
>> This happened over christmas so it's gone a bit hazy by now, we upgraded
>> from 9.x to 14.x during that period, (late yes, but we were tied to
>> supporting Java 6)
>>
>
> Could you share with us a bit more about how the change affected your
> application?
> Your request of keeping support for the old wfs store seems to imply
> switching
> to wfs-ng would be a significant rework of your application
>
>
Sure,

In our application we have an admin define a view/map service (eg. WMS),
while the capabilities are retrieved we also try to discover a WFS (to use
for attribute info, tables, graphs etc.), if not discoverable the admin may
specify one himself (even a different service).

We store the WFS typenames in a table in our backend like so:

id;  description;geom-attr; type_name;
;writable; source id
"68";"Bedrijventerrein_kavels"; "geom";"Economie:Bedrijventerrein_kavels";
FALSE;14
"70";"EcMo_Wonen_regios";   "geom";"Economie:EcMo_Wonen_regios";
FALSE;14
"79";"BeschikbarePandenOpBedrijventerreinen";"geom";"IBIS_BeschikbarePandenOpBedrijventerreinen";FALSE;16
"80";"";"geom";"bedrijvenkavels_tijdelijk";
 TRUE;11


"Economie:Bedrijventerrein_kavels" being the typename reported by the WFS,
a test case illustrating this [0]. We use this typename to request features
from the WFS [1],

If I recall correctly wfs-ng reports "Economie_EcMo_Wonen_regios" as a
typename (instead of "Economie:EcMo_Wonen_regios") which breaks when just
using that in a getfeatures request also because the underscore is a
legitimate character in both layer name and namespace name (contrary to the
colon) it becomes impossible to determine which part is which (not possible
to use a simple split on ':' anymore). Our stored type_name is used in
various places in our application.

We can/should probably expand our model to store typename and namespace
separately to use this new naming convention and update the code to use
that. And cook up a migration script for our users.


[0]
https://github.com/flamingo-geocms/flamingo/blob/master/viewer-admin/src/test/java/nl/b3p/viewer/admin/stripes/WFSTypeNamingTest.java#L199
[1] eg.
https://github.com/flamingo-geocms/flamingo/blob/master/viewer-config-persistence/src/main/java/nl/b3p/viewer/config/services/WFSFeatureSource.java#L246



-- 
Disclaimer;
This message is just a reflection of what I thought at the time of sending.
The message may contain information that is not intended for you or that
you don't understand.
--
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] ImageMosaic API refactor proposal

2016-04-22 Thread Niels Charlier 

On 21-04-16 20:01, Jody Garnett wrote:
Thanks for the clarification Niels, I was not aware that Prop was part 
of the current API.


Still we have some duplication here right? Should we try running the 
property collectors, look at the results, and generate the schema 
appropriate to store the results?


Jody, have a look at this: 
http://geoserver.geo-solutions.it/multidim/en/imagemosaic/mosaic_indexer.html#imagemosaic-xml-indexer-example
You see that in the current index configuration file, there is a 
separate section for the schemas and the property collectors. Both 
sections provide a completely different kind of information.


I do not think that there is duplication. You cannot run property 
collectors if you do not have a schema yet. The property collectors are 
run per feature (granule). The schema is made before you can even create 
features. The schema is defined per coverage inside the store and can 
differ between them. The property collectors are unaware of the 
coverages, and they may operate for multiple. Once the schema has been 
created for a coverage, the features are created and then values might 
be needed for each feature. This is the time that (for some properties) 
the appropriate property collectors are looked up and called to fill in 
those values.

In summary:
the schema -> defines the metadata (table structure) of the index per 
coverage
the property collectors -> defines how to grab the data for the index 
(from the metadata of the granules)

two different functions altogether.





--
Jody Garnett

On 21 April 2016 at 03:00, Niels Charlier > wrote:


On 17-04-16 04:17, Jody Garnett wrote:

This proposal (and email thread) should be orthogonal to the
RnD discussion on setting up a mosaic with multiple projections.

This mosaic is focused on creating/managing the index used.
One part I cannot wrap my head around is the workflow behind
these two methods - createSchema(String name,
CoordinateReferenceSystem crs) which creates the FeatureType
used for the index, and Collection
getPropertiesCollectors().
It seems these two methods need to be implemented together, a
single PropertyCollector may collect more than once value on
examining a raster. I am not sure how to line up these
collected values with the created schema without more information.


Just to be clear, the proposal doesn't change anything about
PropertyCollectors work. This would simply continue to work as it
is working right now, until somebody makes a proposal to change it
to something else!

These two things have a completely different function. The schema
defines the structure of the index.

Property Collectors are customisable mechanisms that provides
(some of the) values of the index per granule, taking information
from the file and/or the coverage metadata.

The schema creation includes attributes that are not collected
with a PropertyCollector (such as location), while one Propery
Collector can potentially supply multiple attribute values.


 We also have String getParameter(Prop prop) defined, but the
class Prop is not included in the proposal.


Prop already exists in the current API.


Suggestions:
- Use a list of PropertiesCollectors to provide order, which
can then be used to determine the Schema?
- Language consistency - raster / granule
- Language consistency - could we use AttributeDescriptor
rather than introduce a new thing called "Prop" ?


You do know that Props are configuration properties, nothing to do
with features?

- Are the class responsibilities clear: GranuleCatalogManager,
GranuleCatalog, GranuleStore, MosaicHarvester, etc...

(It could also be that since I am unfamiliar with the
internals being refactored that I am just struggling with the
language - so if everyone else is okay ...)


Regards
Niels




--
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] wfs-ng vs wfs datastore fight!

2016-04-22 Thread Andrea Aime 
On Thu, Apr 21, 2016 at 2:27 PM, Mark Prins  wrote:

> > Where is it documented?
>
> Quick search got us this:
> https://osgeo-org.atlassian.net/browse/GEOT-4883
> and we googled some more messages on it.
>
> This happened over christmas so it's gone a bit hazy by now, we upgraded
> from 9.x to 14.x during that period, (late yes, but we were tied to
> supporting Java 6)
>

Could you share with us a bit more about how the change affected your
application?
Your request of keeping support for the old wfs store seems to imply
switching
to wfs-ng would be a significant rework of your application

Cheers
Andrea

-- 
==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.
==

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054  Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39  339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.



The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility  for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

---
--
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel