Re: [Geotools-devel] RFC: Transform DataStore

[Jim Hughes]

Hi Ian,

Neat!  Thanks!

I want to do two things:  first, I want to share the list of 
FeatureSource modifications that I've encountered so far, and second to 
draw an analogy to other query engines.


Since I pretty much never work with SQL databases, I have encountered a 
number of problems would SQL Views (or Database table views) probably 
would have solved.  As those were unavailable, the options have been 
do-without/work-around or implement a GeoTools DataStore (or use a 
GS-based solution). Here's my quick list of things I've seen:


1.  Desire to enrich data from a GeoTools DataStore with some external 
datasource.  The first example I encountered with this involved getting 
data from a triple store (so there was no sensible GT DataStore for 
it).  I believe this case can be thought of as a left outer join (where 
the GT DS is on the left).  Your WPS seems to work in the same vein.  
One handy thing that this DS did was to split up a CQL filter into the 
part which could passed to the GT DS.


2.  Desire to glue together one or more DataStores.  GeoMesa has a 
Merged View DS to do this.  The assumption is that each of the DS has a 
FS with the same SFT.  This is a kin to a table union.


3.  Desire to query across layers.  The GeoServer Query Layer extension 
does this.  One co-worker suggested that this could be thought of as a 
nested inner join.


4.  Desire to modify or add columns based on functions.  (That's what 
this PR is aiming at.)


All of this makes me think of the Catalyst query engine in Spark.  
Apache Spark offers an SQL interface.  The SQL query is broken down into 
logical plan, which is optimized, and then converted into a physical 
plan.  There is a DataSource interface which is similar to the GT 
DataSource API.  Taken together, I'm starting to think of GeoServer in 
that same role as a query engine.  The various places that a 
FeatureCollection is wrapped effectively line up with parts of a 
physical plan.  The challenges I see are 1) that's there is presently 
little in terms of "explanability" (e.g. one cannot inspect a query's 
execution plan) and 2) there's no obvious place to plug any sort of 
optimization rule (instead any wrapping point needs to provide a 
FeatureCollection which will delegate visitors appropriately, etc.)


Anyhow, hopefully the concrete list helps us see places where we could 
focus on supporting individual query patterns / use cases. For my second 
point, I don't have a concrete goal; I suppose I'm sharing this thought 
/ analogy to see if others have had similar thoughts (or have an obvious 
plan around it).


Cheers,

Jim

On 11/3/2020 3:33 PM, Ian Turton wrote:
Currently, it's a WPS process - the code is at 
https://github.com/ianturton/tablejoin 
<https://github.com/ianturton/tablejoin>


It's built around a FilterVisitor that replaces join values with 
literals I think.




Ian

On Tue, 3 Nov 2020 at 19:25, Jim Hughes <mailto:jhug...@ccri.com>> wrote:


Hi Ian,

Interesting.  Is the Joining datastore somewhere online already?

I'll admit to having written similar datastores previously.  One
of my ideas for using this kind of DataStore would be to leverage
functions like those in the GS Query Layer extension to pull
attributes from other layers.  Similarly, it ought to be trivial
to pull a column from a CSV on disk.

As one starts down that path, there are syntactical annoyances  (I
cannot see an easy way to say "get a bunch of columns from other
there" without having a function call per column), and performance
considerations (reading from a file/database repeated for the same
information among separate queries is less than ideal, so caching
would be nice).  That said, in a pinch, this approach may provide
a really, really quick solution.

Cheers,

Jim

On 11/3/20 1:53 PM, Ian Turton wrote:

This looks interesting, I wonder how hard it would be to merge in
the work I did on Joining datastores that could create "views"
across non-jdbc datastores so you could add a CSV to a geometry
in another store, I should dig that code out and look at it
again. Maybe a christmas lockdown project?

    Ian

On Tue, 3 Nov 2020 at 18:19, Jim Hughes mailto:jhug...@ccri.com>> wrote:

Hi all,

At various times, Jody and I have chatted about having a "CQL
View" in
GeoServer (or something similar akin to the SQL Views) that'd
leverage
the Transform module and allow one to add columns to a
FeatureSource
based on expressions.  Such expressions could use existing or
custom CQL
functions and that'd open a world of possibilities.

Last week, I asked Jody about this idea again and he
indicated that one
would really just need to write a DataStoreFactory.  I banged
out an
initial im

Re: [Geotools-devel] RFC: Transform DataStore

[Jim Hughes]

Hi Andrea,

Thanks, I'm going to make some time to clean up unit tests, etc. Also, 
thanks for pointing out the PMD error!


I'm curious how storing the transforms in the ResourcePool would work.  
Would it just store a List?  Or should it have an 
encapsulating TransformObject?  (I'm wondering if any of the work that 
I'm doing would help the GS UI work or if the UI work is logically 
orthogonal.)


Cheers,

Jim

On 11/4/2020 2:53 AM, Andrea Aime wrote:

Hi Jim,
for a long time I wanted to utilize the TransformDataStore to power 
renaming and retyping in GeoServer.
I was hoping to get that through configuration in the layer page, 
making the attribute details table editable...
and that's why it never happened in the end, changing the UI and the 
associated config is not trivial
(once you have that, setting up the transformation in the ResourcePool 
is easy).


As far as I can tell, the datastore allows to transform one feature 
type, with some text based definitions.
It will keep some attributes un-modified, and allow to add new 
computed/renamed ones at the end
(or if one goes for specifying only transformations, get full control 
of order too, as one could
add "att=att" "transformations" keeping a certain attribute 
un-modified, but in the desired position).


The approach you're going towards makes it easier to get it going, 
it's probably time to allow "better"

to go forward in face of a "best" that does not want to come :-D
Go for it.

Cheers
Andrea


On Tue, Nov 3, 2020 at 7:20 PM Jim Hughes <mailto:jhug...@ccri.com>> wrote:


Hi all,

At various times, Jody and I have chatted about having a "CQL
View" in
GeoServer (or something similar akin to the SQL Views) that'd
leverage
the Transform module and allow one to add columns to a FeatureSource
based on expressions.  Such expressions could use existing or
custom CQL
functions and that'd open a world of possibilities.

Last week, I asked Jody about this idea again and he indicated
that one
would really just need to write a DataStoreFactory.  I banged out an
initial implementation as a starting place for the conversation. 
This
implementation provides a way to project down to a subset of
attributes
as well as add new columns/attributes to a FeatureStore.

If folks like it as-is, I'd be happy to add unit tests, and
documentation wherever we see fit.  Thanks in advance for some
feedback
on this idea!

https://github.com/geotools/geotools/pull/3196/files
<https://github.com/geotools/geotools/pull/3196/files>

Cheers,

Jim



___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
<mailto:GeoTools-Devel@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/geotools-devel
<https://lists.sourceforge.net/lists/listinfo/geotools-devel>



--

Regards, Andrea Aime

== GeoServer Professional Services from the experts! Visit 
http://goo.gl/it488V <http://goo.gl/it488V> for more information. == 
Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di 
Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 
0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it 
<http://www.geo-solutions.it> http://twitter.com/geosolutions_it 
<http://twitter.com/geosolutions_it> 
--- /Con 
riferimento alla normativa sul trattamento dei dati personali (Reg. UE 
2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si 
precisa che ogni circostanza inerente alla presente email (il suo 
contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è 
riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il 
messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra 
operazione è illecita. Le sarei comunque grato se potesse darmene 
notizia. This email is intended only for the person or entity to which 
it is addressed and may contain information that is privileged, 
confidential or otherwise protected from disclosure. We remind that - 
as provided by European Regulation 2016/679 “GDPR” - copying, 
dissemination or use of this e-mail or the information herein by 
anyone other than the intended recipient is prohibited. If you have 
received this email by mistake, please notify us immediately by 
telephone or e-mail./


___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] RFC: Transform DataStore

[Jim Hughes]

Hi Ian,

Interesting.  Is the Joining datastore somewhere online already?

I'll admit to having written similar datastores previously.  One of my 
ideas for using this kind of DataStore would be to leverage functions 
like those in the GS Query Layer extension to pull attributes from other 
layers.  Similarly, it ought to be trivial to pull a column from a CSV 
on disk.


As one starts down that path, there are syntactical annoyances (I cannot 
see an easy way to say "get a bunch of columns from other there" without 
having a function call per column), and performance considerations 
(reading from a file/database repeated for the same information among 
separate queries is less than ideal, so caching would be nice).  That 
said, in a pinch, this approach may provide a really, really quick 
solution.


Cheers,

Jim

On 11/3/20 1:53 PM, Ian Turton wrote:
This looks interesting, I wonder how hard it would be to merge in the 
work I did on Joining datastores that could create "views" across 
non-jdbc datastores so you could add a CSV to a geometry in another 
store, I should dig that code out and look at it again. Maybe a 
christmas lockdown project?


Ian

On Tue, 3 Nov 2020 at 18:19, Jim Hughes <mailto:jhug...@ccri.com>> wrote:


Hi all,

At various times, Jody and I have chatted about having a "CQL
View" in
GeoServer (or something similar akin to the SQL Views) that'd
leverage
the Transform module and allow one to add columns to a FeatureSource
based on expressions.  Such expressions could use existing or
custom CQL
functions and that'd open a world of possibilities.

Last week, I asked Jody about this idea again and he indicated
that one
would really just need to write a DataStoreFactory.  I banged out an
initial implementation as a starting place for the conversation. 
This
implementation provides a way to project down to a subset of
attributes
as well as add new columns/attributes to a FeatureStore.

If folks like it as-is, I'd be happy to add unit tests, and
documentation wherever we see fit.  Thanks in advance for some
feedback
on this idea!

https://github.com/geotools/geotools/pull/3196/files

Cheers,

Jim



___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
<mailto:GeoTools-Devel@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/geotools-devel



--
Ian Turton



___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

[Geotools-devel] RFC: Transform DataStore

[Jim Hughes]

Hi all,

At various times, Jody and I have chatted about having a "CQL View" in 
GeoServer (or something similar akin to the SQL Views) that'd leverage 
the Transform module and allow one to add columns to a FeatureSource 
based on expressions.  Such expressions could use existing or custom CQL 
functions and that'd open a world of possibilities.


Last week, I asked Jody about this idea again and he indicated that one 
would really just need to write a DataStoreFactory.  I banged out an 
initial implementation as a starting place for the conversation.  This 
implementation provides a way to project down to a subset of attributes 
as well as add new columns/attributes to a FeatureStore.


If folks like it as-is, I'd be happy to add unit tests, and 
documentation wherever we see fit.  Thanks in advance for some feedback 
on this idea!


https://github.com/geotools/geotools/pull/3196/files

Cheers,

Jim



___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] Counting in GeoTools: going beyond the integer range

[Jim Hughes]

Hi all,

The JavaDoc on this method reminded me of one of the points I wanted to 
suggest.  If there are multiple methods for counting records, it may be 
good to discuss the semantics.  If I recall, some of the methods in 
GeoTools have the idea that returning a -1 is suitable way to 
communicate that getting the exact count would be too expensive.


As an anchor point, I'm kinda happy when I see a distributed system 
scanning over a million-ish records per second.  With that back of the 
envelope, if a filter matched a billion or so features, it may take 
10-15 minutes to get an exact count.


In GeoMesa, we implemented a system property around returning exact 
counts or not.  I think we noticed this the most when GeoServer returned 
GeoJson since that request pathway calls getCount and then getFeatures 
(which is GeoMesa's case basically repeats the query!).


Anyhow, having an 'int' method and a 'long' method to call may help 
limit the amount of time spent counting;).


Cheers,

Jim

On 9/2/2020 1:54 PM, Andrea Aime wrote:

Hi Jody,
I like this road, works fine for FeatureSource.
What about FeatureCollection though? It's already using "int size()", 
from the interface:


/** * Please note this operation may be expensive when working with 
remote content. * * @see java.util.Collection#size() */ int size();


Cheers
Andrea


On Wed, Sep 2, 2020 at 7:45 PM Jody Garnett > wrote:


Here is another softer approach:
/**
 * @return Returns the number of features in this collection, if
this collection contains more than Integer.MAX_VALUE elements,
returns Integer.MAX_VALUE.
 * @deprecated Please use count()
 */
public int getCount();

public long size();

This gives us a clear api migration and does not immediately break
projects when they upgrade. The size() name matches how
Collections.size() handles a size greater than the range of
Integer.MAX_VALUE.

--
Jody Garnett


On Wed, 2 Sep 2020 at 06:43, Andrea Aime
mailto:andrea.a...@geo-solutions.it>> wrote:

Hi, any other opinion on this?

Personally I would not like breaking all existing store
implementations and clients, for a "clean break" it seems
quite bloody :-D
But Jody suggests to go that way.

Some tie breaker, or even multiple votes leading to another
tie, would be appreciated, ha!

Cheers
Andrea

On Thu, Aug 27, 2020 at 11:16 PM Andrea Aime
mailto:andrea.a...@geo-solutions.it>> wrote:

Hi Jody,
comment inline.

On Thu, Aug 27, 2020 at 11:06 PM Jody Garnett
mailto:jody.garn...@gmail.com>>
wrote:

We are just about to start a new release cycle, API
changes are a short-term pain, but the most
maintainable approach long term.

As for the change, how about returning a returning
long? Existing client code that used integrer would be
easy to update.

long count = featureSource.getCount(query);

Or:

int count = (int) featureSource.getCount(query);


We can gauge how easy it is by doing the switch in GT/GS...
Maybe it could be done as a refactor... but I cannot
imagine exactly how yet. Closest thing to something
working may be:

1) Rename existing getCount to getCountOld via refactor
2) Add a new method called getCount, returning long, make
it abstract, have a default implementation of getCountOld
delegating to getCount
3) Fix all implementations, switching them from
getCountOld to getCount
4) Inline getCountOld, that should fix all calling points

Hopefully that should not be too much work, I hope there
are few implementations of FeatureSource/Collection but many
client code calls using them.


If you really want Java 8 has Math.toIntExact
method, that produces an exception if the long is out
of range:

int count = Math. toIntExact(
featureSource.getCount(query) );


That would also work yes

Cheers
Andrea

== GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information. == Ing. Andrea
Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di
Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313
fax: +39 0584 1660272 mob: +39 339 8844549
http://www.geo-solutions.it
http://twitter.com/geosolutions_it
---
/Con riferimento alla normativa sul trattamento dei dati
personali 

Re: [Geotools-devel] QA on generics: unchecked and diamond

[Jim Hughes]

Hi all,

I poked at this a little bit today.  I managed to find some common 
patterns, so I worked across the code base and knocked out a few smaller 
modules.


When I tried gt-coverage, I hit issues with JAI using raw Maps.  Should 
the plan there be to Suppress those readily?


In gt-referencing, I hit two issues:

A.  Raw 'Unit' needing to be cast to Unit and the like.  Sounds 
like that a suppression?


B.  Lots of declarations in the AuthorityFactoryProxy[1] look like this:
privatestaticfinalClass/**/[] TYPES

Can the comments be removed?

Generally, how much of the API should be up for change as part of this?

For instance, GeneralParameterValue maybe should take type 
parameters[2], and ObjectCache[3] could also be parameterized (or 
replaced with something like Caffeine or Guava.


Thoughts?

Cheers,

Jim

1. 
https://github.com/geotools/geotools/blob/master/modules/library/referencing/src/main/java/org/geotools/referencing/factory/AuthorityFactoryProxy.java#L77


2. 
https://github.com/geotools/geotools/blob/master/modules/library/opengis/src/main/java/org/opengis/parameter/GeneralParameterValue.java#L28


3.


On 7/6/20 2:46 AM, Andrea Aime wrote:
On Sun, Jul 5, 2020 at 8:44 PM Jody Garnett > wrote:



Looking at the diff that is indeed much more readable.

I would like to help out and really focus on the docs and code
examples.

Do you have specific examples that you did not know how to fix
that you could point me at?


There are cases that I think cannot be fixed, but only suppressed, 
like these:


  * Using reflection to get value of a field, the cast on return is
unchecked, two cases here:

https://github.com/geotools/geotools/pull/3043/files#diff-bd3a99c28048a08c5f16c25cb33d5fe5R123
  * Using the Array class, which returns Object results that need
casting:

https://github.com/geotools/geotools/pull/3043/files#diff-70e0570500f24cc12d8ac92e170d5aa2R172
  * Using a enum value to replace null, Null.PLACEHOLDER is not the
same type as the keys in the map, by design:

https://github.com/geotools/geotools/pull/3043/files#diff-4eb4e400417d4aa1f96fe91b67e5a704R457

However there are also cases where I'm not sure... RangeSet is the 
festival of unchecked casts, but I could not
find an easy way to address them: 
https://github.com/geotools/geotools/pull/3043/files#diff-a6babed8dadc28c52af15c4c531afbecR50
I have the feeling the API and functionality ought to be changed in 
order to avoid the current issues, which is one the

things I'm trying _not_ to do in this effort.

Cheers
Andrea

== GeoServer Professional Services from the experts! Visit 
http://goo.gl/it488V for more information. == Ing. Andrea Aime 
@geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 
55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: 
+39 339 8844549 http://www.geo-solutions.it 
http://twitter.com/geosolutions_it 
--- /Con 
riferimento alla normativa sul trattamento dei dati personali (Reg. UE 
2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si 
precisa che ogni circostanza inerente alla presente email (il suo 
contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è 
riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il 
messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra 
operazione è illecita. Le sarei comunque grato se potesse darmene 
notizia. This email is intended only for the person or entity to which 
it is addressed and may contain information that is privileged, 
confidential or otherwise protected from disclosure. We remind that - 
as provided by European Regulation 2016/679 “GDPR” - copying, 
dissemination or use of this e-mail or the information herein by 
anyone other than the intended recipient is prohibited. If you have 
received this email by mistake, please notify us immediately by 
telephone or e-mail./




___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel



___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

[Geotools-devel] Eclipse EMF dependencies in GeoTools and GeoServer

[Jim Hughes]

Hi all,

A colleague noticed that GeoTools is pointing at one version of Eclipse 
EMF jars[1] and GeoServer is pointing at another[2].


Anyhow, it appears that version 2.15.0 is available for all the 
dependencies GeoTools/GeoServer uses, so I've tossed up PRs to update to 
Eclipse EMF versions 2.15.0:


https://github.com/geotools/geotools/pull/3072
https://github.com/geoserver/geoserver/pull/4419

I imagine if CI/CD builds, then we should be 'good to go'. Admittedly, I 
just tossed this together quickly, and I'm not an EMF expert.  My goal 
is to help keep dependencies lined up:).


Let me know if I can help with this one any more.

Cheers,

Jim

1. https://github.com/geotools/geotools/blob/master/pom.xml#L1291-L1310
2. 
https://github.com/geoserver/geoserver/blob/master/src/pom.xml#L1062-L1081




___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] JTS 1.17 upgrade planned for early July

[Jim Hughes]

Hi all,

Thanks to Jody for merging my PRs.  Since the PRs are coordinated, there 
may be downstream build failures until there are GeoTools and then GWC 
snapshots available on the snapshot repos.


Cheers,

Jim

On 7/3/2020 11:13 AM, Jim Hughes wrote:


Hi all,

As an update, Andrea suggested full builds of GeoTools (using 'mvn 
clean install -Dall') and GeoServer (using 'mvn clean install 
-Prelease').  I also did the suggested extra credit of building the 
GeoServer community modules (mvn clean install -PcommunityRelease 
-DskipTests in src/community); that passed as well.


Since the JTS upgrade will happen for GT/GWC/GS all together or not at 
all, I'm using the GT PR 
(https://github.com/geotools/geotools/pull/3034) to coordinate 
things.  I've marked it as a Draft to represent that it isn't quite 
ready to merge.


In particular, we are looking for updates from two projects:
Flatgeobuf: https://github.com/bjornharrtell/flatgeobuf and
java-vector-tile: 
https://github.com/ElectronicChartCentre/java-vector-tile

before we can see the full/release builds work.

Thanks to Daniele from GeoSolutions for turning around a very quick 
release of Jai-ext!


As a note, Jody and I released a new version of Jaitools (version 
1.6.0) to Maven central as part of this JTS upgrade.


Last time we did this, we also needed to upgrade Hatbox, Jeo, and 
GeoDB.  I haven't seen a need to do that yet.


Cheers,

Jim

On 7/2/2020 7:31 PM, Jody Garnett wrote:
The issue is here https://osgeo-org.atlassian.net/browse/GEOT-6637 
for GeoTools.

--
Jody Garnett


On Thu, 2 Jul 2020 at 16:14, Jim Hughes <mailto:jhug...@ccri.com>> wrote:


Hi all,

I'm helping with the JTS release and GT/GWC/GS upgrade.  Ian,
sorry that I didn't see this request until now.  We can
definitely create a ticket to track the update.  That said, JTS
and Jaitools were released to Maven central today.

As noted in the JTS release notes, there is a minor API issue
requiring recompilation.  As such, there is also a PR for JAI-EXT.

JAI-EXT PR: https://github.com/geosolutions-it/jai-ext/pull/269
GT PR: https://github.com/geotools/geotools/pull/3034
GWC PR: https://github.com/GeoWebCache/geowebcache/pull/868
GS PR: https://github.com/geoserver/geoserver/pull/4372

Due to the API change, a new JAI-EXT is required; that means that
the Travis builds all fail as you would expect them to.

There haven't been any real code changes, so if sorting out the
JAI-EXT artifacts take some time, that'll be fine.  After those
are available, the PRs will need to be updated to put to the
newly released JAI-EXT.

Feel free to leave comments on the PRs.  I haven't had time to go
through the checklists thoroughly, so feel free to wait until
later as well.  (I wanted to write sooner in case folks were
willing to churn out JAI-EXT releases quickly.  No rush though.)

Cheers,

Jim

On 6/24/2020 3:46 AM, Ian Turton wrote:

Can you create an issue for this - I have a couple of
enhancements waiting for the upgrade that it would be good to
link to the upgrade issue eg
https://osgeo-org.atlassian.net/browse/GEOT-6592

Cheers

Ian

On Tue, 23 Jun 2020 at 22:01, Jody Garnett
mailto:jody.garn...@gmail.com>> wrote:

As mentioned in today's meeting, a new release of JTS is
planned:
-
https://projects.eclipse.org/projects/locationtech.jts/releases/1.17.0

I would like to upgrade master when this published.
--
Jody Garnett
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
<mailto:GeoTools-Devel@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/geotools-devel



-- 
Ian Turton



___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net  
<mailto:GeoTools-Devel@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/geotools-devel

___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
<mailto:GeoTools-Devel@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/geotools-devel




___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] JTS 1.17 upgrade planned for early July

[Jim Hughes]

Hi all,

As an update, Andrea suggested full builds of GeoTools (using 'mvn clean 
install -Dall') and GeoServer (using 'mvn clean install -Prelease').  I 
also did the suggested extra credit of building the GeoServer community 
modules (mvn clean install -PcommunityRelease -DskipTests in 
src/community); that passed as well.


Since the JTS upgrade will happen for GT/GWC/GS all together or not at 
all, I'm using the GT PR 
(https://github.com/geotools/geotools/pull/3034) to coordinate things.  
I've marked it as a Draft to represent that it isn't quite ready to merge.


In particular, we are looking for updates from two projects:
Flatgeobuf: https://github.com/bjornharrtell/flatgeobuf and
java-vector-tile: https://github.com/ElectronicChartCentre/java-vector-tile
before we can see the full/release builds work.

Thanks to Daniele from GeoSolutions for turning around a very quick 
release of Jai-ext!


As a note, Jody and I released a new version of Jaitools (version 1.6.0) 
to Maven central as part of this JTS upgrade.


Last time we did this, we also needed to upgrade Hatbox, Jeo, and 
GeoDB.  I haven't seen a need to do that yet.


Cheers,

Jim

On 7/2/2020 7:31 PM, Jody Garnett wrote:
The issue is here https://osgeo-org.atlassian.net/browse/GEOT-6637 for 
GeoTools.

--
Jody Garnett


On Thu, 2 Jul 2020 at 16:14, Jim Hughes <mailto:jhug...@ccri.com>> wrote:


Hi all,

I'm helping with the JTS release and GT/GWC/GS upgrade. Ian, sorry
that I didn't see this request until now.  We can definitely
create a ticket to track the update.  That said, JTS and Jaitools
were released to Maven central today.

As noted in the JTS release notes, there is a minor API issue
requiring recompilation.  As such, there is also a PR for JAI-EXT.

JAI-EXT PR: https://github.com/geosolutions-it/jai-ext/pull/269
GT PR: https://github.com/geotools/geotools/pull/3034
GWC PR: https://github.com/GeoWebCache/geowebcache/pull/868
GS PR: https://github.com/geoserver/geoserver/pull/4372

Due to the API change, a new JAI-EXT is required; that means that
the Travis builds all fail as you would expect them to.

There haven't been any real code changes, so if sorting out the
JAI-EXT artifacts take some time, that'll be fine.  After those
are available, the PRs will need to be updated to put to the newly
released JAI-EXT.

Feel free to leave comments on the PRs.  I haven't had time to go
through the checklists thoroughly, so feel free to wait until
later as well.  (I wanted to write sooner in case folks were
willing to churn out JAI-EXT releases quickly.  No rush though.)

Cheers,

Jim

On 6/24/2020 3:46 AM, Ian Turton wrote:

Can you create an issue for this - I have a couple of
enhancements waiting for the upgrade that it would be good to
link to the upgrade issue eg
https://osgeo-org.atlassian.net/browse/GEOT-6592

Cheers

Ian

On Tue, 23 Jun 2020 at 22:01, Jody Garnett
mailto:jody.garn...@gmail.com>> wrote:

As mentioned in today's meeting, a new release of JTS is
planned:
-
https://projects.eclipse.org/projects/locationtech.jts/releases/1.17.0

I would like to upgrade master when this published.
--
Jody Garnett
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
<mailto:GeoTools-Devel@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/geotools-devel



-- 
Ian Turton



___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net  
<mailto:GeoTools-Devel@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/geotools-devel

___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
<mailto:GeoTools-Devel@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/geotools-devel

___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] JTS 1.17 upgrade planned for early July

[Jim Hughes]

Hi all,

I'm helping with the JTS release and GT/GWC/GS upgrade.  Ian, sorry that 
I didn't see this request until now.  We can definitely create a ticket 
to track the update.  That said, JTS and Jaitools were released to Maven 
central today.


As noted in the JTS release notes, there is a minor API issue requiring 
recompilation.  As such, there is also a PR for JAI-EXT.


JAI-EXT PR: https://github.com/geosolutions-it/jai-ext/pull/269
GT PR: https://github.com/geotools/geotools/pull/3034
GWC PR: https://github.com/GeoWebCache/geowebcache/pull/868
GS PR: https://github.com/geoserver/geoserver/pull/4372

Due to the API change, a new JAI-EXT is required; that means that the 
Travis builds all fail as you would expect them to.


There haven't been any real code changes, so if sorting out the JAI-EXT 
artifacts take some time, that'll be fine.  After those are available, 
the PRs will need to be updated to put to the newly released JAI-EXT.


Feel free to leave comments on the PRs.  I haven't had time to go 
through the checklists thoroughly, so feel free to wait until later as 
well.  (I wanted to write sooner in case folks were willing to churn out 
JAI-EXT releases quickly.  No rush though.)


Cheers,

Jim

On 6/24/2020 3:46 AM, Ian Turton wrote:
Can you create an issue for this - I have a couple of enhancements 
waiting for the upgrade that it would be good to link to the upgrade 
issue eg https://osgeo-org.atlassian.net/browse/GEOT-6592


Cheers

Ian

On Tue, 23 Jun 2020 at 22:01, Jody Garnett > wrote:


As mentioned in today's meeting, a new release of JTS is planned:
-
https://projects.eclipse.org/projects/locationtech.jts/releases/1.17.0

I would like to upgrade master when this published.
--
Jody Garnett
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net

https://lists.sourceforge.net/lists/listinfo/geotools-devel



--
Ian Turton


___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] FeatureReaderIterator's error handling

[Jim Hughes]

Hi Jody,

Yeah, this implementation is very locked down:).

We don't use the ContentDataStore API; GeoMesa isn't introducing this 
class into the mix.  GeoServer is doing the wrapping...  One of the 
classes designed to bring along FeatureTypeInfo uses the 
DefaultFeatureCollection and it delegates to this.  (If I recall 
correctly from debugging earlier.)


Thoughts?

Cheers,

Jim

On 5/8/2020 6:59 PM, Jody Garnett wrote:
That is tricky, since this wrapper is trying to mimic iterator 
behaviour. The intension was to wrap any IOExceptions in runtime 
exceptions (such as NoSuchElementException() ) in the event their are 
any IO problems, and help code that was previously based on in memory 
collections transition to streaming IO.


The FeatureReaderIterator method in question is here:

/    public boolean hasNext() {/
/        try {/
/            if (reader == null) return false;/
/            if (reader.hasNext()) {/
/                return true;/
/            } else {/
/                // auto close because we don't trust/
/                // client code to call closed :-)/
/                close();/
/                return false;/
/            }/
/        } catch (Exception e) {/
/            close();/
/            return false; // failure sounds like lack of next to me/
/        }/
/    }/


Q: How does it do if you modify the catch( Exception e) block above to 
throw a runtime exception. I can kind of see why the default 
implementation does not take this approach. I tried going back in git 
/ svn history but I cannot find an earlier implementation to use as a 
reference.


This implementation looks very locked down and I cannot see a way 
around with it in place. You would need to modify the above code 
(which I would support) or introduce your own wrapper (override 
ContentDataStore superclass).


--
Jody Garnett


On Fri, 8 May 2020 at 14:58, Jim Hughes <mailto:jhug...@ccri.com>> wrote:


Hi all,

I'm digging into error handling around timeouts for a GeoTools
DataStore
(the various GeoMesa DataStore implementations).  In GeoServer, the
GeoMesa FeatureReader ends up being wrapped by the GeoTools
FeatureReaderIterator.

This class swallows exceptions from close() and hasNext(). Are there
examples of getting Exceptions through this class?  I tried a few
things, and stuck on how pass information (such as an exception
indicating that a query had a problem on the remote server or that
the
query reached its timeout) through this class.

Thoughts?

Cheers,

Jim

1.

https://github.com/geotools/geotools/blob/master/modules/library/main/src/main/java/org/geotools/feature/FeatureReaderIterator.java



___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
<mailto:GeoTools-Devel@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/geotools-devel

___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

[Geotools-devel] FeatureReaderIterator's error handling

[Jim Hughes]

Hi all,

I'm digging into error handling around timeouts for a GeoTools DataStore 
(the various GeoMesa DataStore implementations).  In GeoServer, the 
GeoMesa FeatureReader ends up being wrapped by the GeoTools 
FeatureReaderIterator.


This class swallows exceptions from close() and hasNext().  Are there 
examples of getting Exceptions through this class?  I tried a few 
things, and stuck on how pass information (such as an exception 
indicating that a query had a problem on the remote server or that the 
query reached its timeout) through this class.


Thoughts?

Cheers,

Jim

1. 
https://github.com/geotools/geotools/blob/master/modules/library/main/src/main/java/org/geotools/feature/FeatureReaderIterator.java




___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] geotools-master-owasp and geotools-master-owasp-check builds disabled on jenkins

[Jim Hughes]

I wonder if there's a sane way to plan in quarterly or twice-a-year 
quick checks for these issues.  That'd let us address the issues without 
needing it to be someone's daily struggle.


Thoughts about an (optional) routine like that being 'added' to the 
six-month release cycle?


Cheers,

Jim

On 4/21/2020 12:33 PM, Andrea Aime wrote:
On Tue, Apr 21, 2020 at 6:21 PM Mark Prins > wrote:


I know some people are not too keen on these kind of notifications
/ noise.


He he... to draw a parallel, let's have a look at the QA checks.
QA stuff could be added in the PR machinery, so that if one breaks it,
they have to fix it before merging. Makes it easy and predictable, 
identifies who

has to work on it automatically.

Security advisories pop up any time, there is no one on the line to 
fix them,
and for a bunch of dependencies we have, fixes are a far cry away from 
just

changing a pom file. You had an easy run with the Oracle driver upgrade,
others are much more painful (international code sprint grade).
It's true that sometimes it's just a matter of updating a dependency, 
or marking

a false positive in the build.

So my observation is, we cannot find the time to look at the build 
right now, and
do the easy ones... and the build stays on red. Say someone makes an 
extra effort*

to care for the easy ones, and keep the build on green for a bit.
The first hard one to fix one is gonna peg the build on red... and we 
are back to square one.


Long story short, the issue IMHO is not having the build, it's finding the
manpower to go after the reports.

Cheers
Andrea

*: hint, not me

== GeoServer Professional Services from the experts! Visit 
http://goo.gl/it488V for more information. == Ing. Andrea Aime 
@geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 
55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: 
+39 339 8844549 http://www.geo-solutions.it 
http://twitter.com/geosolutions_it 
--- /Con 
riferimento alla normativa sul trattamento dei dati personali (Reg. UE 
2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si 
precisa che ogni circostanza inerente alla presente email (il suo 
contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è 
riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il 
messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra 
operazione è illecita. Le sarei comunque grato se potesse darmene 
notizia. This email is intended only for the person or entity to which 
it is addressed and may contain information that is privileged, 
confidential or otherwise protected from disclosure. We remind that - 
as provided by European Regulation 2016/679 “GDPR” - copying, 
dissemination or use of this e-mail or the information herein by 
anyone other than the intended recipient is prohibited. If you have 
received this email by mistake, please notify us immediately by 
telephone or e-mail./




___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] geotools-master-owasp and geotools-master-owasp-check builds disabled on jenkins

[Jim Hughes]

Ian,

Can you say more about the builds failing?

I'm mildly interested.  Occasionally, I end up needing to speak for all 
the dependencies of GeoMesa (which pulls in GeoTools and a large swath 
of Apache/Hadoop projects).  I'm motivated to avoid CVE issues as close 
to the 'source' as possible (managing dependencies later can work).


Cheers,

Jim

On 4/17/2020 10:45 AM, Ian Turton wrote:
Since they appear to never have passed or not passed for the last 8 
months I've disabled them from building every day, as the only effect 
was sending me an email every day to say they failed.


If anyone want's to fix them then feel free to go ahead and enable them

Ian

--
Ian Turton


___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] GeoTools 23-RC ready for testing - part II the return of testing

[Jim Hughes]

Hi all,

I wanted to follow up on this.  I am glad we were able to test the 
GeoTools RC and I think this dicussion has been useful.  At the minute, 
Emilio and I have ways to use either module to do what we want to in 
GeoMesa.  Given that, nothing here should hold up the release, etc.  
Since my PR isn't a great solution (and hence isn't reaching consensus), 
I'm fine closing or having someone else closing it.


Emilio and I had a chance to try out the gt-geojsonstore, and we have a 
few suggestions for improvements:


1.  On the write path, it is our opinion that if you pass in a 
writer/stream, then it shouldn't get closed.  Instead, it'd make sense 
for GeoJSONWriter to implement Flushable.


2.  In terms of dependencies, I'd like to suggest that the module use 
JTS's GeoJsonWriter[*] for writing out geometries instead of jts-jackson 
(which depends on JTS 1.13).


3.  On the read path, there's a tiny bug(**) which prevents the use of 
URLs.


Further, what all should happen around feature ID handling?  Even if the 
GeoJSON spec doesn't cover it, would it make sense for the Writer and 
Reader to handle feature IDs in a way so that they can be written and 
read back in?


Is there a good way to have a consistent FeatureType throughout the 
entire GeoJSONReader?  Having it change feature to feature is somewhat 
unexpected?


Anyhow, I'd love to see GeoTools have a nice SimpleFeatureCollection <-> 
GeoJson set of libraries.  Lemme know how I can help!  Just to ask, is 
the plan to clean up and promotoe the gt-geojsonstore to an extension?


Cheers,

Jim

* 
https://github.com/locationtech/jts/blob/master/modules/io/common/src/main/java/org/locationtech/jts/io/geojson/GeoJsonWriter.java


** 
https://github.com/geotools/geotools/blob/master/modules/unsupported/geojsonstore/src/main/java/org/geotools/data/geojson/GeoJSONDataStoreFactory.java#L147 
should be 'name' not 'file'.


On 3/18/2020 4:18 PM, Jody Garnett wrote:
Ian could we add a utility class to the gt-geojson jar for handling 
this case (parsing geojson in isolation)? The GeoJSONWriter could make 
us of the utility class to avoid duplication.

--
Jody Garnett


On Wed, 18 Mar 2020 at 10:42, Emilio Lahr-Vivaz > wrote:


Hi Ian,

Do you mean using the GeoJSONDataStore? I wasn't actually aware of
it until now, but I think it wouldn't work to use the feature
writer directly as that will write out to a file, correct? We
(GeoMesa) have a command line tool that abstracts around
OutputStream for exporting data in different formats (json, avro,
parquet, csv, etc) and to different places (stdout, file, hdfs, etc).

One small issue that I see is that the writer closes the
OutputStream:

https://github.com/geotools/geotools/blob/master/modules/unsupported/geojsonstore/src/main/java/org/geotools/data/geojson/GeoJSONWriter.java#L160

This makes it slightly tricky for us to use, as for some use cases
we keep writing to the output stream after writing out the geojson.

I've previously looked around for 'best practices' around when to
close an output stream, and haven't really found any. My thought
is that generally the code that creates the stream should be
responsible for closing it. Would you be open to me putting up a
PR to change the writer behavior in that regard?

Thanks,

Emilio

On 3/18/20 12:40 PM, Ian Turton wrote:

Emilio,

could you not just create a FeatureStore or FeatureWriter and
write features out that way as with any other datastore? But
anyway glad that it seems to work for you.

Ian

On Wed, 18 Mar 2020 at 14:00, Emilio Lahr-Vivaz
mailto:elahrvi...@ccri.com>> wrote:

Actually the GeoJSONWriter looks like it would work
perfectly. Our workflow is: start writing the export (i.e.
write the type 'FeatureCollection' and start the 'features'
array); write 0-n features as they come back from various
asynchronous queries; finish writing the export (i.e. write
the closing array bracket and close the FeatureCollection
element). The old gt-geojson module didn't support this pattern.

Thanks,

Emilio

On 3/18/20 9:10 AM, Ian Turton wrote:

I'm not quite sure what the use case for that would be, but
if you would like to propose modifications to GeoJSONWriter
in gt-geojsondatastore I'm happy to consider them.

Ian

On Wed, 18 Mar 2020 at 13:06, Emilio Lahr-Vivaz
mailto:elahrvi...@ccri.com>> wrote:

If that module is unsupported, are there any supported
methods for writing a feature as geojson? For our use
case, we need to write each feature individually, and
not as a feature collection.

Thanks,

Emilio

On 3/18/20 4:00 AM, Andrea Aime wrote:

On Wed, Mar 18, 2020 at 12:03 AM Jody Garnett

Re: [Geotools-devel] GeoTools 23-RC ready for testing - part II the return of testing

[Jim Hughes]

Hi Jody,

This is more or less the approach that I described above.  Since I was 
encouraged to toss up a PR for that approach, I put one together.  (By 
the way, it was a good exercise.  I really like the checkboxes and the 
new improvements to the CI hooks!  Those were awesome.  I missed the 
formatting on the first pass, but managed to address those issues easily 
enough.  All that was very, very easy to use...)


The API as-is allows for implementation details to leak and be 
confusing.  While I was putting up the PR, I checked to see if 
GeoWebCache or GeoServer use this capability and I don't think they do.  
Overall, if this doesn't get addressed, I have a suitable workaround.  
It may be worth it for all to move on.  (I did enjoy the new and 
improved PR process.  That was great!)


Cheers,

Jim

On 3/17/2020 7:02 PM, Jody Garnett wrote:

I am not quite sure what I am seeing in this PR.

I was going to ask you to add a note to the javadoc explaining the API 
contract (ie that we trust users to close the provided writer). But 
then I got confused, if I am reading the code correctly, the other 
writers returned (for example BufferedWriter) will close end up the 
underlying stream.


This results in an inconsistent API contract... which I am not wild about.

Since this is an unsupported module should we just break the existing 
API contract in order to be clear about expectations?

--
Jody Garnett


On Tue, 17 Mar 2020 at 15:33, Jim Hughes <mailto:jhug...@ccri.com>> wrote:


Hi all,

Here's a PR for the discussed approach:
https://github.com/geotools/geotools/pull/2839

I'm not wild about it, but it does fix the 'regression' / change I
saw.  I think the idea of the method GeoJSONUtil.toWriter(Object
output) is a bit too broad, and that means that any solution which
keeps the API in place will be a little silly looking.

If anyone doesn't like it, the identified approach can readily be
used downstream in GeoMesa.

Other than that, the GeoTools RC looks pretty reasonable.  I'm
hoping that we can kick the tires on the GeoServer RC, but that's
uncertain.

Cheers,

Jim

On 3/13/2020 3:34 AM, Andrea Aime wrote:

On Thu, Mar 12, 2020 at 10:11 PM Jim Hughes mailto:jhug...@ccri.com>> wrote:

Hi Andrea,

Ah!  The context that some of the Objects need closing and
others do not helps clarify things.

I just wrote a client side version of such a wrapper to see
if it'd work (it does).  Sounds like an amendment to
'toWriter' to have a no-op-close wrapper around the 1)
BufferedWriter, 2) Writer, and 3) OutputStream would fit the
bill?

If so, I can try to put something together.

Please go ahead!

Cheers
Andrea
==

GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information. == Ing. Andrea Aime
@geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A
55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272
mob: +39 339 8844549 http://www.geo-solutions.it
http://twitter.com/geosolutions_it
--- /Con
riferimento alla normativa sul trattamento dei dati personali
(Reg. UE 2016/679 - Regolamento generale sulla protezione dei
dati “GDPR”), si precisa che ogni circostanza inerente alla
presente email (il suo contenuto, gli eventuali allegati, etc.) è
un dato la cui conoscenza è riservata al/i solo/i destinatario/i
indicati dallo scrivente. Se il messaggio Le è giunto per errore,
è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le
sarei comunque grato se potesse darmene notizia. This email is
intended only for the person or entity to which it is addressed
and may contain information that is privileged, confidential or
otherwise protected from disclosure. We remind that - as provided
by European Regulation 2016/679 “GDPR” - copying, dissemination
or use of this e-mail or the information herein by anyone other
than the intended recipient is prohibited. If you have received
this email by mistake, please notify us immediately by telephone
or e-mail./

___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] GeoTools 23-RC ready for testing - part II the return of testing

[Jim Hughes]

Hi all,

Here's a PR for the discussed approach: 
https://github.com/geotools/geotools/pull/2839


I'm not wild about it, but it does fix the 'regression' / change I saw.  
I think the idea of the method GeoJSONUtil.toWriter(Object output) is a 
bit too broad, and that means that any solution which keeps the API in 
place will be a little silly looking.


If anyone doesn't like it, the identified approach can readily be used 
downstream in GeoMesa.


Other than that, the GeoTools RC looks pretty reasonable.  I'm hoping 
that we can kick the tires on the GeoServer RC, but that's uncertain.


Cheers,

Jim

On 3/13/2020 3:34 AM, Andrea Aime wrote:
On Thu, Mar 12, 2020 at 10:11 PM Jim Hughes <mailto:jhug...@ccri.com>> wrote:


Hi Andrea,

Ah!  The context that some of the Objects need closing and others
do not helps clarify things.

I just wrote a client side version of such a wrapper to see if
it'd work (it does).  Sounds like an amendment to 'toWriter' to
have a no-op-close wrapper around the 1) BufferedWriter, 2)
Writer, and 3) OutputStream would fit the bill?

If so, I can try to put something together.

Please go ahead!

Cheers
Andrea
==

GeoServer Professional Services from the experts! Visit 
http://goo.gl/it488V for more information. == Ing. Andrea Aime 
@geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 
55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: 
+39 339 8844549 http://www.geo-solutions.it 
http://twitter.com/geosolutions_it 
--- /Con 
riferimento alla normativa sul trattamento dei dati personali (Reg. UE 
2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si 
precisa che ogni circostanza inerente alla presente email (il suo 
contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è 
riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il 
messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra 
operazione è illecita. Le sarei comunque grato se potesse darmene 
notizia. This email is intended only for the person or entity to which 
it is addressed and may contain information that is privileged, 
confidential or otherwise protected from disclosure. We remind that - 
as provided by European Regulation 2016/679 “GDPR” - copying, 
dissemination or use of this e-mail or the information herein by 
anyone other than the intended recipient is prohibited. If you have 
received this email by mistake, please notify us immediately by 
telephone or e-mail./


___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] GeoTools 23-RC ready for testing - part II the return of testing

[Jim Hughes]

Hi Andrea,

Ah!  The context that some of the Objects need closing and others do not 
helps clarify things.


I just wrote a client side version of such a wrapper to see if it'd work 
(it does).  Sounds like an amendment to 'toWriter' to have a no-op-close 
wrapper around the 1) BufferedWriter, 2) Writer, and 3) OutputStream 
would fit the bill?


If so, I can try to put something together.

Cheers,

Jim

On 3/12/20 4:33 PM, Andrea Aime wrote:

Hi,
the API is problematic because it happens to create a Writer out of 
Object, using this method:

https://github.com/geotools/geotools/blob/master/modules/unsupported/geojson/src/main/java/org/geotools/geojson/GeoJSONUtil.java#L117

Some of the "things" passed in need closing (the FileWriter created 
out of file), others do not.
Maybe someone can write a Writer wrapper that does not delegate the 
close() method down,

and use it for the sources that happen to already be writer or a stream?

Cheers
Andrea



On Thu, Mar 12, 2020 at 8:47 PM Jim Hughes <mailto:jhug...@ccri.com>> wrote:


From API point-of-view, that makes sense.  From a code
point-of-view, there's a writer created on line 274:

https://github.com/geotools/geotools/blob/master/modules/unsupported/geojson/src/main/java/org/geotools/geojson/GeoJSONUtil.java#L274-L278...
that writer isn't returned, so as-is, the code maybe doing
something sensible.

Thoughts?


On 3/12/20 3:39 PM, Jody Garnett wrote:

Good timing on testing, it seems rough closing the writer after
one feature? I would update the javadoc to be clear about the API
contract and leave the writer open.

On Thu, Mar 12, 2020 at 11:50 AM Jim Hughes mailto:jhug...@ccri.com>> wrote:

Hi all,

As a happy coincidence, I was able to test the GeoTools RC
with GeoMesa.  I did hit an issue or two.

The first one is that GeoMesa uses GeoTools library code to
write out GeoJson.  Calls to
FeatureJSON.writeFeature(SimpleFeature feature, Object
output) use GeoJSONUtil.encode(String, Writer).

This encode method was changed[1] to use the
try-with-resources pattern, and that means the writer we pass
in gets closed.  This messes up that previous wrote out GeoJSON.

I see two options:
A.  This change is a 'bug' and we chat through a GeoTools
update.
B.  This change is an improvement, and I should update
GeoMesa.  (Maybe to use "new
FeatureEncoder(feature).toJSONString()" along with managing
the writes to outputstreams, etc.)

Thoughts?

Jim

1.

https://github.com/geotools/geotools/commit/8b5d6279474c9dc8962d279a884e508ab670ee3c#diff-208951a6b5feb5f96a890c79c9846389
2. Example GeoMesa code:

https://github.com/locationtech/geomesa/blob/master/geomesa-tools/src/main/scala/org/locationtech/geomesa/tools/export/formats/GeoJsonExporter.scala

On 3/12/20 12:53 AM, Jody Garnett wrote:

23-RC is deployed and on SF, will work on blog post while
waiting for future builds.
--
Jody Garnett


On Wed, 11 Mar 2020 at 07:25, Jody Garnett
mailto:jody.garn...@gmail.com>> wrote:

Thanks for letting me know, speed seems good from this
end. Thanks for building from the tag, that may have to
do for timezone testing.
I will start geowebcache release cycle at end-of-day.
--
Jody Garnett


On Wed, 11 Mar 2020 at 02:17, Andrea Aime
mailto:andrea.a...@geo-solutions.it>> wrote:

Hi Jody,
I'm having trouble connecting to anything that's in
the US, getting to the build server is very slow,
downloading the package times out instead.
I've tried building out of github checkouts (that
one works) but it still gets stuck trying to download
from the boundless repository...

Will try again later, hopefully it's a short lived
situation

Cheers
Andrea


On Wed, Mar 11, 2020 at 4:30 AM Jody Garnett
mailto:jody.garn...@gmail.com>> wrote:

Build is once again complete, please test
artifacts and report back:
-

https://build.geoserver.org/view/release/job/geotools-release/230/artifact/build/release/distribution/23-RC/

Andrea if testing is successful please proceed
with geowebcache, or I will catch it tomorrow
after work.
--
Jody Garnett
___
GeoTools-Devel mailing list
GeoTools-Devel@lists

Re: [Geotools-devel] GeoTools 23-RC ready for testing - part II the return of testing

[Jim Hughes]

I'm hitting issues with the looking up EPSG codes in with 
FilterFactory(2).bbox.  E.g. things like


ff.bbox("geom",49.0,79.0,51.0,80.0,"EPSG:4326")

Are throwing errors like:

java.lang.RuntimeException: Failed to setup bbox SRS
Caused by: org.opengis.referencing.FactoryException: Type "EPSG" is unknow in 
this context.
Caused by: java.text.ParseException: Type "EPSG" is unknow in this context.

I didn't mention it in the first email 'cause I'm having a little trouble 
debugging it and I'm trying to make sure I'm not missing something obvious like 
gt-epsg-* jars.

At this point, I've looked a little, and I'm not seeing anything obvious.  If 
folks have really quick suggestions, I'd appreciate it.  That said, I'll keep 
digging.

Cheers,

Jim


On 3/12/20 4:34 PM, Andrea Aime wrote:
On Thu, Mar 12, 2020 at 7:51 PM Jim Hughes <mailto:jhug...@ccri.com>> wrote:


Hi all,

As a happy coincidence, I was able to test the GeoTools RC with
GeoMesa.  I did hit an issue or two.


Hum... or two? What's the second?

Cheers
Andrea
==

GeoServer Professional Services from the experts! Visit 
http://goo.gl/it488V for more information. == Ing. Andrea Aime 
@geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 
55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: 
+39 339 8844549 http://www.geo-solutions.it 
http://twitter.com/geosolutions_it 
--- /Con 
riferimento alla normativa sul trattamento dei dati personali (Reg. UE 
2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si 
precisa che ogni circostanza inerente alla presente email (il suo 
contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è 
riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il 
messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra 
operazione è illecita. Le sarei comunque grato se potesse darmene 
notizia. This email is intended only for the person or entity to which 
it is addressed and may contain information that is privileged, 
confidential or otherwise protected from disclosure. We remind that - 
as provided by European Regulation 2016/679 “GDPR” - copying, 
dissemination or use of this e-mail or the information herein by 
anyone other than the intended recipient is prohibited. If you have 
received this email by mistake, please notify us immediately by 
telephone or e-mail./




___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] GeoTools 23-RC ready for testing - part II the return of testing

[Jim Hughes]

From API point-of-view, that makes sense.  From a code point-of-view, 
there's a writer created on line 274: 
https://github.com/geotools/geotools/blob/master/modules/unsupported/geojson/src/main/java/org/geotools/geojson/GeoJSONUtil.java#L274-L278... 
that writer isn't returned, so as-is, the code maybe doing something 
sensible.


Thoughts?


On 3/12/20 3:39 PM, Jody Garnett wrote:
Good timing on testing, it seems rough closing the writer after one 
feature? I would update the javadoc to be clear about the API contract 
and leave the writer open.


On Thu, Mar 12, 2020 at 11:50 AM Jim Hughes <mailto:jhug...@ccri.com>> wrote:


Hi all,

As a happy coincidence, I was able to test the GeoTools RC with
GeoMesa.  I did hit an issue or two.

The first one is that GeoMesa uses GeoTools library code to write
out GeoJson.  Calls to FeatureJSON.writeFeature(SimpleFeature
feature, Object output) use GeoJSONUtil.encode(String, Writer).

This encode method was changed[1] to use the try-with-resources
pattern, and that means the writer we pass in gets closed.  This
messes up that previous wrote out GeoJSON.

I see two options:
A.  This change is a 'bug' and we chat through a GeoTools update.
B.  This change is an improvement, and I should update GeoMesa. 
(Maybe to use "new FeatureEncoder(feature).toJSONString()" along
with managing the writes to outputstreams, etc.)

Thoughts?

Jim

1.

https://github.com/geotools/geotools/commit/8b5d6279474c9dc8962d279a884e508ab670ee3c#diff-208951a6b5feb5f96a890c79c9846389
2. Example GeoMesa code:

https://github.com/locationtech/geomesa/blob/master/geomesa-tools/src/main/scala/org/locationtech/geomesa/tools/export/formats/GeoJsonExporter.scala

On 3/12/20 12:53 AM, Jody Garnett wrote:

23-RC is deployed and on SF, will work on blog post while waiting
for future builds.
--
Jody Garnett


On Wed, 11 Mar 2020 at 07:25, Jody Garnett
mailto:jody.garn...@gmail.com>> wrote:

Thanks for letting me know, speed seems good from this end.
Thanks for building from the tag, that may have to do for
timezone testing.
I will start geowebcache release cycle at end-of-day.
--
Jody Garnett


On Wed, 11 Mar 2020 at 02:17, Andrea Aime
mailto:andrea.a...@geo-solutions.it>> wrote:

Hi Jody,
I'm having trouble connecting to anything that's in the
US, getting to the build server is very slow,
downloading the package times out instead.
I've tried building out of github checkouts (that one
works) but it still gets stuck trying to download
from the boundless repository...

Will try again later, hopefully it's a short lived situation

Cheers
Andrea


On Wed, Mar 11, 2020 at 4:30 AM Jody Garnett
mailto:jody.garn...@gmail.com>>
wrote:

Build is once again complete, please test artifacts
and report back:
-

https://build.geoserver.org/view/release/job/geotools-release/230/artifact/build/release/distribution/23-RC/

Andrea if testing is successful please proceed with
geowebcache, or I will catch it tomorrow after work.
--
Jody Garnett
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
<mailto:GeoTools-Devel@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/geotools-devel



-- 


Regards, Andrea Aime == GeoServer Professional Services
from the experts! Visit http://goo.gl/it488V for more
information. == Ing. Andrea Aime @geowolf Technical Lead
GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa

<https://www.google.com/maps/search/Via+di+Montramito+3%2FA%0D%0A55054++Massarosa?entry=gmail=g>
(LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob:
+39 339 8844549 http://www.geo-solutions.it
http://twitter.com/geosolutions_it
---
/Con riferimento alla normativa sul trattamento dei dati
personali (Reg. UE 2016/679 - Regolamento generale sulla
protezione dei dati “GDPR”), si precisa che ogni
circostanza inerente alla presente email (il suo
contenuto, gli eventuali allegati, etc.) è un dato la cui
conoscenza è riservata al/i solo/i destinatario/i
indicati dallo scrivente. Se il messaggio Le è giunto per
errore, è tenuta/o a cancellarlo, ogni altra operazione è
illecita. Le sarei comu

Re: [Geotools-devel] GeoTools 23-RC ready for testing - part II the return of testing

[Jim Hughes]

Hi all,

As a happy coincidence, I was able to test the GeoTools RC with 
GeoMesa.  I did hit an issue or two.


The first one is that GeoMesa uses GeoTools library code to write out 
GeoJson.  Calls to FeatureJSON.writeFeature(SimpleFeature feature, 
Object output) use GeoJSONUtil.encode(String, Writer).


This encode method was changed[1] to use the try-with-resources pattern, 
and that means the writer we pass in gets closed.  This messes up that 
previous wrote out GeoJSON.


I see two options:
A.  This change is a 'bug' and we chat through a GeoTools update.
B.  This change is an improvement, and I should update GeoMesa.  (Maybe 
to use "new FeatureEncoder(feature).toJSONString()" along with managing 
the writes to outputstreams, etc.)


Thoughts?

Jim

1. 
https://github.com/geotools/geotools/commit/8b5d6279474c9dc8962d279a884e508ab670ee3c#diff-208951a6b5feb5f96a890c79c9846389
2. Example GeoMesa code: 
https://github.com/locationtech/geomesa/blob/master/geomesa-tools/src/main/scala/org/locationtech/geomesa/tools/export/formats/GeoJsonExporter.scala


On 3/12/20 12:53 AM, Jody Garnett wrote:
23-RC is deployed and on SF, will work on blog post while waiting for 
future builds.

--
Jody Garnett


On Wed, 11 Mar 2020 at 07:25, Jody Garnett > wrote:


Thanks for letting me know, speed seems good from this end. Thanks
for building from the tag, that may have to do for timezone testing.
I will start geowebcache release cycle at end-of-day.
--
Jody Garnett


On Wed, 11 Mar 2020 at 02:17, Andrea Aime
mailto:andrea.a...@geo-solutions.it>> wrote:

Hi Jody,
I'm having trouble connecting to anything that's in the US,
getting to the build server is very slow,
downloading the package times out instead.
I've tried building out of github checkouts (that one works)
but it still gets stuck trying to download
from the boundless repository...

Will try again later, hopefully it's a short lived situation

Cheers
Andrea


On Wed, Mar 11, 2020 at 4:30 AM Jody Garnett
mailto:jody.garn...@gmail.com>> wrote:

Build is once again complete, please test artifacts and
report back:
-

https://build.geoserver.org/view/release/job/geotools-release/230/artifact/build/release/distribution/23-RC/

Andrea if testing is successful please proceed with
geowebcache, or I will catch it tomorrow after work.
--
Jody Garnett
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net

https://lists.sourceforge.net/lists/listinfo/geotools-devel



-- 


Regards, Andrea Aime == GeoServer Professional Services from
the experts! Visit http://goo.gl/it488V for more information.
== Ing. Andrea Aime @geowolf Technical Lead GeoSolutions
S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39
0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549
http://www.geo-solutions.it http://twitter.com/geosolutions_it
--- /Con
riferimento alla normativa sul trattamento dei dati personali
(Reg. UE 2016/679 - Regolamento generale sulla protezione dei
dati “GDPR”), si precisa che ogni circostanza inerente alla
presente email (il suo contenuto, gli eventuali allegati,
etc.) è un dato la cui conoscenza è riservata al/i solo/i
destinatario/i indicati dallo scrivente. Se il messaggio Le è
giunto per errore, è tenuta/o a cancellarlo, ogni altra
operazione è illecita. Le sarei comunque grato se potesse
darmene notizia. This email is intended only for the person or
entity to which it is addressed and may contain information
that is privileged, confidential or otherwise protected from
disclosure. We remind that - as provided by European
Regulation 2016/679 “GDPR” - copying, dissemination or use of
this e-mail or the information herein by anyone other than the
intended recipient is prohibited. If you have received this
email by mistake, please notify us immediately by telephone or
e-mail./



___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel



___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] [GSoC] Possibility of contributing to GeoTools project as a GSoC project

[Jim Hughes]

Hi Sameera,

Should some of these contributions be to JTS directly?  JTS already has 
an implementation of DelaunayTriangulation[1].  How is your work different?


It may be worth chatting with the JTS team on gitter here: 
https://gitter.im/locationtech/jts (or joining the JTS mailing list).


Cheers,

Jim

1. 
https://locationtech.github.io/jts/javadoc/org/locationtech/jts/triangulate/DelaunayTriangulationBuilder.html


On 3/1/20 5:49 PM, Sameera Kannangara wrote:

Hi GeoTools developers,

I'm Sameera Kannangara, a PhD student from the university of 
Melbourne. My PhD is on spatial data analysis of location based social 
network data. As part of my research work I have implemented Delaunay 
triangulation [1], Gabriel graph [2] and shortest-path graph [3]. Also 
I have introduced a new graph named stepping stone graph [4] and 
presented it at SIGSPATIAL 2018 conference. We got nominated for the 
best paper award and were invited to extend and publish the paper in 
journal of ACM Transactions on Spatial Algorithms and Systems [5]. 
Stepping stone graph has a wide variety of use cases such as inferring 
movement corridors, evaluating nearest neighbour queries, Inferring 
travel networks, most popular path finding, tour recommendation, 
trajectory clustering and group movement detection. For some of the 
use cases we need trajectory information which can be found from 
social media on top of spatial data.


As a part of my research project implementation, I have been a user of 
GeoTools project for over 3 years now. I have heavily used GeoTools UI 
and basic structures such as coordinates and spatial indexes in my 
calculations. I have crude implementations of S-hull algorithm for 
creating Delaunay triangulation, shortest path graph and stepping 
stone graph, which I would like to contribute to the GeoTools project. 
Shortest path graph is also useful when determining boundaries of 
imprecise regions given as a coordinate set. I would like to know the 
possibility of doing these contributions as a GSoC 2020 project. Thank 
you for your consideration and hope to hear from you soon.


Best wishes,
Sameera

[1] https://en.wikipedia.org/wiki/Delaunay_triangulation
[2] https://en.wikipedia.org/wiki/Gabriel_graph
[3] https://en.wikipedia.org/wiki/Shortest-path_graph
[4] https://dl.acm.org/doi/10.1145/3274895.3274913
[5] https://dl.acm.org/doi/abs/10.1145/3324883


___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel



___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] Commit message must be in the form "[GEOT-XYZW] Title of the Jira ticket"

[Jim Hughes]

Hi Jody,

For what it is worth, I do this on projects like GeoMesa with merge 
squashes both as I put a PR and when I merge a PR.  This means that the 
first line of commit will have the ticket number from the first commit.


Generally, I like linear histories with one commits solving one ticket!

Cheers,

Jim

On 1/9/20 3:53 PM, Jody Garnett wrote:
Okay, I will do my best to adjust my workflow; and use git interactive 
rebase when I fail :)


Any other feedback on the geotools templates?
--
Jody Garnett


On Thu, 9 Jan 2020 at 09:50, Nuno Oliveira 
> wrote:


I cannot stress enough how valuable this is:


When you have to clean up problems in code, being able to
annotate the code (git blame), find the origin,
track the original description and discussion, and the people
involved in it, is invaluable


The other day I had to investigate what look like a regression, a
git blame give me the relevant commit, but unfortunately the
commit didn't follow the recommended pattern ... and just by
looking at the code and commit message it was not clear what was
going on and why the code was changed.
I had to ping directly the developer and we spend some time
finding the original issue. The original issue give all the info
we need and clarified everything :)

On Thu, 2020-01-09 at 08:04 +0100, Andrea Aime wrote:

On Wed, Jan 8, 2020 at 8:08 PM Jody Garnett
mailto:jody.garn...@gmail.com>> wrote:

What is the advantage of this format? Does it come from an
automatic tool?



The commit message habit has been there for a while... the
automated tool is called Andrea, crushing several weekends
to look up tickets in Jira that have been left abandoned but
open, while the work is actually done.
You all have been relying on this automation for over 10 years
without much noticing I think, but has limited
the Jira tendency to work towards becoming a dumpster.

Generally speaking, you want to have the ticket number in all
your commit messages, for very basic reasons
(I'm actually surprised you work on projects where that does not
happen, must be a tracking nightmare to look
in their history...):


https://softwareengineering.stackexchange.com/questions/308084/is-there-any-downside-to-commit-messages-containing-the-ticket-number

When you have to clean up problems in code, being able to
annotate the code (git blame), find the origin,
track the original description and discussion, and the people
involved in it, is invaluable
While it does not always work, it often allows to figure out why
things were done that way,
instead of just having a shortsighted and preconceived personal
notion of what the code should be doing.

Regards
Andrea

== GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information. == Ing. Andrea Aime
@geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A
55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272
mob: +39 339 8844549 http://www.geo-solutions.it
http://twitter.com/geosolutions_it
--- /Con
riferimento alla normativa sul trattamento dei dati personali
(Reg. UE 2016/679 - Regolamento generale sulla protezione dei
dati “GDPR”), si precisa che ogni circostanza inerente alla
presente email (il suo contenuto, gli eventuali allegati, etc.) è
un dato la cui conoscenza è riservata al/i solo/i destinatario/i
indicati dallo scrivente. Se il messaggio Le è giunto per errore,
è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le
sarei comunque grato se potesse darmene notizia. This email is
intended only for the person or entity to which it is addressed
and may contain information that is privileged, confidential or
otherwise protected from disclosure. We remind that - as provided
by European Regulation 2016/679 “GDPR” - copying, dissemination
or use of this e-mail or the information herein by anyone other
than the intended recipient is prohibited. If you have received
this email by mistake, please notify us immediately by telephone
or e-mail./

___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net  

https://lists.sourceforge.net/lists/listinfo/geotools-devel


-- 
Regards,

Nuno Oliveira
==
GeoServer Professional Services from the
experts!
Visithttp://goo.gl/it488V  for more information.
==

Nuno Miguel Carvalho Oliveira
@nmcoliveira
Software Engineer

GeoSolutions S.A.S.
Via di Montramito 3/A
55054  Massarosa (LU)
Italy
phone: +39 0584 962313
fax:      +39 0584 1660272

http://www.geo-solutions.it

Re: [Geotools-devel] [Geoserver-devel] circle-ci cross project integration tests

[Jim Hughes]

Hi Andrea,

Congrats, this is awesome!  I'm glad you were able to clean up what I 
started at the OSGeo code sprint!


Cheers,

Jim

On 5/19/2019 5:18 PM, Andrea Aime wrote:

Hi,
following on Jim's work I've prepared a build that seems to be doing 
the trick, and building

in sequence:

  * GeoTools (without tests, other builds cover that)
  * GeoWebCache (with tests)
  * GeoServer (with tests and community modules compile only)

All build in sequence in the same container, thus using the same 
repository and, what's important,

the same snapshots produced by the GeoTools build.

Jim was right, making the silly thing build without breaking on build 
limits has been the challenging
point, took a number of experiments to find the right way to pass the 
JVM params and set them

to a working condition.
Build times are surprisingly not bad, around 20 minutes total, which 
is compatible with the other

Travis builds the PRs are triggering.

Here is the build file:
https://github.com/aaime/geotools/blob/circle/.circleci/config.yml

and branch:
https://github.com/aaime/geotools/tree/circle

A successful build looks as follows:

image.png

I'm sure some adjustments are going to be needed, and we won't be able 
to trust these builds always
(like if a PR is meant to introduce a breaking change, well, it will 
break this build for sure).
I can imagine maybe some conventions in the commit message referring 
to the repos
and branches for the corresponding gwc and gs to use but... I'll leave 
that as an exercise for those

interested :-p

If there are no objections I can make a PR to introduce this in 
GeoTools in the next few days.


Cheers
Andrea

PS: and yes, we'll need to setup a similar build for GeoWebCache too, 
to check eventual downstream

breakage in GeoServer.

== GeoServer Professional Services from the experts! Visit 
http://goo.gl/it488V for more information. == Ing. Andrea Aime 
@geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 
55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: 
+39 339 8844549 http://www.geo-solutions.it 
http://twitter.com/geosolutions_it 
--- /Con 
riferimento alla normativa sul trattamento dei dati personali (Reg. UE 
2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si 
precisa che ogni circostanza inerente alla presente email (il suo 
contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è 
riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il 
messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra 
operazione è illecita. Le sarei comunque grato se potesse darmene 
notizia. This email is intended only for the person or entity to which 
it is addressed and may contain information that is privileged, 
confidential or otherwise protected from disclosure. We remind that - 
as provided by European Regulation 2016/679 “GDPR” - copying, 
dissemination or use of this e-mail or the information herein by 
anyone other than the intended recipient is prohibited. If you have 
received this email by mistake, please notify us immediately by 
telephone or e-mail./




___
Geoserver-devel mailing list
geoserver-de...@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] Filter-CQL issues

[Jim Hughes]

Hi Jody, Devon,

Thanks for asking.  From chatting with Emilio, we don't think it'd 
affect anything in GeoMesa.


From a quick look at the ECQL reference, it looks like attributes with 
non-alphanumeric characters should be double-quoted: 
https://docs.geoserver.org/latest/en/user/filter/ecql_reference.html#attribute 
?


Does that help any?

Devon, what does your unit test do?  If it is calling CQL.toFilter and 
then CQL.toCQL and checking equality, that seems like an excellent unit 
test!


Cheers,

Jim


On 2/4/19 2:17 PM, Jody Garnett wrote:
I would check with the GeoMesa project as they heavily rely on toCQL 
to package up query strings. As long as James (CCed) knows when making 
to upgrade fixing the issue should be fine.


James are you aware of this issue? Or has it not come up in your 
testing...


On Mon, Feb 4, 2019 at 9:41 AM Devon Tucker > wrote:


Hi all,

Working with the Filter -> CQL code recently (CQL.toCQL) I noticed
that it is not producing correct CQL for attribute names. For
example the following test case fails:

https://gist.github.com/dvntucker/1d7f8c227eea44d9bd1563f0269e4b6a

There are a number of issues here, but the biggest one is the
forward-slash instead of a period, which is not correct by the CQL
spec. Also, this identifier does not need to be quoted and I don't
think quoting identifiers is in the CQL spec either.

It looks like this is mainly an artifact from the parsing phase,
where the periods are replaced with slashes to match XPath. I
believe this part is correct, but needs to be corrected when
producing CQL output. My only worry with changing this is
potentially people relying on this behavior. Looking through
GS/GT, it doesn't appear that this code path (Filter->CQL) is used
a whole lot, and in fact where it is seems very likely to fail
because of this (forward slash in identifier doesn't parse).

Anyone see an issue with fixing this behavior?

Cheers,
Devon
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net

https://lists.sourceforge.net/lists/listinfo/geotools-devel

--
--
Jody Garnett


___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel



___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] Filter function to extract bits from json

[Jim Hughes]

Hi Andrea,

We've done something similar in GeoMesa.  We needed to implement a 
JsonPathPropertyAccessor[1] and a JsonPathFilterFunction[2].  We used 
Jayway for that and maybe json-simple2 somewhere along the way.


The unit tests cover lots of little corner cases we've already fussed 
through.


Does any of this look useful?

Cheers,

Jim

1. 
https://github.com/locationtech/geomesa/blob/master/geomesa-features/geomesa-feature-kryo/src/main/scala/org/locationtech/geomesa/features/kryo/json/JsonPathPropertyAccessor.scala

https://github.com/locationtech/geomesa/blob/master/geomesa-features/geomesa-feature-kryo/src/test/scala/org/locationtech/geomesa/features/kryo/json/JsonPathPropertyAccessorTest.scala
2. 
https://github.com/locationtech/geomesa/blob/master/geomesa-features/geomesa-feature-kryo/src/main/scala/org/locationtech/geomesa/features/kryo/json/JsonPathFilterFunction.scala
https://github.com/locationtech/geomesa/blob/master/geomesa-features/geomesa-feature-kryo/src/test/scala/org/locationtech/geomesa/features/kryo/json/JsonPathFilterFunctionTest.scala 




On 1/23/19 4:29 PM, Torben Barsballe wrote:
For completeness, we actually have a few more JSON libraries in 
GeoTools/GeoServer in addition to what you listed:


  * Google GSON for MongoDB and ArcGIS REST (in GeoTools) and for WMS
TopoJSON format and the GeoGig community module (in GeoServer),
which is active.
  * Jettison 1.0.1 (via XStream) for the JSON portion of the REST API
(Only in GeoServer). While the project is active, we're stuck on a
10-year old version for backwards compatibility reasons.

GSON is primarily intended for transforming Java to and from JSON , 
but it does have minimal JsonPath 
 
support (With more robust integration added by the Jayway 
implementation that you already mentioned). There's also a seemingly 
abandoned fork 
 
that adds Json-Pointer support.
Overall, it seems like GSON doesn't quite have the capabilities you 
are looking for, but it is already a dependency of some GeoTools 
modules so maybe consider it?


More generally, I'd definitely say go for a JSON library we already 
depend upon (given that we have too many already), be it Jackson or 
something else.
I think it's reasonable to add Jackson to gt-main. Having a JSON 
library as part of a core module might encourage others to use it 
rather than adding their own (although I rather doubt it).


Torben





On Wed, Jan 23, 2019 at 8:24 AM Ian Turton > wrote:


I'd say go for Jackson as a main dependency and then someone (or
me) could try to bring the other JSON modules uptodate by using
it, I might even merge the two at the same time, but not until the
weekend :-)

Ian

On Wed, 23 Jan 2019 at 15:45, Andrea Aime
mailto:andrea.a...@geo-solutions.it>> wrote:

Hi all,
as you probably know GeoTools is now sometimes carrying around
JSON in feature attributes
(e.g., see JSON support in PostGIS data store).
The JSON is represented as a String.

Now, I'd like to add a filter function to parse and extract
specific properties out of the JSON string... which
requires new dependencies. And boy we have a mess of them in
the classpath already:

  * json-simple, used in geojson and mbstyles, but dead, last
release is 2012
  * json-lib, used in GeoServer for geojson generation, and
also dead, last release is 2010
  * jackson, used by Spring and directly by some GeoServer
community modules, with recent releases and latest commits
of only a few days ago


There are basically two languages to extract stuff from JSON,
json-path and json-pointer.
json-path is the one I would have preferred as it's more
powerful, but its situation library wise is not
exactly in great shape, the jayway implementation uses yet
another json java library, json-smart2, also dead.
There are a couple of other options for json-path, either dead
or with a bizzarre "do not evil" licence.

json-pointer is instead supported directly by jackson-core,
which is well alive, and comes with a 300kb jar
that has no external dependencies. This seems the best bet,
just one hesitation... should I add a jackson-core
to gt-main? On one side, it seems a large-ish dependency for
just a function, on the other side having a single
class new module also seems a bit overkill.

Opinions?

Cheers
Andrea

== GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information. == Ing. Andrea Aime
@geowolf Technical Lead GeoSolutions S.A.S. Via di 

[Geotools-devel] JTS upgrade and Jaitools+Jai-ext

[Jim Hughes]

Hi all,

Jody and I have been working on some of the background tasks to upgrade 
GeoTools/GWC/GeoServer to JTS 1.15.0 (or later).  We have a proposal 
here (https://github.com/geotools/geotools/wiki/Upgrade-to-JTS-1.15.0) 
to track progress.


At the last GT/GS meeting, it was noted that Jaitools might need to 
migrate into Jai-ext.  If we are indeed going to update the JTS version, 
both of these projects would need to be updated and released before 
GeoTools could be updated.


So, my question is:  What should the plan for Jaitools and Jai-Ext be?

* Should there be one 'last' Jaitools with JTS 1.15 released?

* Or should Jaitools be rolled into Jai-ext and then have the JTS 
version bumped for the newer, greater Jai-extensions project?


I'm asking 'cause I'm going to be vacation-volunteering for JTS+GeoTools 
work this Friday and all next week.  If we reach a consensus, I'd be 
happy to run down some of this background legwork.


Cheers,

Jim


--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] its 1.15.0 upgrade chat

[Jim Hughes]

Hi Jody,

To be honest, I'm not too worried about merging order.

As a bit more info, I ended up having forks of GeoDB, Jai-ext, and 
Jaitools in order to build GeoTools (and GeoWebCache and GeoServer).


Given that 1) I haven't pushed those projects to have a JTS 1.15.0 
release, 2) we don't have a GT/GS proposal to track the changes, 3) the 
changes can mostly be made with sed*, and 4) any changes which aren't 
find-and-replaceable might warrant a conversation, I'm happy to hold off 
a little bit.


All that said, what timeline would folks be up for the JTS upgrade?

Cheers,

Jim

*
git grep -l com.vividsolutions | grep pom.xml | xargs sed -i 
"s/com.vividsolutions/org.locationtech.jts/g"
git grep -l com.vividsolutions | xargs sed -i 
"s/com.vividsolutions/org.locationtech/g"


On 03/20/2018 12:02 PM, Jody Garnett wrote:
Jim you have a couple branches for JTS 1.15.0 upgrade, we are 
considering whitespace reformat.


Are you interested in submitting these before the whitespace?
--
Jody Garnett


--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot


___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel



--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] [Geoserver-devel] SourceForge exit strategy

[Jim Hughes]

I don't think there's a hard requirement that all dependencies have to 
be available on Maven central...


That said, it is a typical nice-to-have thing...

I'd say that having GT/GWC/GS jars on Maven central and having OSGeo, or 
interested companies host JAI on their Maven repos seems like a 
reasonable plan.


http://central.sonatype.org/pages/requirements.html

Basically, I'm suggesting Maven central for Jars, and then 
GitHub+SourceForge for wars/other binaries.


Cheers,

Jim

On 03/07/2018 10:38 AM, Andrea Aime wrote:
On Wed, Mar 7, 2018 at 4:34 PM, Jim Hughes <jn...@ccri.com 
<mailto:jn...@ccri.com>> wrote:


Hi all,

As a quick question, are there any impediments to publishing
GeoTools and GeoServer jars on Maven Central?


JAI, I think? :-)

Cheers
Andrea

==GeoServer Professional Services from the experts! Visit 
http://goo.gl/it488V for more information.==Ing. Andrea Aime 
@geowolfTechnical LeadGeoSolutions S.A.S.Via di Montramito 3/A55054 
 Massarosa (LU)phone: +39 0584 962313fax: +39 0584 1660272mob: +39 
 339 8844549http://www.geo-solutions.ithttp://twitter.com/geosolutions_it


AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o 
nel/i file/s allegato/i sono da considerarsi strettamente riservate. 
Il loro utilizzo è consentito esclusivamente al destinatario del 
messaggio, per le finalità indicate nel messaggio stesso. Qualora 
riceviate questo messaggio senza esserne il destinatario, Vi preghiamo 
cortesemente di darcene notizia via e-mail e di procedere alla 
distruzione del messaggio stesso, cancellandolo dal Vostro sistema. 
Conservare il messaggio stesso, divulgarlo anche in parte, 
distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità 
diverse, costituisce comportamento contrario ai principi dettati dal 
D.Lgs. 196/2003.


The information in this message and/or attachments, is intended solely 
for the attention and use of the named addressee(s) and may be 
confidential or proprietary in nature or covered by the provisions of 
privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New 
Data Protection Code).Any use not in accord with its purpose, any 
disclosure, reproduction, copying, distribution, or either 
dissemination, either whole or partial, is strictly forbidden except 
previous formal approval of the named addressee(s). If you are not the 
intended recipient, please contact immediately the sender by 
telephone, fax or e-mail and delete the information in this message 
that has been received in error. The sender does not give any warranty 
or accept liability as the content, accuracy or completeness of sent 
messages and accepts no responsibility  for changes made after they 
were sent or for other risks which arise as a result of e-mail 
transmission, viruses, etc.




--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] [Geoserver-devel] SourceForge exit strategy

[Jim Hughes]

Hi all,

As a quick question, are there any impediments to publishing GeoTools 
and GeoServer jars on Maven Central?


I believe the requirements could be met easily enough: 
http://central.sonatype.org/pages/requirements.html. (I've done this for 
LocationTech projects like JTS and GeoMesa, and I'd be happy to help.)


Cheers,

Jim

On 03/07/2018 10:06 AM, Nuno Oliveira wrote:

Hi all,

I agree with Andrea ... so far my experience with the OSGeo maven 
repository doesn't
give me too much confidence for such an important move and one trouble 
per 2.5 years

is not that bad indeed.

Regards,

Nuno Oliveira

On 03/07/2018 02:24 PM, Andrea Aime wrote:

Hi Jody,
I don't think it's a specific event, my colleagues not involved in 
the full stack (and thus not having maybe an up to date GeoTools 
checkout)

report repository slowness periodically (e.g., weekly or monthly).
If the artifactory on SAC would get separation from other downloads, 
that is, its own dedicated bandwidth, it may be an interesting option 
to look into.


Cheers
Andrea


On Wed, Mar 7, 2018 at 3:20 PM, Jody Garnett > wrote:


The recent qgis release hit the OSGeo download servers hard, and
our maven repo was affected. After all the fun with
repo.boundlessgeo.org  we probably
know a) enough to host artifactory on SAC b) not to try cloud
hosting of the same.
On Tue, Mar 6, 2018 at 11:43 PM Andrea Aime
> wrote:

Hi Torben,
I'm tentatively -1 on the idea but willing to discuss.

The proposal you're citing is 2.5 years old and the mailing
list trouble we have had in the past week is
been the only really serious issue we have had since, as far
as I remember.

Migrating the lists to OSGeo will generate a very significant
disturbance again, but one of our own making,
we'll probably lose all subscribers* and create two places
that people have to search into before posting a request.

I'd be willing to consider the idea again if we have another
SF issue in the short term, but besides that,
I believe we can tolerate one trouble in 2.5 years, I doubt
OSGeo would be able to significantly
outperform that.

Speaking of which, we keep on having problems with the OSGeo
hosted maven repository,
my colleagues report slowness and occasional inability to
connect, I normally dodge the problem
by building from sources my snapshot and using -nsu in all
builds. Now, I understand we have not
seen similar issues on the mailing list, but it gives the
impression SAC is a bit short handed
(looking at our community, everybody seems to be busy up to
their eyeballs too, so don't think we
can offer help there).

Cheers
Andrea

* even assuming SF will give us the list of subscribers,
which they might not due to privacy,
I don't think we can force subscribe anyone to a new list
server hosted by a different
organization... so we'd likely have to start from scratch


On Tue, Mar 6, 2018 at 10:32 PM, Torben Barsballe
> wrote:

*

SourceForge had outages / reduced service during the week
of the GeoTools 19-beta / GeoServer 2.13-beta release.


This caused:

 *

Delays on mailing list discussions and announcements
(including the release announcement),

 *

Loss of all download statistics for the release, so
we don't know how many people are downloading /
testing it.


Once again, this brings up the topic of whether we should
consider something a bit more reliable than sourceforge
for hosting mailing lists and release artifacts. There is
a preexisting migration proposal here:
*https://github.com/geotools/geotools/wiki/SourceForge-exit-strategy

*

**

*One suggestion was to start by migrating the GeoTools
project (As it is a lower-risk migration, given that most
people get there artifacts via maven), and use that to
determine feasibility of migrating the GeoServer project.*

**

*Any thoughts?*

**

*Torben*

*



--
Check out the vibrant tech community on one of the
world's most
engaging tech sites, Slashdot.org! 

Re: [Geotools-devel] Trouble using WFS-NG DataStore with https

[Jim Hughes]

On 12/15/2017 08:50 AM, Mark Prins wrote:


Broken isn't a very well defined state, it's quite a valid use case to 
have (valid ofcourse) urls pointing all over the place.


Fair enough.:)

wrt following redirects; as geotools is still on an old version of 
apache httpclient which doesn't provide this functionality (see eg. 
https://hc.apache.org/httpclient-legacy/redirects.html for a workaround)

I've started porting to the new version, but it will take a while.
Also there are some know issues and vulnerabilities with this apache 
lib when used in a TLS environment, see 
https://osgeo-org.atlassian.net/browse/GEOT-5859


As a note, the WFS-NG datastore is using GeoTools's own 
'SimpleHttpClient' (1) which is an implementation of this interface 
(2).  Perhaps part of the solution will be to have an Apache 
implementation of that interface and sub it in?


Thoughts?

Cheers,

Jim

1. 
https://github.com/geotools/geotools/blob/master/modules/library/main/src/main/java/org/geotools/data/ows/SimpleHttpClient.java
2. 
https://github.com/geotools/geotools/blob/master/modules/library/main/src/main/java/org/geotools/data/ows/HTTPClient.java 



Mark




--
Disclaimer;
This message is just a reflection of what I thought at the time of 
sending. The message may contain information that is not intended for 
you or that you don't understand.



--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot


___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel



--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] Trouble using WFS-NG DataStore with https

[Jim Hughes]

Hi all,

So I'm slow on the uptake tonight...  In the event that the remote 
server is a GeoServer, then the Global settings proxy base url 
(http://docs.geoserver.org/latest/en/user/configuration/globalsettings.html#proxy-base-url) 
can be set to sort this issue out.


I suppose it might still be worth considering if the GeoTools WFS-NG 
datastore should have a work-around for a 'broken' environment?


Cheers,

Jim

On 12/12/2017 11:52 PM, Jim Hughes wrote:

Hi all,

And to answer my own question...  the incorrect URL is coming from the 
GetCapabilities document.  Basically, I'm getting screwed by a proxy 
in the way.


One solution may be for the WFS-NG datastore to follow re-directs in 
the HTTP code.  Would that be sensible?


Other than that, this seems like a time when a GetCapabilities 
document should be re-written to be 'correct'.


Are there other approaches or ideas to consider?

Cheers,

Jim

On 12/12/2017 11:14 PM, Jim Hughes wrote:

Hi all,

Today I tried out the WFS-NG DataStore with a GeoServer configured 
with HTTPS, and I hit two issues.  I'll start with the one where I 
could use some help...


I passed a url of the form https://hostname/geoserver/..., and after 
connecting correctly (using a username and password), subsequent 
requests started going against http://hostname:80/geoserver/...


From some debugging, I haven't found where the URL is being broken 
down and reconstructed.  Can someone give me a hint where to look?  
With a little pointer, I can hopefully update some documentation or 
otherwise fix a bug with WFS-NG with HTTPS.


The second issue I ran into is environment specific.  The 
error/exception I saw in Java land was just a '403'.  Using " 
-Djavax.net.debug=all" helped debug the issue.


The proxy/configuration in front of GeoServer required that the HTTP 
agent is set.  This can be remedied easily with a call to 
System.setProperty("http.agent", "foo") and/or -Dhttp.agent=foo.


Cheers, and thanks in advance for any help,

Jim


-- 


Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel




-- 


Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel




--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] Trouble using WFS-NG DataStore with https

[Jim Hughes]

Hi all,

And to answer my own question...  the incorrect URL is coming from the 
GetCapabilities document.  Basically, I'm getting screwed by a proxy in 
the way.


One solution may be for the WFS-NG datastore to follow re-directs in the 
HTTP code.  Would that be sensible?


Other than that, this seems like a time when a GetCapabilities document 
should be re-written to be 'correct'.


Are there other approaches or ideas to consider?

Cheers,

Jim

On 12/12/2017 11:14 PM, Jim Hughes wrote:

Hi all,

Today I tried out the WFS-NG DataStore with a GeoServer configured 
with HTTPS, and I hit two issues.  I'll start with the one where I 
could use some help...


I passed a url of the form https://hostname/geoserver/..., and after 
connecting correctly (using a username and password), subsequent 
requests started going against http://hostname:80/geoserver/...


From some debugging, I haven't found where the URL is being broken 
down and reconstructed.  Can someone give me a hint where to look?  
With a little pointer, I can hopefully update some documentation or 
otherwise fix a bug with WFS-NG with HTTPS.


The second issue I ran into is environment specific.  The 
error/exception I saw in Java land was just a '403'.  Using " 
-Djavax.net.debug=all" helped debug the issue.


The proxy/configuration in front of GeoServer required that the HTTP 
agent is set.  This can be remedied easily with a call to 
System.setProperty("http.agent", "foo") and/or -Dhttp.agent=foo.


Cheers, and thanks in advance for any help,

Jim


-- 


Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel




--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

[Geotools-devel] Trouble using WFS-NG DataStore with https

[Jim Hughes]

Hi all,

Today I tried out the WFS-NG DataStore with a GeoServer configured with 
HTTPS, and I hit two issues.  I'll start with the one where I could use 
some help...


I passed a url of the form https://hostname/geoserver/..., and after 
connecting correctly (using a username and password), subsequent 
requests started going against http://hostname:80/geoserver/...


From some debugging, I haven't found where the URL is being broken down 
and reconstructed.  Can someone give me a hint where to look?  With a 
little pointer, I can hopefully update some documentation or otherwise 
fix a bug with WFS-NG with HTTPS.


The second issue I ran into is environment specific.  The 
error/exception I saw in Java land was just a '403'.  Using " 
-Djavax.net.debug=all" helped debug the issue.


The proxy/configuration in front of GeoServer required that the HTTP 
agent is set.  This can be remedied easily with a call to 
System.setProperty("http.agent", "foo") and/or -Dhttp.agent=foo.


Cheers, and thanks in advance for any help,

Jim


--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] Composing feature calcs in a composite collection

[Jim Hughes]

On 10/31/2017 01:47 PM, Jody Garnett wrote:
Opinion - great direction for visitor, think it would allow us to make 
better use of collections model as a result. The functionality is used 
for optimization in JDBC, but the ability to split/merge feature 
visitors (including their result) would be advantageous for concurrency.


I remember exploring some of these design options when we looked at 
the feature collection api; and there were examples from other 
languages being considered.


Q: Now that we have a streaming API in Java does it offer any 
inspiration or functionality that we could take advantage of?


--
Jody Garnett

On 31 October 2017 at 09:16, Andrea Aime > wrote:


Hi,
I'm looking at a possible optimization for a composite feature
collection, that is,
the ability to optimize down feature visitors instead of running
them in memory.

So the code gets a visitor, and I was thinking, ok, if it's a
FeatureCalc


 there
is
a way to get the CalcResult(s) and merge them.

However, the collection is given a FeatureCalc that might have
aggregated
something already in memory, that needs to be preserved, and at
the end
I need to set the "merge" result back into it.

I looked into what JDBCDataStore can do, and it's basically using
reflection
to do its own business, since FeatureCalc per se provides no much
methods.

Thinking out loud maybe a "MutableFeatureCalc" is needed?

interface MutableFeatureCalc extends FeatureCalc {

    // available in many implementations of FeatureCalc, but not all!
    public void reset();

    // this is what JDBCDataStore is relying upon
    public void setValue(Object result);

}

Then aggregating from N sources in a generic way would look
something like:

CalcResult result = calc.getResult();
for(FeatureCollection fc : collections) {
    calc.reset();
    // need a wrapping sub progress listener here
    fc.accept(visitor, progressListener);
    CalcResult newResult = calc.getResult();
    result = result.merge(newResult);
}

calc.setValue(result.getValue());


Opinions?

Cheers
Andrea

PS: no promises it will be implemented, I'm in exploration and
feasibility mode here

==GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.==Ing. Andrea Aime
@geowolfTechnical LeadGeoSolutions S.A.S.Via di Montramito
3/A55054  Massarosa (LU)phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39  339 8844549

http://www.geo-solutions.ithttp://twitter.com/geosolutions_it


AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica
e/o nel/i file/s allegato/i sono da considerarsi strettamente
riservate. Il loro utilizzo è consentito esclusivamente al
destinatario del messaggio, per le finalità indicate nel messaggio
stesso. Qualora riceviate questo messaggio senza esserne il
destinatario, Vi preghiamo cortesemente di darcene notizia via
e-mail e di procedere alla distruzione del messaggio stesso,
cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti,
copiarlo, od utilizzarlo per finalità diverse, costituisce
comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended
solely for the attention and use of the named addressee(s) and may
be confidential or proprietary in nature or covered by the
provisions of privacy act (Legislative Decree June, 30 2003,
no.196 - Italy's New Data Protection Code).Any use not in accord
with its purpose, any disclosure, reproduction, copying,
distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please
contact immediately the sender by telephone, fax or e-mail and
delete the information in this message that has been received in
error. The sender does not give any warranty or accept liability
as the content, accuracy or completeness of sent messages and
accepts no responsibility  for changes made after they were sent
or for other risks which arise as a result of e-mail transmission,
viruses, etc.



--
Check out the vibrant tech community on one of the world's most

Re: [Geotools-devel] JTS Use of Oracle read/write code

[Jim Hughes]

No worries!  Thanks for the letter!

Jim

On 10/24/2017 05:07 PM, Jody Garnett wrote:
Sorry I have not kept up with meeting actions - I obtained permission 
to write a letter but did not yet do so:


 *

Jody: to write a wiki page letter for the requested JTS classes


Here is resulting wiki page: 
https://github.com/geotools/geotools/wiki/JTS-ORA-Contribution

--
Jody Garnett

On 24 October 2017 at 13:19, Jim Hughes <jn...@ccri.com 
<mailto:jn...@ccri.com>> wrote:


Hi all,

To close the loop on this, did a letter get drafted and/or
approved for the Ora{Reader,Writer} bits?

I see the letter for Shapefile code here:
https://github.com/geotools/geotools/wiki/JTS-Shapefile-Contribution
<https://github.com/geotools/geotools/wiki/JTS-Shapefile-Contribution>.
I see notes about it in the 09-05 minutes, and no mention in the
09-19 minutes.

Cheers,

Jim


On 08/20/2017 09:27 AM, Jody Garnett wrote:

Let's write up a letter like last time and approve it Tuesday
meeting.

For context this oracle work was done by me at Refractions for
the same customer who later paid for vivid solutions to continue
work on it in JTS. The two implementations have diverged to fix
issues and support curves (so we cannot at present combine
implementations). The test cases for JTS SDO are very nicely done
with a literate STRUCT factory used to define test cases.
On Sun, Aug 20, 2017 at 5:52 AM Andrea Aime
<andrea.a...@geo-solutions.it
<mailto:andrea.a...@geo-solutions.it>> wrote:

+1

Cheers
Andrea

On Sat, Aug 19, 2017 at 4:41 PM, Jim Hughes <jn...@ccri.com
<mailto:jn...@ccri.com>> wrote:

Hi all,

As a continuation of relicensing JTS we are hunting down
where external contributions came from.

For reading and writing with Oracle, JTS has based some
of its code on GeoTools. Examples of JTS code are below.

Would the GeoTools PSC approve give permission for JTS to
relicense the code which went into the files below?

Thanks in advance,

Jim


https://github.com/locationtech/jts/blob/master/modules/io/ora/src/main/java/org/locationtech/jts/io/oracle/OraReader.java

<https://github.com/locationtech/jts/blob/master/modules/io/ora/src/main/java/org/locationtech/jts/io/oracle/OraReader.java>

https://github.com/locationtech/jts/blob/master/modules/io/ora/src/main/java/org/locationtech/jts/io/oracle/OraWriter.java

<https://github.com/locationtech/jts/blob/master/modules/io/ora/src/main/java/org/locationtech/jts/io/oracle/OraWriter.java>

https://github.com/locationtech/jts/blob/master/modules/io/ora/src/test/java/org/locationtech/jts/io/oracle/OraReaderCreateTest.java

<https://github.com/locationtech/jts/blob/master/modules/io/ora/src/test/java/org/locationtech/jts/io/oracle/OraReaderCreateTest.java>

https://github.com/locationtech/jts/blob/master/modules/io/ora/src/test/java/org/locationtech/jts/io/oracle/OraWriterCreateTest.java

<https://github.com/locationtech/jts/blob/master/modules/io/ora/src/test/java/org/locationtech/jts/io/oracle/OraWriterCreateTest.java>



--
Check out the vibrant tech community on one of the
world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
<mailto:GeoTools-Devel@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/geotools-devel
<https://lists.sourceforge.net/lists/listinfo/geotools-devel>




-- 


Regards,

Andrea Aime

==GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.==Ing. Andrea Aime

<https://maps.google.com/?q=Via+di+Montramito+3/A+55054+%C2%A0Massarosa=gmail=g>@geowolfTechnical
LeadGeoSolutions S.A.S.Via di Montramito 3/A

<https://maps.google.com/?q=Via+di+Montramito+3/A+55054+%C2%A0Massarosa=gmail=g>55054
 Massarosa

<https://maps.google.com/?q=Via+di+Montramito+3/A+55054+%C2%A0Massarosa=gmail=g>
(LU)phone: +39 0584 962313 <tel:+39%200584%20962313>fax: +39
0584 1660272 <tel:+39%200584%20166%200272>mob: +39  339
8844549

<tel:+39%20339%20884%204549>http://www.geo-solutions.ithttp://twitter.com/geosolutions_it
<http://twitter.com/geosolutions_it>

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messagg

Re: [Geotools-devel] JTS Use of Oracle read/write code

[Jim Hughes]

Hi all,

To close the loop on this, did a letter get drafted and/or approved for 
the Ora{Reader,Writer} bits?


I see the letter for Shapefile code here: 
https://github.com/geotools/geotools/wiki/JTS-Shapefile-Contribution. I 
see notes about it in the 09-05 minutes, and no mention in the 09-19 
minutes.


Cheers,

Jim

On 08/20/2017 09:27 AM, Jody Garnett wrote:

Let's write up a letter like last time and approve it Tuesday meeting.

For context this oracle work was done by me at Refractions for the 
same customer who later paid for vivid solutions to continue work on 
it in JTS. The two implementations have diverged to fix issues and 
support curves (so we cannot at present combine implementations). The 
test cases for JTS SDO are very nicely done with a literate STRUCT 
factory used to define test cases.
On Sun, Aug 20, 2017 at 5:52 AM Andrea Aime 
<andrea.a...@geo-solutions.it <mailto:andrea.a...@geo-solutions.it>> 
wrote:


+1

Cheers
Andrea

On Sat, Aug 19, 2017 at 4:41 PM, Jim Hughes <jn...@ccri.com
<mailto:jn...@ccri.com>> wrote:

Hi all,

As a continuation of relicensing JTS we are hunting down where
external contributions came from.

For reading and writing with Oracle, JTS has based some of its
code on GeoTools.  Examples of JTS code are below.

Would the GeoTools PSC approve give permission for JTS to
relicense the code which went into the files below?

Thanks in advance,

Jim


https://github.com/locationtech/jts/blob/master/modules/io/ora/src/main/java/org/locationtech/jts/io/oracle/OraReader.java

https://github.com/locationtech/jts/blob/master/modules/io/ora/src/main/java/org/locationtech/jts/io/oracle/OraWriter.java

https://github.com/locationtech/jts/blob/master/modules/io/ora/src/test/java/org/locationtech/jts/io/oracle/OraReaderCreateTest.java

https://github.com/locationtech/jts/blob/master/modules/io/ora/src/test/java/org/locationtech/jts/io/oracle/OraWriterCreateTest.java



--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
<mailto:GeoTools-Devel@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/geotools-devel




-- 


Regards,

Andrea Aime

==GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.==Ing. Andrea Aime
@geowolfTechnical LeadGeoSolutions S.A.S.Via di Montramito
3/A55054  Massarosa (LU)phone: +39 0584 962313fax: +39 0584
1660272mob: +39  339
8844549http://www.geo-solutions.ithttp://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica
e/o nel/i file/s allegato/i sono da considerarsi strettamente
riservate. Il loro utilizzo è consentito esclusivamente al
destinatario del messaggio, per le finalità indicate nel messaggio
stesso. Qualora riceviate questo messaggio senza esserne il
destinatario, Vi preghiamo cortesemente di darcene notizia via
e-mail e di procedere alla distruzione del messaggio stesso,
cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti,
copiarlo, od utilizzarlo per finalità diverse, costituisce
comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended
solely for the attention and use of the named addressee(s) and may
be confidential or proprietary in nature or covered by the
provisions of privacy act (Legislative Decree June, 30 2003,
no.196 - Italy's New Data Protection Code).Any use not in accord
with its purpose, any disclosure, reproduction, copying,
distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please
contact immediately the sender by telephone, fax or e-mail and
delete the information in this message that has been received in
error. The sender does not give any warranty or accept liability
as the content, accuracy or completeness of sent messages and
accepts no responsibility  for changes made after they were sent
or for other risks which arise as a result of e-mail transmission,
viruses, etc.


--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Sla

Re: [Geotools-devel] Geotools Inclusion of locationtech JTS instead of vividsolutions

[Jim Hughes]

Hi Bret, Ian,

The JTS team is close to having a LocationTech approved release. (I'm 
helping with some of the IP/Eclipse-related bits which the release 
depends on.)


If there a bits you need in JTS, feel free to discuss them on the JTS 
mailing list (https://dev.locationtech.org/mailman/listinfo/jts-dev) or 
raise/comment on an appropriate issue here 
(https://github.com/locationtech/jts/issues).


As an additional note, Jody has worked to upgrade the version of JTS 
from 1.13 to 1.14.0.  I don't know if that helps with the Z/ZM geometries.


Cheers,

Jim

On 08/31/2017 03:48 PM, Ian Turton wrote:
Probably after there is a location tech release - it really depends on 
when Martin gets a chance to make it.


There is nothing to stop you giving us a jump on that release by 
making a PR against a snapshot to make use of Z and ZM geometries in 
the mean time though,


Cheers

Ian

On 31 August 2017 at 17:04, Bret Webb > wrote:


Can someone please tell me if Geotools plans on incorporating
locationtechs version of JTS into the codebase instead of
vividsolutions? My understanding is that vividsolutions is not
being actively maintained anymore, and does not support Z and ZM
geometries.  It appears that locationtech may have taken on this
task? and seems to be getting more attention. If so, is there an
ETA on when this might happen? Post Vegas?



--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net

https://lists.sourceforge.net/lists/listinfo/geotools-devel





--
Ian Turton


--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot


___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel



--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

[Geotools-devel] JTS Use of Oracle read/write code

[Jim Hughes]

Hi all,

As a continuation of relicensing JTS we are hunting down where external 
contributions came from.


For reading and writing with Oracle, JTS has based some of its code on 
GeoTools.  Examples of JTS code are below.


Would the GeoTools PSC approve give permission for JTS to relicense the 
code which went into the files below?


Thanks in advance,

Jim

https://github.com/locationtech/jts/blob/master/modules/io/ora/src/main/java/org/locationtech/jts/io/oracle/OraReader.java
https://github.com/locationtech/jts/blob/master/modules/io/ora/src/main/java/org/locationtech/jts/io/oracle/OraWriter.java
https://github.com/locationtech/jts/blob/master/modules/io/ora/src/test/java/org/locationtech/jts/io/oracle/OraReaderCreateTest.java
https://github.com/locationtech/jts/blob/master/modules/io/ora/src/test/java/org/locationtech/jts/io/oracle/OraWriterCreateTest.java


--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] Public domain contribution

[Jim Hughes]

Ah, so you are using the ElasticSearch REST client then?

Sounds like a reasonable solution.

On 06/13/2017 05:56 PM, sjudeng wrote:
Hi Jim, In the last ElasticGeo release I removed the 
org.elasticsearch:elasticsearch dependency and as a result freed it 
from the transitive Guava dependency. Guava is still used in the 
project but I backed it down to version 17.0 for consistency with 
GeoServer (see POM 
).


On Fri, Jun 2, 2017 at 4:42 PM, sjudeng > wrote:


Jody,

Thanks for your quick response. It does sound like a pain. I'll
look into the CLA.

On Fri, Jun 2, 2017 at 4:24 PM, Jody Garnett
> wrote:

You can see the language used here -
http://docs.geotools.org/latest/userguide/welcome/license.html


Indeed we tried to make our demo code public domain, but
because that was not a license it made it even harder for
people to use. We now have our demo code as BSD.

--
Jody Garnett

On 2 June 2017 at 14:22, Jody Garnett > wrote:

It has been done before in the geotools library. Mostly
for us government employees who's work is in the public
domain (since it was paid for by the people).

Please understand that public domain is an alternative to
copyright - not an alternative to having a license. Indeed
copyright is the stick we use to enforce our open source
license.

I cannot remember the details now but you basically put
your contribution into the public domain and we pick it up
(changing the packages and so on becoming contributors and
donating it to the geotools project).

So this is really not a lot of fun; do you have a legal
requirement to use the public domain? Or are you just
looking for an alternative to your employer signing a code
contribution agreement?


--
Jody Garnett

On 2 June 2017 at 14:18, sjudeng > wrote:

Is it possible to make a public domain contribution to
GeoTools and GeoServer without a CLA? I'm interested
in contributing an Elasticsearch plugin
(https://github.com/ngageoint/elasticgeo
), which is
licensed based on the GeoTools (LGPL) and GeoServer
(GPL) components. My contributions to the plugin are
public domain. I can reach out to the list again in a
separate thread on whether this would be welcome and
what would be required from a technical perspective
but wanted to start with this question first.


--
Check out the vibrant tech community on one of the
world's most
engaging tech sites, Slashdot.org!
http://sdm.link/slashdot
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net

https://lists.sourceforge.net/lists/listinfo/geotools-devel








--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot


___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel



--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] Public domain contribution

[Jim Hughes]

Hi all,

Separate from the licensing, I believe that ElasticSearch uses Guava 
18.  GeoTools and GeoServer uses version 17.


Does ElasticGeo work with Guava 17?

Cheers,

Jim

On 06/02/2017 05:42 PM, sjudeng wrote:

Jody,

Thanks for your quick response. It does sound like a pain. I'll look 
into the CLA.


On Fri, Jun 2, 2017 at 4:24 PM, Jody Garnett > wrote:


You can see the language used here -
http://docs.geotools.org/latest/userguide/welcome/license.html


Indeed we tried to make our demo code public domain, but because
that was not a license it made it even harder for people to use.
We now have our demo code as BSD.

--
Jody Garnett

On 2 June 2017 at 14:22, Jody Garnett > wrote:

It has been done before in the geotools library. Mostly for us
government employees who's work is in the public domain (since
it was paid for by the people).

Please understand that public domain is an alternative to
copyright - not an alternative to having a license. Indeed
copyright is the stick we use to enforce our open source license.

I cannot remember the details now but you basically put your
contribution into the public domain and we pick it up
(changing the packages and so on becoming contributors and
donating it to the geotools project).

So this is really not a lot of fun; do you have a legal
requirement to use the public domain? Or are you just looking
for an alternative to your employer signing a code
contribution agreement?


--
Jody Garnett

On 2 June 2017 at 14:18, sjudeng > wrote:

Is it possible to make a public domain contribution to
GeoTools and GeoServer without a CLA? I'm interested in
contributing an Elasticsearch plugin
(https://github.com/ngageoint/elasticgeo
), which is
licensed based on the GeoTools (LGPL) and GeoServer (GPL)
components. My contributions to the plugin are public
domain. I can reach out to the list again in a separate
thread on whether this would be welcome and what would be
required from a technical perspective but wanted to start
with this question first.


--
Check out the vibrant tech community on one of the world's
most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net

https://lists.sourceforge.net/lists/listinfo/geotools-devel







--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot


___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel



--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] geotools process raster - JAI native dependency problems in fat jar

[Jim Hughes]

Hi Georg,

I don't have a complete answer for you, but you might check to see what 
the META-INF/services directory in your fat jar has.  From a quick read 
of the Java docs for ParameterBlockJAI, it looks like a file like this 
(1) may be important.


If that's the case, then you'll just need to sort out having SBT 
concatenate or preserve such SPI registry files during fat jar creation.


Cheers,

Jim

1. 
https://github.com/geotools/geotools/blob/master/modules/unsupported/process-raster/src/main/resources/META-INF/services/javax.media.jai.OperationsRegistrySpi


On 6/4/2017 5:38 AM, Georg Heiler wrote:

Hi,

I want to use a library (JAI) with spark to parse some spatial raster 
files. Unfortunately, there are some strange issues. JAI only works 
when running via the build tool i.e. `sbt run` when executed in spark.

When executed via spark-submit the error is:

java.lang.IllegalArgumentException: The input argument(s) may not 
be null.
at 
javax.media.jai.ParameterBlockJAI.getDefaultMode(ParameterBlockJAI.java:136)

at javax.media.jai.ParameterBlockJAI.(ParameterBlockJAI.java:157)
at javax.media.jai.ParameterBlockJAI.(ParameterBlockJAI.java:178)
at 
org.geotools.process.raster.PolygonExtractionProcess.execute(PolygonExtractionProcess.java:171)


Which looks like some native dependency is not there correctly.
Assuming something is wrong with the class path I tried to run a plain 
java/scala function. but this one works just fine.
Is spark messing with the class paths? Actually when trying to run the 
jar in NiFi, I see the same problem.


I created a minimal example here:
https://github.com/geoHeil/jai-packaging-problem 



Are you aware of any other library which can read ESRi ASCII Grid 
files and polygonize them?


I think so far the problem could be tracked down to the instantiation 
of  new ParameterBlockJAI("Vectorize") which is returning null 
i.e. the JAI.registry not finding the desired operation but returning 
a different error when the one you usually get.


Regards,
Georg


--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot


___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel



--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] Community Module Proposal: s3-geotiff

[Jim Hughes]

Hi Devon,

Big +1 for this module.

The module for S3 will be useful right now.  Generally, I'd note that 
there's a logical extension to move this module to use the Hadoop 
FileSystem abstraction.  With that kind of move, you'd be able to pick 
up support for whatever someone implements support for (1).  Once this 
happens, the module would support additional cloud vendors like Azure.


Also, code like (2) would become more general and less brittle. That 
code is forcing the system to use s3:// urls rather than s3a or s3n.  
(There are implementation differences between the protocols.)


Lastly, another +1 on the caching layer.  As an extension, I'd love to 
see the caching layer separated from the remote/distributed file system 
so that the two can be configured separately.


Cheers,

Jim

1. https://en.wikipedia.org/wiki/Apache_Hadoop#Other_file_systems
https://wiki.apache.org/hadoop/HCFS

2. 
https://github.com/dvntucker/geotools/blob/s3-geotiff-module/modules/unsupported/s3-geotiff/src/main/java/org/geotools/s3/geotiff/S3GeoTiffFormat.java#L105-L120


On 05/23/2017 12:56 PM, Devon Tucker wrote:

Hi all,

I mentioned this in a previous email either here or on gs-devel, but 
we have a simple-ish module we'd like to make into 
community/unsupported modules for GeoTools and GeoServer. They enable 
GeoTiffs to be served via Amazon S3 directly.


The modules are fairly simple. In GeoTools there is a 
GridFormatFactory interface that's largely derivative of the GeoTiff 
module, as well as an ImageInputStream implementation that does most 
of the heavy lifting. This is also tied into EhCache caching in order 
to improve performance.


In GeoServer there is a very trivial store UI screen. This exists only 
to get around some of the limitations currently in GeoServer when it 
comes to URL validation in the default grid store screens.


I have branches created for each of these in my fork:

https://github.com/dvntucker/geotools/tree/s3-geotiff-module/modules/unsupported/s3-geotiff
https://github.com/dvntucker/geoserver/tree/s3-geotiff-gs/src/community/s3-geotiff

Take a look if you like and let me know what you think.

Cheers,
Devon


--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot


___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel



--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] [Geoserver-devel] proposed change: coverage readers custom dimension data type conversion

[Jim Hughes]

Hi all,

Sorry I missed this until now.  I'm not following too closely, but it 
sounds like this is an improvement.  If it requires a slight update to 
any Coverage Readers, we can sort things out.


Cheers,

Jim

On 03/22/2017 09:56 AM, Andrea Aime wrote:

Hi Niels,
there is one test about custom dimensions treatment that was 
contributed by Mike Benowitz years ago, I've
cc'ed him along with another fellow at GDIT, they might be affected 
(or not).
I also know GeoMesa has their own custom readers, no idea if they are 
dimension enabled or not. Cc'ed Jim just in case.


Cheers
Andrea


On Wed, Mar 22, 2017 at 2:27 PM, Niels Charlier > wrote:


Hello,

This message is particularly aimed at people who develop/use coverage
readers that support custom dimensions, but are outside of the main
geotools/geoserver code base.

I am proposing a change that may affect them (see
https://osgeo-org.atlassian.net/browse/GEOS-7989
). In short:

At this moment Coverage Reader may or may not advertise a Java class
through their metadata for each custom dimension.

However, geoserver WMS and WCS1 ignore this and always put custom
dimension values in String format inside the query filter. My PR would
change this and attempt to convert to the advertised type first, if
possible.

The only affected readers in the official code base are netcdf and
imagemosaic. My change would break neither, but rather improve
both (by
supporting Dates). However, it is possible that there are other
readers
around that advertise a class but still assume a String in the query
(because that is how it works now).

So if that is the case, please respond to this message.

In general, I would like to know whether this change is OK with the
community.

Kind Regards

Niels



--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Geoserver-devel mailing list
geoserver-de...@lists.sourceforge.net

https://lists.sourceforge.net/lists/listinfo/geoserver-devel





--
==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.
==

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39  339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o 
nel/i file/s allegato/i sono da considerarsi strettamente riservate. 
Il loro utilizzo è consentito esclusivamente al destinatario del 
messaggio, per le finalità indicate nel messaggio stesso. Qualora 
riceviate questo messaggio senza esserne il destinatario, Vi preghiamo 
cortesemente di darcene notizia via e-mail e di procedere alla 
distruzione del messaggio stesso, cancellandolo dal Vostro sistema. 
Conservare il messaggio stesso, divulgarlo anche in parte, 
distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità 
diverse, costituisce comportamento contrario ai principi dettati dal 
D.Lgs. 196/2003.


The information in this message and/or attachments, is intended solely 
for the attention and use of the named addressee(s) and may be 
confidential or proprietary in nature or covered by the provisions of 
privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New 
Data Protection Code).Any use not in accord with its purpose, any 
disclosure, reproduction, copying, distribution, or either 
dissemination, either whole or partial, is strictly forbidden except 
previous formal approval of the named addressee(s). If you are not the 
intended recipient, please contact immediately the sender by 
telephone, fax or e-mail and delete the information in this message 
that has been received in error. The sender does not give any warranty 
or accept liability as the content, accuracy or completeness of sent 
messages and accepts no responsibility  for changes made after they 
were sent or for other risks which arise as a result of e-mail 
transmission, viruses, etc.



---



--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

[Geotools-devel] Question about final marker on ContentDataStore method

[Jim Hughes]

Hi all,

In GeoMesa, we are using the ContentDataStore, etc abstract interfaces.  
In the case of ContentDataStore.getSchema functions, I'd like to suggest 
that getSchema(String typename) ought not be marked final.

For the DataStore in question, sufficient metadata is available at the 
DataStore level, and instantiating a FeatureSource creates additional 
threads and connections (which in this case is very undesirable).

Thoughts?

Cheers,

Jim

https://github.com/geotools/geotools/blob/master/modules/library/data/src/main/java/org/geotools/data/store/ContentDataStore.java#L342


--
The Command Line: Reinvented for Modern Developers
Did the resurgence of CLI tooling catch you by surprise?
Reconnect with the command line and become more productive. 
Learn the new .NET and ASP.NET CLI. Get your free copy!
http://sdm.link/telerik
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] Idea: FeatureStream

[Jim Hughes]

On 8/6/2016 4:41 AM, Andrea Aime wrote:
On Sat, Aug 6, 2016 at 9:01 AM, Jody Garnett > wrote:


The sad part / danger / risk is that that our Filter interface
would only be one kind of predicate, and I would expect normal
java developers to make a query quickly in java code - which is
something we cannot optimize.


No objections to having streams around, just make sure to properly 
document them and give warning to people

about performance issues.


As a suggestion, maybe it make sense to have 
FeatureSource.getFeatureStream() and 
FeatureSource.getFeatureStream(Filter) right next to each other.


Having a method which can either push-down a query or pull back all your 
data at runtime has me worried for large datasets.


Cheers,

Jim

From what I gathered in the discussion so far, it's a bit like 
hibernate, it can simplify things, but in order to use it
efficiently one needs to understand how it works internally. Maybe 
indicate the docs examples of what will
be efficiently sent down to the store for native translation, and what 
cannot be.


Wondering, are there risks about properly closing feature iterators 
here? If you are using visitor I guess not,
but in that case there are other potential issues with our current 
interface, like inability to bail out of the iteration early.


Cheers
Andrea

--
==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.
==

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39  339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o 
nel/i file/s allegato/i sono da considerarsi strettamente riservate. 
Il loro utilizzo è consentito esclusivamente al destinatario del 
messaggio, per le finalità indicate nel messaggio stesso. Qualora 
riceviate questo messaggio senza esserne il destinatario, Vi preghiamo 
cortesemente di darcene notizia via e-mail e di procedere alla 
distruzione del messaggio stesso, cancellandolo dal Vostro sistema. 
Conservare il messaggio stesso, divulgarlo anche in parte, 
distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità 
diverse, costituisce comportamento contrario ai principi dettati dal 
D.Lgs. 196/2003.


The information in this message and/or attachments, is intended solely 
for the attention and use of the named addressee(s) and may be 
confidential or proprietary in nature or covered by the provisions of 
privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New 
Data Protection Code).Any use not in accord with its purpose, any 
disclosure, reproduction, copying, distribution, or either 
dissemination, either whole or partial, is strictly forbidden except 
previous formal approval of the named addressee(s). If you are not the 
intended recipient, please contact immediately the sender by 
telephone, fax or e-mail and delete the information in this message 
that has been received in error. The sender does not give any warranty 
or accept liability as the content, accuracy or completeness of sent 
messages and accepts no responsibility  for changes made after they 
were sent or for other risks which arise as a result of e-mail 
transmission, viruses, etc.



---



--
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] Idea: FeatureStream

[Jim Hughes]

Hi Devon, Jody,

This looks exciting!  For the (E)CQL filters as predicates, is there any 
chance those predicates could be passed down to the FeatureSource?  Or 
would there be different execution from...


featureSource.stream.filter(predicate)
and
featureSource.getFeatures(predicate).stream

Naively, it seems what you suggested (and my first example) would stream 
back all the features and then apply all filtering 'client' side.  If 
there are tricks to write the first and have it perform database queries 
sensibly, that might be good.


If that support is only partial or is intuitive, I'd worry that we are 
giving folks a way to cause too much trouble.  On the other hand, if 
there are really clever options for pushing around computation, weaving 
that through the FeatureSource/Collection API could be really powerful.


Cheers,

Jim

On 8/5/2016 7:07 PM, Jody Garnett wrote:
Reviewing the methods with Devon a lot of it looked pretty straight 
forward using FeatureSource and visitor.


featureSource.stream().forEach(
   f -> System.out.println( f.getId() ) );
);

Once concern I had, well Kevin had, was how to handle predicates 
cleanly. There is an overlap of functionality between Predicates and 
our Filter interface - it would be great Filter could implement 
Predicate (allowing them to be used for java streams) and allowing us 
to detect the use of:


Filter predicate = CQL.toFilter("attName >= 5");

boolean found = featureSource.stream().anyMatch( predicate );

featureSource.stream().filter( predicate ).forEach(
   f -> System.out.println( f.getId() ) );
);

Collection geoms = featureSource.stream().map(
f -> (Geometry) f.getDefaultGeometry();
).collect( Collectors.toList() );



--
Jody Garnett

On 5 August 2016 at 14:45, Devon Tucker > wrote:


Hi all,

Jody and I were talking today about and idea he has been agitating
for a while. We'd like to basically implement the Java 8 streaming
interface for features on top of FeatureSource or
FeatureCollection. Just throwing this out there for early feedback
to see what people think.

Jody thinks that much of that API can be implemented in terms of
the existing functionality fairly simply/efficiently while others
can be delegated to simpler Stream objects (created from builder
utilities).

Interested in what people think. We'll probably do a quick
proposal shortly.

Cheers,
Devon


--

___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net

https://lists.sourceforge.net/lists/listinfo/geotools-devel




--


___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel



--
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

[Geotools-devel] Deleting the unsupported coverage-multidim module

[Jim Hughes]

Hi all,

During the bug stomp, I looked into 
https://osgeo-org.atlassian.net/browse/GEOT-4821 and noticed that the 
coverage-multidim module has a few issues (1).  First, the geotiff bit 
depends on GeoTools 2.6-SNAPSHOT (and an gt-coverage-api-old module).  
Second, for the hdf4 Maven module, the build is broken while looking for 
some netcdf version 4.3.16 artifacts.

Does anyone use either of these?  Is it safe to nuke?

Cheers,

Jim

1. 
https://github.com/geotools/geotools/tree/master/modules/unsupported/coverage-multidim

--
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are 
consuming the most bandwidth. Provides multi-vendor support for NetFlow, 
J-Flow, sFlow and other flows. Make informed decisions using capacity planning
reports.http://sdm.link/zohodev2dev
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] [Geoserver-users] [Geoserver-devel] Call for Online MiniSprint

[Jim Hughes]

Hi all,

I'd like to participate as well; maybe you can kick a few of the easy 
tasks my way.

Cheers,

Jim

On 7/11/2016 7:15 PM, Ben Caradoc-Davies wrote:
> Count me in!
>
> Kind regards,
> Ben.
>
> On 12/07/16 10:28, Jody Garnett wrote:
>> I tomorrows meeting I would like to firmly establish July 22nd as our
>> intended date for this (or establish an alternate if the above does not
>> work out).
>>
>> In anticipation, how is everyone's availability for July 22nd?
>>
>> --
>> Jody Garnett
>>
>> On 6 July 2016 at 12:04, Jody Garnett  wrote:
>>
>>> My +1 for July 22nd.
>>>
>>> Hard to schedule a monthly "bug rodeo" at the end of the month.
>>>
>>> As per meeting discussion It may be helpful to have a couple volunteers to
>>> prep the day before so we can be effective on July 22nd.
>>>
>>> --
>>> Jody Garnett
>>>
>>> On 6 July 2016 at 08:16, Simone Giannecchini <
>>> simone.giannecch...@geo-solutions.it> wrote:
>>>
 Dear All,
 sorry for cross posting but I believe this is something that might
 impact both users as well as developers of GetoTools and GeoServer.

 I wanted to propose an online, distributed, loosely coordinated 1 day
 sprint in the short term where people can help GeoServer as well as
 GeoTools with things like:
 - reviewing open JIRA
 - fixing bugs
 - improving documentation
 - doing testing and so on
 - cheering the others up

 Coordination could be as simple as a google spreadsheet where people
 can log what they are working on + an hangout to chat if/as needed.
 This could become a recurrent event, like one a month in the future.
 The goal is not to develop new features but to improve quality of code
 and documentation.

 Right now I have in mind two dates:
 - 22nd of July
 - 29th of July

 We at GeoSolutions are happy with both dates and we will do this
 anyway, to get the ball rolling.

 Looking forward to reading your feedback. I might move this email over
 to a post on the GeoServer blog to gain more attention.

 Regards,
 Simone Giannecchini
 ==
 GeoServer Professional Services from the experts!
 Visit http://goo.gl/it488V for more information.
 ==
 Ing. Simone Giannecchini
 @simogeo
 Founder/Director

 GeoSolutions S.A.S.
 Via di Montramito 3/A
 55054  Massarosa (LU)
 Italy
 phone: +39 0584 962313
 fax: +39 0584 1660272
 mob:   +39 333 8128928

 http://www.geo-solutions.it
 http://twitter.com/geosolutions_it

 ---
 AVVERTENZE AI SENSI DEL D.Lgs. 196/2003
 Le informazioni contenute in questo messaggio di posta elettronica e/o
 nel/i file/s allegato/i sono da considerarsi strettamente riservate.
 Il loro utilizzo è consentito esclusivamente al destinatario del
 messaggio, per le finalità indicate nel messaggio stesso. Qualora
 riceviate questo messaggio senza esserne il destinatario, Vi preghiamo
 cortesemente di darcene notizia via e-mail e di procedere alla
 distruzione del messaggio stesso, cancellandolo dal Vostro sistema.
 Conservare il messaggio stesso, divulgarlo anche in parte,
 distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità
 diverse, costituisce comportamento contrario ai principi dettati dal
 D.Lgs. 196/2003.

 The information in this message and/or attachments, is intended solely
 for the attention and use of the named addressee(s) and may be
 confidential or proprietary in nature or covered by the provisions of
 privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New
 Data Protection Code).Any use not in accord with its purpose, any
 disclosure, reproduction, copying, distribution, or either
 dissemination, either whole or partial, is strictly forbidden except
 previous formal approval of the named addressee(s). If you are not the
 intended recipient, please contact immediately the sender by
 telephone, fax or e-mail and delete the information in this message
 that has been received in error. The sender does not give any warranty
 or accept liability as the content, accuracy or completeness of sent
 messages and accepts no responsibility  for changes made after they
 were sent or for other risks which arise as a result of e-mail
 transmission, viruses, etc.


 --
 Attend Shape: An AT Tech Expo July 15-16. Meet us at AT Park in San
 Francisco, CA to explore cutting-edge tech and listen to tech luminaries
 present their vision of the future. This family event has something for
 everyone, including kids. Get more information and register today.
 http://sdm.link/attshape
 ___
 GeoTools-Devel 

[Geotools-devel] In-memory vs on-disk sorting

[Jim Hughes]

Hi all,

Will looking into some slow WFS paging/sorting results, I found that the 
hint MAX_MEMORY_SORT is both 1) really low (1000) and 2) not 
configurable from the commandline or GeoServer's UI.

Would anyone be fussed if we bumped the value from 1000? (Configured 
here: (1).)  Would something like 100k be ok given that RAM is cheap?

Second, would it make sense for this option to be configured on 
GeoServer's Settings->Global page?

For my current need, I can use some of my own classes which are 
instantiated a GeoServer startup to call 
'Hints.putSystemDefault(Hints.MAX_MEMORY_SORT, BIGGER_NUMBER)' manually

Thanks in advance for any feedback,

Jim

1. 
https://github.com/geotools/geotools/blob/master/modules/library/main/src/main/java/org/geotools/data/sort/MergeSortDumper.java#L93

--
Mobile security can be enabling, not merely restricting. Employees who
bring their own devices (BYOD) to work are irked by the imposition of MDM
restrictions. Mobile Device Manager Plus allows you to control only the
apps on BYO-devices by containerizing them, leaving personal data untouched!
https://ad.doubleclick.net/ddm/clk/304595813;131938128;j
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] [Geoserver-devel] Reading GeoTiffs from HDFS

[Jim Hughes]

Hi Chris,

Nice!  That's a fun find.

Generally, I do like the idea of using Map/Reduce or Spark to 
pre-generate tiles or an image pyramid.  We've kicked around the idea of 
GWC + M/R a few times in passing.  If one has Hadoop infrastructure 
hanging around, it might make sense to use GeoTrellis, SpatialHadoop 
(GeoJini), etc. for some of that processing.

Either way, being able to read the odd raster file straight from hdfs:// 
or s3:// and have it cached in memory seems like an amusing/useful 
project.  I'm hopeful we can nail down the details.

Cheers,

Jim

On 04/22/2016 02:27 PM, Chris Snider wrote:
> I did find this reference (helpful ?):
> https://github.com/openreserach/bin2seq/blob/master/src/main/java/com/openresearchinc/hadoop/sequencefile/GeoTiff.java
>
> " /@formatter:off
> /**
>   *
>   * A program to demo retrive attributes from Geotiff images as Hadoop 
> SequenceFile stored on hdfs:// or s3://
>   *
>   *
>   * @author heq
>   */
> // @formatter:on"
>
> Chris Snider
> Senior Software Engineer
> Intelligent Software Solutions, Inc.
>
>
>
> -Original Message-
> From: Chris Snider [mailto:chris.sni...@issinc.com]
> Sent: Friday, April 22, 2016 12:11 PM
> To: Jim Hughes <jn...@ccri.com>; Simone Giannecchini 
> <simone.giannecch...@geo-solutions.it>
> Cc: geoserver-de...@lists.sourceforge.net; GeoTools Developers list 
> <geotools-devel@lists.sourceforge.net>
> Subject: Re: [Geoserver-devel] [Geotools-devel] Reading GeoTiffs from HDFS
>
> Hi,
>
> I don't know that much about HDFS, but is there something that can be setup 
> like a map/reduce function directly in the HDFS servers that can do some of 
> the restriction of byte level data returned? Yarn/Sparql, some other acronym? 
>  I assume it would have to be administrator responsibility to add said 
> process to the server stack if it is even possible.
>
> Chris Snider
> Senior Software Engineer
> Intelligent Software Solutions, Inc.
>
>
> -Original Message-
> From: Jim Hughes [mailto:jn...@ccri.com]
> Sent: Friday, April 22, 2016 12:05 PM
> To: Simone Giannecchini <simone.giannecch...@geo-solutions.it>
> Cc: geoserver-de...@lists.sourceforge.net; GeoTools Developers list 
> <geotools-devel@lists.sourceforge.net>
> Subject: Re: [Geoserver-devel] [Geotools-devel] Reading GeoTiffs from HDFS
>
> Hi Simone,
>
> Thanks for the feedback!
>
> As quick response, for #1, I agree that using mosaicing / an image
> pyramid would be a great option.  I was mainly working at the prototype
> phase, and I wanted to have a discussion on the mailing lists
> (especially since changes are required in ImageIO-Ext or GeoTools and
> GeoServer.)
>
> For #2, I do like the idea of having a cahce in the ImageInputStream.
>   From that suggestion, I take it that you'd be willing to entertain
> changes to the current ImageInputStreams and the additional of some way
> to cache data.
>
> In terms of caching, do you have any suggestions?  Also, I'd be
> interested in any advice for how we can configure that cache and make
> those options available to a GeoServer admin appropriately.
>
> Further, at a high-level, should the goal for this work be a community
> module?
>
> Cheers,
>
> Jim
>
> On 04/22/2016 01:49 PM, Simone Giannecchini wrote:
>> Dear Jim,
>> quick feedback.
>>
>> First of all congratulation on making this work. As I suspected the
>> bottleneck is getting the data out of HDFS.
>> I can think about two things (which we are not mutually exclusive):
>>
>> -1- Maybe complex, put smaller bits into HFDS and use the mosaic to
>> serve or even develop a light(er)weight layer that can pull the
>> granules.
>>
>> This would help with WMS requests over large files as you'll end up
>> use smaller chunks to satisfy them most of the time
>>
>> -2- We could build a more complex ImageInputStream that:
>>
>> - has an internal cache (file and or memory) that does not get thrown
>> away upon each request but tends to live longer for each single file
>> in HDF
>> - we would have different streams reuse the same cache. Multiple
>> requests might read data from the cache concurrently but when data is
>> not there, we would block the thread for the request, go back to HFDS,
>> pull the data, write to the cache and so on
>>
>> We could put together 1 and 2 to make things faster.
>>
>> Hope this helps, anyway, I am in favour of exploring this in order to
>> allow the GeoServer stack to support data from HDFS.
>>
>> Regards,
>> Simone Giannecchini
>> ==
>> GeoServer Professional Services from the experts

Re: [Geotools-devel] Reading GeoTiffs from HDFS

[Jim Hughes]

Hi Simone,

Thanks for the feedback!

As quick response, for #1, I agree that using mosaicing / an image 
pyramid would be a great option.  I was mainly working at the prototype 
phase, and I wanted to have a discussion on the mailing lists 
(especially since changes are required in ImageIO-Ext or GeoTools and 
GeoServer.)

For #2, I do like the idea of having a cahce in the ImageInputStream.  
 From that suggestion, I take it that you'd be willing to entertain 
changes to the current ImageInputStreams and the additional of some way 
to cache data.

In terms of caching, do you have any suggestions?  Also, I'd be 
interested in any advice for how we can configure that cache and make 
those options available to a GeoServer admin appropriately.

Further, at a high-level, should the goal for this work be a community 
module?

Cheers,

Jim

On 04/22/2016 01:49 PM, Simone Giannecchini wrote:
> Dear Jim,
> quick feedback.
>
> First of all congratulation on making this work. As I suspected the
> bottleneck is getting the data out of HDFS.
> I can think about two things (which we are not mutually exclusive):
>
> -1- Maybe complex, put smaller bits into HFDS and use the mosaic to
> serve or even develop a light(er)weight layer that can pull the
> granules.
>
> This would help with WMS requests over large files as you'll end up
> use smaller chunks to satisfy them most of the time
>
> -2- We could build a more complex ImageInputStream that:
>
> - has an internal cache (file and or memory) that does not get thrown
> away upon each request but tends to live longer for each single file
> in HDF
> - we would have different streams reuse the same cache. Multiple
> requests might read data from the cache concurrently but when data is
> not there, we would block the thread for the request, go back to HFDS,
> pull the data, write to the cache and so on
>
> We could put together 1 and 2 to make things faster.
>
> Hope this helps, anyway, I am in favour of exploring this in order to
> allow the GeoServer stack to support data from HDFS.
>
> Regards,
> Simone Giannecchini
> ==
> GeoServer Professional Services from the experts!
> Visit http://goo.gl/it488V for more information.
> ==
> Ing. Simone Giannecchini
> @simogeo
> Founder/Director
>
> GeoSolutions S.A.S.
> Via di Montramito 3/A
> 55054  Massarosa (LU)
> Italy
> phone: +39 0584 962313
> fax: +39 0584 1660272
> mob:   +39  333 8128928
>
> http://www.geo-solutions.it
> http://twitter.com/geosolutions_it
>
> ---
> AVVERTENZE AI SENSI DEL D.Lgs. 196/2003
> Le informazioni contenute in questo messaggio di posta elettronica e/o
> nel/i file/s allegato/i sono da considerarsi strettamente riservate.
> Il loro utilizzo è consentito esclusivamente al destinatario del
> messaggio, per le finalità indicate nel messaggio stesso. Qualora
> riceviate questo messaggio senza esserne il destinatario, Vi preghiamo
> cortesemente di darcene notizia via e-mail e di procedere alla
> distruzione del messaggio stesso, cancellandolo dal Vostro sistema.
> Conservare il messaggio stesso, divulgarlo anche in parte,
> distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità
> diverse, costituisce comportamento contrario ai principi dettati dal
> D.Lgs. 196/2003.
>
> The information in this message and/or attachments, is intended solely
> for the attention and use of the named addressee(s) and may be
> confidential or proprietary in nature or covered by the provisions of
> privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New
> Data Protection Code).Any use not in accord with its purpose, any
> disclosure, reproduction, copying, distribution, or either
> dissemination, either whole or partial, is strictly forbidden except
> previous formal approval of the named addressee(s). If you are not the
> intended recipient, please contact immediately the sender by
> telephone, fax or e-mail and delete the information in this message
> that has been received in error. The sender does not give any warranty
> or accept liability as the content, accuracy or completeness of sent
> messages and accepts no responsibility  for changes made after they
> were sent or for other risks which arise as a result of e-mail
> transmission, viruses, etc.
>
>
> On Sun, Apr 17, 2016 at 9:49 PM, Jim Hughes <jn...@ccri.com> wrote:
>> Hi all,
>>
>> I want to report on my success with registering and displaying GeoTiffs
>> stored on HDFS.  There are some limitations with this approach;
>> particularly, I am unsure if there's anyway to cache / memory-map the
>> data.  As such, I believe each request is re-downloading the entire file.
>>
>> Generally, I hope

Re: [Geotools-devel] GeoTools / GeoServer Meeting 2016-04-19

[Jim Hughes]

Good question.  Andrew Hulbert had initially submitted the PR against 
master here: https://github.com/geotools/geotools/pull/1128.


In some sense, I'm looking for a backport of that PR for GeoTools and 
GeoServer.


On 04/19/2016 04:30 PM, Jody Garnett wrote:

That is for 14.x, does master need the same update?

--
Jody Garnett

On 19 April 2016 at 13:29, Jim Hughes <jn...@ccri.com 
<mailto:jn...@ccri.com>> wrote:


Hi all,

The GeoTools PR for the Apache Collections update is here:
https://github.com/geotools/geotools/pull/1175.

Cheers,

Jim

On 04/19/2016 04:19 PM, Ben Caradoc-Davies wrote:
> GeoTools / GeoServer Meeting 2016-04-19
> ===
>
> Attending
> -
>
> Ben Caradoc-Davies
    > Kevin Smith
> Jim Hughes
> Torben Barsballe
> Jody Garnett
> Dave Blasby
> Jukka Rahkonen
>
> Apologies
> -
>
> Brad Hards
> Andrea Aime
>
> Agenda
> --
>
> - Release of 15-beta2 / 2.9-beta2
> - Windows service wrapper licence
> - Apache Commons Upgrade?
> - Can we put AbstractDataStore in jail?
> - GeoMesa discussion
>
> Actions
> ---
>
> Actions from last meeting
> -
>
> - Ben: send an email to Emanuele to see if we can lock down GeoFence
> dependency for release [DONE]
> - Jody:  email GeoServer developer mailing list calling for
nominations
> for OSGeo project officer
>
> Release of 15-beta2 / 2.9-beta2
> ---
>
> - Release process and pre-flight testing under way. We will need
it to
> at least run prior to announcing.
>
> Todo:
> - login issue GEOS-7482 - who can work on this?
> - (jody) mac DMG not starting
> - (jody) bin download not starting on mac
> - (kevin) linux bin startup (openjdk and oracle) - worked fine
except
> for GEOS-7482
> - windows not starting, Brad provided a workaround?
>
> Reason to do the beta is to confirm packaging is working, so we
need to
> fix the above.
>
> Should we double back and include Justin's fix? Does it prevent
PostGIS
> from functioning ... Ben says we should not release with failing
build.
>
> Actions:
> - take startup issues to email
> - redo the geotools release
>
> Windows service wrapper licence
> ---
>
> [GEOS-7488] Windows service wrapper appears to have problematic
license
> https://osgeo-org.atlassian.net/browse/GEOS-7488
>
> Compare:
> -
https://github.com/geoserver/geoserver/tree/master/src/release/wrapper
> -
>
https://github.com/geoserver/geoserver/tree/master/src/release/installer/win
> (MIT-License)
>
> The two wrapper.exe files are two different programs (different
size).
>
> Action:
> - there are two versions in the codebase, do they both work?
> - We need more information, take to email list
>
> Apache Commons Upgrade?
> ---
>
> Can we update apache-commons-collections 3.1.0 --> 3.2.2 ?
>
> Yes - GeoTools 14.4 release is scheduled for next month.
>
> Can we put AbstractDataStore in jail?
> -
>
> Idea to put these into seperate gt-abstract-datastore module?
>
> Good communication/perhaps too late.
>
> GeoMesa discussion
> --
>
> GeoMesa is now updated to ContentDataStore - congrats!
>
> Discussion about hdfs URLs for distributed file system, anyone
review
> tweaks to geotools/geoserver?
>
> (Tell wicket that other URL schemes are valid, and update
geoserver file
> model, tell geotools SPI handlers for objects->inputstreams).
>
>



--
Find and fix application performance issues faster with
Applications Manager
Applications Manager provides deep performance insights into
multiple tiers of
your business applications. It resolves application problems
quickly and
reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
<mailto:GeoTools-Devel@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/geotools-devel




-

Re: [Geotools-devel] GeoTools / GeoServer Meeting 2016-04-19

[Jim Hughes]

Hi all,

The GeoTools PR for the Apache Collections update is here: 
https://github.com/geotools/geotools/pull/1175.

Cheers,

Jim

On 04/19/2016 04:19 PM, Ben Caradoc-Davies wrote:
> GeoTools / GeoServer Meeting 2016-04-19
> ===
>
> Attending
> -
>
> Ben Caradoc-Davies
> Kevin Smith
> Jim Hughes
> Torben Barsballe
> Jody Garnett
> Dave Blasby
> Jukka Rahkonen
>
> Apologies
> -
>
> Brad Hards
> Andrea Aime
>
> Agenda
> --
>
> - Release of 15-beta2 / 2.9-beta2
> - Windows service wrapper licence
> - Apache Commons Upgrade?
> - Can we put AbstractDataStore in jail?
> - GeoMesa discussion
>
> Actions
> ---
>
> Actions from last meeting
> -
>
> - Ben: send an email to Emanuele to see if we can lock down GeoFence
> dependency for release [DONE]
> - Jody:  email GeoServer developer mailing list calling for nominations
> for OSGeo project officer
>
> Release of 15-beta2 / 2.9-beta2
> ---
>
> - Release process and pre-flight testing under way. We will need it to
> at least run prior to announcing.
>
> Todo:
> - login issue GEOS-7482 - who can work on this?
> - (jody) mac DMG not starting
> - (jody) bin download not starting on mac
> - (kevin) linux bin startup (openjdk and oracle) - worked fine except
> for GEOS-7482
> - windows not starting, Brad provided a workaround?
>
> Reason to do the beta is to confirm packaging is working, so we need to
> fix the above.
>
> Should we double back and include Justin's fix? Does it prevent PostGIS
> from functioning ... Ben says we should not release with failing build.
>
> Actions:
> - take startup issues to email
> - redo the geotools release
>
> Windows service wrapper licence
> ---
>
> [GEOS-7488] Windows service wrapper appears to have problematic license
> https://osgeo-org.atlassian.net/browse/GEOS-7488
>
> Compare:
> - https://github.com/geoserver/geoserver/tree/master/src/release/wrapper
> -
> https://github.com/geoserver/geoserver/tree/master/src/release/installer/win
> (MIT-License)
>
> The two wrapper.exe files are two different programs (different size).
>
> Action:
> - there are two versions in the codebase, do they both work?
> - We need more information, take to email list
>
> Apache Commons Upgrade?
> ---
>
> Can we update apache-commons-collections 3.1.0 --> 3.2.2 ?
>
> Yes - GeoTools 14.4 release is scheduled for next month.
>
> Can we put AbstractDataStore in jail?
> -
>
> Idea to put these into seperate gt-abstract-datastore module?
>
> Good communication/perhaps too late.
>
> GeoMesa discussion
> --
>
> GeoMesa is now updated to ContentDataStore - congrats!
>
> Discussion about hdfs URLs for distributed file system, anyone review
> tweaks to geotools/geoserver?
>
> (Tell wicket that other URL schemes are valid, and update geoserver file
> model, tell geotools SPI handlers for objects->inputstreams).
>
>


--
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

[Geotools-devel] Reading GeoTiffs from HDFS

[Jim Hughes]

Hi all,

I want to report on my success with registering and displaying GeoTiffs 
stored on HDFS.  There are some limitations with this approach; 
particularly, I am unsure if there's anyway to cache / memory-map the 
data.  As such, I believe each request is re-downloading the entire file.

Generally, I hope to document my approach well enough so that others 
could follow it (if needed) and to solicit feedback.  In terms of 
feedback, I'd love to hear 1) if there are improvements, and 2) if the 
changes are reasonable enough to be considered for a proposal/merge request.

That out of the way, here's the rough outline:

1.  Register additional URL handlers.
2.  Convince validation layers in GeoServer that 'hdfs' is an ok URL scheme.
3.  Get bytes out of the HDFS file.

For step 1, note that Java's URL scheme is pluggable via 
java.net.URLStreamHandler.  The docs(1) point out that one can call 
URL.setURLStreamHandlerFactory to setup a Factory to provide such a 
handler.  This method can only be called once, and folks from the 
internet (2) do yoga since Tomcat already registers a factory.  They 
seem to have missed the fact that the Tomcat factory actually lets you 
add your own.  I provide a gist (3) to show a little bean which will 
instantiate a Hadoop URL handler and try to install it using both of 
those methods.

There are two places I found in GeoServer which validate the URL given 
in the page for adding a GeoTiff.  The first is the GeoServer 
FileExistValidator which calls out to a Wicket UrlValidator. Telling the 
Wicket class to allow_all_schemes knocks out that issue.  For the 
second, in the FileModel, one needs to provide a happy path for URLs 
which are not local to the filesystem.  Those two small changes are here 
(4).

Once GeoServer will register a GeoTiff coverage with a non-'file://' 
URL, we need to read the bytes.  Javax has an interface 
javax.imageio.spi.ImageInputStreamSpi which adapts between instances of 
a particular class and an ImageInputStream.

For my prototype, I wrote an instance of this interface which takes a 
string, checks if it starts with "hdfs", creates a URL, and returns new 
MemoryCacheImageInputStream(url.openStream()).  The only problem with 
this approach is that there is already an implementation which handles 
Strings, and GeoTools's ImageIOExt tries the first one and skips any 
others.  One can update that handling (5) slightly to try all the 
handlers.  It'd probably be better to update (6) to try url.openStream 
as a fallback.

During testing, I worked with the sfdem.tif which ships with GeoServer.  
The hdfs layer was a little slower than the local filesystem layer, but 
it wasn't unusable.  To crank things up, I tried out a 600+ megabyte 
GeoTiff from Natural Earth, and it was downright slow.  Using a network 
monitor, I was able to observe network traffic consistent with the 
entire file being re-read for most requests.  I think this approach may 
be slightly useful for layers which are infrequently accessed and then 
only be a few users.

Thanks to everyone who had suggestions and encouragement for the 
original thread!

Cheers,

Jim

Step 1: Register additional URL handlers:

1. 
http://download.java.net/jdk7/archive/b123/docs/api/java/net/URL.html#URL%28java.lang.String,%20java.lang.String,%20int,%20java.lang.String%29

2. http://skife.org/java/url/library/2012/05/14/java_url_handlers.html

3. Gist for a bean to register the Hadoop URL handlers:
https://gist.github.com/jnh5y/1739baa42466d66e383fa26ffd7235ca

Step 2: GeoServer changes:
4. 
https://github.com/jnh5y/geoserver/commit/5320f26a0574f034433aa96097054ec1ec782d45
The FileModel change could be a little more robust.

Step 3: GeoTools changes:
5. 
https://github.com/jnh5y/geotools/commit/f2db29339c7f7e43d0c52ab93195babc1abb6f49

Or one could modify the URL handling here:
6. 
https://github.com/geosolutions-it/imageio-ext/blob/master/library/streams/src/main/java/it/geosolutions/imageio/stream/input/spi/URLImageInputStreamSpi.java#L88-L97




--
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] WFS Transaction XML parsing

[Jim Hughes]

Hi Andrea,

Sure.  To clarify, my goal is to add data to WFS transaction which will 
influence how it is written to Accumulo (and consequently queried back 
out).  In some sense, it'd be akin to providing ingest hints / 
instructions.  For example, suppose that some data could only be seen by 
'admin' users.  In a transaction, it'd be great to indicate which 
features are 'admin' only.


Just to say it out loud, the GeoServer authentication/authorization 
framework is awesome.  Anytime something hasn't been handled already, it 
seems that there is usually a pluggable extension point to address the 
situation.  For this task, the TransactionPlugins and Native tags do 
provide one general solution already.


Thanks,

Jim

On 04/08/2016 11:50 AM, Andrea Aime wrote:
On Fri, Apr 8, 2016 at 4:09 PM, Jim Hughes <jn...@ccri.com 
<mailto:jn...@ccri.com>> wrote:


My high-level goal is to find a way to add security info to a WFS
transaction. 



Can you elaborate why just an authentication + authorization rules
would not work?

Cheers
Andrea


--
==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.
==

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39  339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o 
nel/i file/s allegato/i sono da considerarsi strettamente riservate. 
Il loro utilizzo è consentito esclusivamente al destinatario del 
messaggio, per le finalità indicate nel messaggio stesso. Qualora 
riceviate questo messaggio senza esserne il destinatario, Vi preghiamo 
cortesemente di darcene notizia via e-mail e di procedere alla 
distruzione del messaggio stesso, cancellandolo dal Vostro sistema. 
Conservare il messaggio stesso, divulgarlo anche in parte, 
distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità 
diverse, costituisce comportamento contrario ai principi dettati dal 
D.Lgs. 196/2003.


The information in this message and/or attachments, is intended solely 
for the attention and use of the named addressee(s) and may be 
confidential or proprietary in nature or covered by the provisions of 
privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New 
Data Protection Code).Any use not in accord with its purpose, any 
disclosure, reproduction, copying, distribution, or either 
dissemination, either whole or partial, is strictly forbidden except 
previous formal approval of the named addressee(s). If you are not the 
intended recipient, please contact immediately the sender by 
telephone, fax or e-mail and delete the information in this message 
that has been received in error. The sender does not give any warranty 
or accept liability as the content, accuracy or completeness of sent 
messages and accepts no responsibility  for changes made after they 
were sent or for other risks which arise as a result of e-mail 
transmission, viruses, etc.



---


--
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial! http://pubads.g.doubleclick.net/
gampad/clk?id=1444514301=/ca-pub-7940484522588532___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] WFS Transaction XML parsing

[Jim Hughes]

Hi Justin,

Thanks.  I forgot to mention that I had seen the TransactionPlugin 
extension point; that one is awesome and I see how to use it support my 
second suggestion.


Would changes in the GML2/3ParsingUtils to inject attributes into a 
SimpleFeature's userData be sensible / more broadly useful?  I'm a bit 
shy about pushing hard for such a change since I am not sure about any 
other ramifications.


Thanks,

Jim

On 04/08/2016 10:45 AM, Justin Deoliveira wrote:


Hey Jim,

The wfs native element might be the cleanest way to handle this. I’ve 
seen folks use it before for security and validation type stuff with 
some success. In GeoServer you can plug in WFS transactions callbacks 
pretty easily using the TransactionPlugin 
<[https://github.com/geoserver/geoserver/blob/master/src/wfs/src/main/java/org/geoserver/wfs/TransactionPlugin.java]%28https://github.com/geoserver/geoserver/blob/master/src/wfs/src/main/java/org/geoserver/wfs/TransactionPlugin.java> 
extension point. It has direct access to the transaction object which 
has the native content on it.


If you do want to go with approach (1): during GML parsing I don’t 
believe there is any way to tag a GML attribute as something you want 
to store in user data. Folks might correct me on that one though. To 
support it I wonder if we could do something simple like a custom 
attribute on the gml attribute element. Something like:


|foo |

One potential downside might be validation… however in GeoServer last 
I checked we don’t validate features on the way in… mostly because 
it’s a pain to have to specify all the schema references properly so a 
parser can actually validate. It may also be there in the gml schema 
there is some attribute we might be able to use /abuse for this 
purpose… I can’t think of one off hand though.


As far as I know the feature parsing logic still lives in the 
GML3ParsingUtils and GML2ParsingUtils classes. To add some support for 
putting an attribute in user data rather than a normal attribute I 
think you would be looking at modifying this method 
<[https://github.com/geotools/geotools/blob/master/modules/extension/xsd/xsd-gml2/src/main/java/org/geotools/gml2/bindings/GML2ParsingUtils.java#L338-L362]%28https://github.com/geotools/geotools/blob/master/modules/extension/xsd/xsd-gml2/src/main/java/org/geotools/gml2/bindings/GML2ParsingUtils.java#L338-L362>.


As you’ll notice this only applies to simple features. If working with 
complex features i am actually not sure where that logic lives 
anymore. One of the app-schema folks can point you at that if need be.


Hope that helps!

-Justin



On Fri, Apr 8, 2016 at 8:09 AM Jim Hughes <jn...@ccri.com 
<mailto:jn...@ccri.com>> wrote:


Hi all,

Apologies for the cross-post, but my question hits a little of how
GeoServer handles WFS transactions and a little bit of how
GeoTools may
handle WFS/GML parsing.

My high-level goal is to find a way to add security info to a WFS
transaction.  I can see three big options:  1. add some bits to
the GML
feature / WFS call which would set a SimpleFeature's userData or
otherwise provide additional data, 2. leverage a WFS 'Native' tag,
or 3.
store the security info in the SimpleFeature.

I've got a pretty good handle on the last two options.  The first
option
would provide the greatest flexibility, so I'm wondering if anyone can
help point me in the right direction there.  (Or otherwise say
that this
is impossible without re-writing GT/GS XML handling...)

Thanks in advance,

Jim


--
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
<mailto:GeoTools-Devel@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/geotools-devel



--
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

[Geotools-devel] WFS Transaction XML parsing

[Jim Hughes]

Hi all,

Apologies for the cross-post, but my question hits a little of how 
GeoServer handles WFS transactions and a little bit of how GeoTools may 
handle WFS/GML parsing.

My high-level goal is to find a way to add security info to a WFS 
transaction.  I can see three big options:  1. add some bits to the GML 
feature / WFS call which would set a SimpleFeature's userData or 
otherwise provide additional data, 2. leverage a WFS 'Native' tag, or 3. 
store the security info in the SimpleFeature.

I've got a pretty good handle on the last two options.  The first option 
would provide the greatest flexibility, so I'm wondering if anyone can 
help point me in the right direction there.  (Or otherwise say that this 
is impossible without re-writing GT/GS XML handling...)

Thanks in advance,

Jim

--
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

[Geotools-devel] More ImageIO-Ext questions

[Jim Hughes]

Hi all,

As I mentioned on the GeoTools/GeoServer call today, the Eclipse 
Foundation is doing a legal review of GeoTools and its transitive 
dependencies.  Jody, Tyler and others have helped extensively with the 
process so far.  Some more questions about ImageIO-Ext have come up...

The questions from Eclipse legal are below.  For the last three, I've 
tried to comment and provide some context.

Thanks in advance,

Jim

1.  Review.apt files;  There are 32 of these files and many of them of
reference data included and the origin of the data.  I've investigated a 
couple
of them and there may be possible issues.  As such, before I dig further, it
would help to understand if your attachment contains the data being 
referenced?
If it does not, we can move forward.  A couple of examples follow:

Example 1
plugin/gdal/gdalenvisat/src/site/apt/review.apt
plugin/gdal/gdalwcs/src/site/apt/review.apt
plugin/gdal/gdalwms/src/site/apt/review.apt

2.  The following files contain copyright to
* (c) 2008 Quality Nighthawk Teleradiology Group, Inc.

plugin/kakadujp2/src/main/java/it/geosolutions/imageio/plugins/jp2k/JP2KKakaduImageWriteParam.java

plugin/kakadujp2/src/main/java/it/geosolutions/imageio/plugins/jp2k/JP2KKakaduImageWriter.java

plugin/kakadujp2/src/main/java/it/geosolutions/imageio/plugins/jp2k/JP2KKakaduImageWriterSpi.java

Are either of you familiar with this copyright holder?  Is this third party
content or was it contributed?  Maybe a question for Simmone

3.  The content appears to contain content from the following third party
projects (this version):

*Nitro - http://sourceforge.net/projects/nitro-nitf/
*TurboJpeg  - http://sourceforge.net/projects/libjpeg-turbo/
*JMatio - http://sourceforge.net/projects/jmatio/

We may require third party CQs for these - to be advised.  Capturing 
more for
our IP review at this time.

4.  EnhancedRandomAccessFile.java - Do we know where this code initially 
came from?  Can we get more info/approval from Alex McManus?

5.
(Eclipse:) ECW/JP2 CODEC SDK End-User License Agreements (2 copies) - 
ECWEULA.txt - I'm assuming these were included in error?

(My Note:) Is the ECW/JP2 coded shipped with ImageIO-Ext?  Does that 
EULA need to be included or can I sensibly remove it?

6.
(Eclipse) The following file contains, "JPEG2000 driver documented on 
this page (the
JP2KAK driver) is implemented on top of the commercial Kakadu 
library"... Can
you help us understand what this means?

plugin/gdal/gdalkakadujp2/doc/frmt_jp2kak.html

(My note again: Where did the JP2KAK driver come from?  Any worries there?)


--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] [Geoserver-devel] Raster data source configuration in GeoServer

[Jim Hughes]

Hi Andrea,

Thanks for the further clarification.  While the url format is likely 
sufficient for many use cases, it is odd in the case we have.  We have a 
separate EditPanel for this, and we'll be contributing that to the 
GeoMesa community module in GeoServer.


It sounds like an extension would be to implement handling for 
additional objects (maybe a wrapper object containing a List of 
Parameters?), and that could be added to the general EditPanel code.  As 
Jody mentioned, that kind of thing would be a little R work...


Cheers,

Jim

On 02/06/2016 04:40 AM, Andrea Aime wrote:
On Thu, Feb 4, 2016 at 3:44 PM, Jim Hughes <jn...@ccri.com 
<mailto:jn...@ccri.com>> wrote:


Hi all,

Since this falls between GeoTools and GeoServer, I figured I'd
bump it and cross-post.

Phrased differently, "Does the CoverageFactory interface have the
same hooks for auto EditPanel generation as the DataStore interface?"

As the minute, in order to add a raster data source, we are
finding that we have to override GeoServer classes. For vector
data sources, we have never had to do that. Are we missing something?


You probably are? A GUI is automatically generated for rasters too.
Contrary to what Jody suggest, there are no parameters in a raster 
source, the Format class just accepts
a "Object source" without any description of what it could be, the 
list of "read parameters" is to be used

once the GridCoverageReader is already acquired, from the interface:

/**
 * Retrieve the parameter information for a {@link 
GridCoverageReader#read read} operation.

 */
@UML(identifier="getParameterInfo, numParameters", 
obligation=MANDATORY, specification=OGC_01004)

ParameterValueGroup getReadParameters();

The default DefaultCoverageStoreEditPanel assumes this object is a 
URL. If your source requires something

else then yes, you have to build your own panel

Cheers
Andrea

--
==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.
==

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054  Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39  339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o 
nel/i file/s allegato/i sono da considerarsi strettamente riservate. 
Il loro utilizzo è consentito esclusivamente al destinatario del 
messaggio, per le finalità indicate nel messaggio stesso. Qualora 
riceviate questo messaggio senza esserne il destinatario, Vi preghiamo 
cortesemente di darcene notizia via e-mail e di procedere alla 
distruzione del messaggio stesso, cancellandolo dal Vostro sistema. 
Conservare il messaggio stesso, divulgarlo anche in parte, 
distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità 
diverse, costituisce comportamento contrario ai principi dettati dal 
D.Lgs. 196/2003.


The information in this message and/or attachments, is intended solely 
for the attention and use of the named addressee(s) and may be 
confidential or proprietary in nature or covered by the provisions of 
privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New 
Data Protection Code).Any use not in accord with its purpose, any 
disclosure, reproduction, copying, distribution, or either 
dissemination, either whole or partial, is strictly forbidden except 
previous formal approval of the named addressee(s). If you are not the 
intended recipient, please contact immediately the sender by 
telephone, fax or e-mail and delete the information in this message 
that has been received in error. The sender does not give any warranty 
or accept liability as the content, accuracy or completeness of sent 
messages and accepts no responsibility  for changes made after they 
were sent or for other risks which arise as a result of e-mail 
transmission, viruses, etc.



---


--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] [Geoserver-devel] Raster data source configuration in GeoServer

[Jim Hughes]

Hi Jody,

Thanks.  That sounds like it squares with what Emilio had found.  I 
wanted to makes sure that we weren't overlooking something obvious.


Cheers,

Jim

On 02/05/2016 01:17 PM, Jody Garnett wrote:
I always figured that the raster work would be done in imageio-ext, or 
geotools as appropriate.

And any user interface work should be done as geoserver community module.

The raster Format API does provide connection parameters, which you 
can use to advertise your configuration options. You may need to work 
hard to make a user-interface dynamically from the connection 
parameters - but it would be a fun bit of RnD that would benefit all 
of geoserver.



--
Jody Garnett

On 4 February 2016 at 06:44, Jim Hughes <jn...@ccri.com 
<mailto:jn...@ccri.com>> wrote:


Hi all,

Since this falls between GeoTools and GeoServer, I figured I'd
bump it and cross-post.

Phrased differently, "Does the CoverageFactory interface have the
same hooks for auto EditPanel generation as the DataStore interface?"

As the minute, in order to add a raster data source, we are
finding that we have to override GeoServer classes. For vector
data sources, we have never had to do that. Are we missing something?

Cheers,

Jim


On 02/01/2016 09:25 AM, Emilio Lahr-Vivaz wrote:

Hello,

I'm a developer working on the GeoMesa project. We're trying to
add a custom raster data source and integrate it into GeoServer
2.8.x. However, we'd like to not extend any GeoServer classes out
of an abundance of caution around GPL licensing (we're Apache 2).
With vector data sources, all of the information regarding
configuration is pulled out of the getParametersInfo method -
perfect for us. However, it seems that for raster data sources,
any custom configuration has to be done by extending the
GeoServer StoreEditPanel - something we'd prefer to avoid.

One potential work-around we found is to register a custom URL
handler, and use it to parse our configuration. However, that
involves messy classloading issues, such that we would have to
install code in the JRE.

Are there any other extension points we're missing for
configuration of raster data sources?

Thanks,

Emilio Lahr-Vivaz



--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=267308311=/4140


___
Geoserver-devel mailing list
geoserver-de...@lists.sourceforge.net
<mailto:geoserver-de...@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/geoserver-devel




--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
<mailto:GeoTools-Devel@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/geotools-devel




--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] [Geoserver-devel] Raster data source configuration in GeoServer

[Jim Hughes]

Hi all,

Since this falls between GeoTools and GeoServer, I figured I'd bump it 
and cross-post.


Phrased differently, "Does the CoverageFactory interface have the same 
hooks for auto EditPanel generation as the DataStore interface?"


As the minute, in order to add a raster data source, we are finding that 
we have to override GeoServer classes.  For vector data sources, we have 
never had to do that.  Are we missing something?


Cheers,

Jim

On 02/01/2016 09:25 AM, Emilio Lahr-Vivaz wrote:

Hello,

I'm a developer working on the GeoMesa project. We're trying to add a 
custom raster data source and integrate it into GeoServer 2.8.x. 
However, we'd like to not extend any GeoServer classes out of an 
abundance of caution around GPL licensing (we're Apache 2). With 
vector data sources, all of the information regarding configuration is 
pulled out of the getParametersInfo method - perfect for us. However, 
it seems that for raster data sources, any custom configuration has to 
be done by extending the GeoServer StoreEditPanel - something we'd 
prefer to avoid.


One potential work-around we found is to register a custom URL 
handler, and use it to parse our configuration. However, that involves 
messy classloading issues, such that we would have to install code in 
the JRE.


Are there any other extension points we're missing for configuration 
of raster data sources?


Thanks,

Emilio Lahr-Vivaz


--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=267308311=/4140


___
Geoserver-devel mailing list
geoserver-de...@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel


--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

[Geotools-devel] Imageio-ext questions

[Jim Hughes]

Hi all,

 From an IP review for GeoMesa's use of GeoTools (and hence 
imageio-ext), the following questions came up:

1.  For Benjamin Stark's contributions, is there any agreement or 
license allowing for them to be in Imageio-ext?  See these two files:

https://github.com/geosolutions-it/imageio-ext/blob/master/library/streams/src/main/java/it/geosolutions/io/output/MathUtils.java#L4
https://github.com/geosolutions-it/imageio-ext/blob/master/library/streams/src/main/java/it/geosolutions/io/input/BitInputStream.java#L17

2.  In this file: 
https://github.com/geosolutions-it/imageio-ext/blob/1.1.13/library/imagereadmt/src/main/java/it/geosolutions/imageio/imageioimpl/imagereadmt/ImageReadOpImageMT.java
 
are the functions getSourceRegion (line 308) andcomputeRegions (line 
333) copied from Java source code?  As such, what is the license for it?

3.  Given the comment here: 
https://github.com/geosolutions-it/imageio-ext/blob/1.1.13/library/streams/src/main/java/it/geosolutions/imageio/stream/eraf/EnhancedRandomAccessFile.java#L56
 
for EnhancedRandomAccessFile, is there more info about how Alex 
McManus's code (or the underlying code) is licensed?

Thanks in advance,

Jim

--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=267308311=/4140
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] release schedule / blocker

[Jim Hughes]

Jody,

For the IP issue, if we don't hear back from Emanuele, what options do 
we have?  Would it be possible to remove the code or would we need a 
'clean room' implementation?


In the latter case, I haven't seen the function in question (either in 
the GeoTools or JAI core codebases).  Perhaps I could help out?


Other than that, what's the process for volunteering to help with a 
release?  I get the sense that a release takes an experienced 
contributor around 4 hours (most of which is watching a build and 
writing the blog post).  Would it take longer the first time?


Cheers,

Jim

On 07/10/2015 05:03 PM, Jody Garnett wrote:
I updated the release schedule 
https://github.com/geoserver/geoserver/wiki/Release-Schedule last 
week, prior to the Skype meeting.  can I ask for a sanity check, and 
who is available to help with releases over these summer months.


I have not heard back from Emanuele about GEOT-5155 LGPL 
implementation of GridCoverage2DRIA.computeRect method 
https://osgeo-org.atlassian.net/browse/GEOT-5155 which is labeled as 
a blocker.

--
Jody Garnett


--
Don't Limit Your Business. Reach for the Cloud.
GigeNET's Cloud Solutions provide you with the tools and support that
you need to offload your IT needs and focus on growing your business.
Configured For All Businesses. Start Your Cloud Today.
https://www.gigenetcloud.com/


___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel


--
Don't Limit Your Business. Reach for the Cloud.
GigeNET's Cloud Solutions provide you with the tools and support that
you need to offload your IT needs and focus on growing your business.
Configured For All Businesses. Start Your Cloud Today.
https://www.gigenetcloud.com/___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] [JIRA] (GEOT-5139) CSV datastore won't honor the configured namespace URI

[Jim Hughes]

First, the PR looks great, and I think my question is tangential. As a 
broad question, is this something we could address at the 
ContentDataStore level?


I ask since we had to solve this problem separately for the GeoMesa 
datastore(1).  We end up cooking up a new DataStore for something 
interesting every couple of months around here, and having the abstract 
implementation handle it would be awesome.


Perhaps getSchema(Name name) here
https://github.com/geotools/geotools/blob/master/modules/library/data/src/main/java/org/geotools/data/store/ContentDataStore.java#L711-713
could use the requested URI from the name?

I'll admit that I've seen namespaces only in the context of GeoServer's 
use of a GT DS, so I might be missing something. Assuming that the main 
way to inject a namespace is by requesting a FeatureSource/Schema with a 
specified URI + feature name, then either working with the CDS.getSchema 
or the CFS.getSchema should give us some room to address this generally.


https://github.com/geotools/geotools/blob/master/modules/library/main/src/main/java/org/geotools/data/DataUtilities.java#L1948

(1)  Here's the ticket where we addressed a similar ticket: 
https://github.com/locationtech/geomesa/commit/6a30b184e784d8a66a0e78c7aa71d4620804170b 
Please excuse our Scala;)  Also this addresses the problem in the 
context of the old, deprecated AbstractDataStore.


Thanks,

Jim

On 05/31/2015 10:24 AM, Andrea Aime [Administrator] (JIRA) wrote:

Message Title
	Andrea Aime [Administrator] 
https://osgeo-org.atlassian.net/secure/ViewProfile.jspa?name=aaime 
*created* an issue


GeoTools https://osgeo-org.atlassian.net/browse/GEOT / Bug 
https://osgeo-org.atlassian.net/browse/GEOT-5139 GEOT-5139 
https://osgeo-org.atlassian.net/browse/GEOT-5139
CSV datastore won't honor the configured namespace URI 
https://osgeo-org.atlassian.net/browse/GEOT-5139


Issue Type: Bug Bug
Assignee: 	Andrea Aime [Administrator] 
https://osgeo-org.atlassian.net/secure/ViewProfile.jspa?name=aaime

Components: csv
Created:31/May/15 4:23 PM
Priority:   Medium Medium
Reporter: 	Andrea Aime [Administrator] 
https://osgeo-org.atlassian.net/secure/ViewProfile.jspa?name=aaime


The namespace parameter is not advertised, and even when provided, 
it's not actually set in the datastore


Add Comment 
https://osgeo-org.atlassian.net/browse/GEOT-5139#add-comment 	Add 
Comment https://osgeo-org.atlassian.net/browse/GEOT-5139#add-comment


This message was sent by Atlassian JIRA 
(v6.5-OD-04-052#65000-sha1:8b5a8f9) 	

Atlassian logo



--


___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel


--
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] milestone release dependency roundup

[Jim Hughes]

I replied about the same topic on the GeoServer-devel.  For visibility, 
I'll mention things here as well...


Eclipse had approved hsqldb 2.3.0, so we happily used Maven to manage 
our version, and we didn't encounter any problems with that.


The biggest hsqldb issue we had is that Hadoop also pulls in various 
versions of hsqldb.  One work-around would be for downstream projects to 
shade hsqldb as they ship gt-epsg-hsql.


Cheers,

Jim

On 05/13/2015 03:45 PM, Jody Garnett wrote:

Taking this into a seperate discussion:


* hsqldb 2.2.8 - 2.3

References:
- geotools builds locally for me with a 2.3.0 dependency
- http://hsqldb.org/doc/2.0/changelist_2_0.txt (do not see any
important regressions here)


No objection, just wondering, what do we owe the upgrade to?


I suspect the IP team has reviewed a newer version and/or older 
version and has asked Tyler if it is possible to upgrade.


Checking ... GeoMesa used hsqldb 2.3 so it had been reviewed already 
(not sure why they needed that version)




--
One dashboard for servers and applications across Physical-Virtual-Cloud 
Widest out-of-the-box monitoring support with 50+ applications
Performance metrics, stats and reports that give you Actionable Insights
Deep dive visibility with transaction tracing using APM Insight.
http://ad.doubleclick.net/ddm/clk/290420510;117567292;y___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] Replacing Vecmath

[Jim Hughes]

Hi all,

I've had a chance to review some the PR for the matrix branch.

It looks like we'll need to follow up on

1. Any fixes for precision we want to sort out.

Jody clearly fought a decent battle with something there.  If this is 
unimportant, we can let it go.  Otherwise, this may require digging into 
the new and old implementations until we find a subtle thing.


2.  Removing unused classes and functions.

The MatrixN's clone functions may not all work.  (I confirmed that 
Matrix3.clone is broken.)


I'm a little unsure about how to view Point3d.  If it is sufficiently 
internal, I wonder about making it protected in some manner.  (This 
would require changing some unit tests.)


3.  Git history clean-up.

Andrea pointed out that there were multiple commits for the branch. I 
had goofed up while using a shared laptop and committed a few things as 
Alex rather than myself.


Anyhow, I think we are getting close.  Any feedback on the remaining 
issues is definitely appreciated.  I'm happy to contribute some more, 
and I'd like to make sure that we don't duplicate efforts.


Cheers,

Jim

On 04/29/2015 08:23 AM, Jody Garnett wrote:

Thanks - that explains it - I was indeed running the tests from eclipse.

--
Jody Garnett

On 29 April 2015 at 03:22, Andrea Aime andrea.a...@geo-solutions.it 
mailto:andrea.a...@geo-solutions.it wrote:


Jody, we have not been using the sql in a long time, it has been
replaced by the zipped
binary indeed.
However, the maven builds fine.

Are you trying to use those classes you added back then, that have
been de facto unsupported
since then? They have been dead code for a long time, we should
probably drop them?
For reference, the usage of the zip file predates the history we
have in git, that starts January 2011

It indeed looking at the code it seems HsqlEpsgDatabase is part of
that lot, and the tests that
are failing (I've run them from eclipse) are explicitly excluded
in the pom.xml:
configuration
  excludes
!-- disable the mediator tests, they are all broken
by the EPSG database upgrade --
exclude**/*Mediator*.java/exclude
exclude**/HsqlDialectEpsgFactoryTest.java/exclude
  /excludes
/configuration

Looks like I kicked them out at revision 33776, back in 2009:

-
r33776 | aaime | 2009-08-20 15:42:09 +0200 (gio, 20 ago 2009) | 1 line

Update the datase to 7.1 and make its creation faster and more
solid, GEOT-2667 and GEOT-2493
-

Cheers
Andrea


On Wed, Apr 29, 2015 at 4:52 AM, Jody Garnett
jody.garn...@gmail.com mailto:jody.garn...@gmail.com wrote:

Got a question .. perhaps for Andrea. I think we may be
running into problems based on changing serialization.

The HsqlEPSGDatabase is failing here:

final BufferedReader in = new BufferedReader(new
InputStreamReader(
HsqlEpsgDatabase.class.getResourceAsStream(EPSG.sql),
ISO-8859-1)); -- throws NullPointerException

The stack trace is:

java.lang.NullPointerException
at java.io.Reader.init(Reader.java:78)
at java.io.InputStreamReader.init(InputStreamReader.java:97)
at

org.geotools.referencing.factory.epsg.HsqlEpsgDatabase.generateData(HsqlEpsgDatabase.java:150)
at

org.geotools.referencing.factory.epsg.HsqlEpsgDatabase.createDataSource(HsqlEpsgDatabase.java:124)
at

org.geotools.referencing.factory.epsg.HsqlEpsgDatabase.createDataSource(HsqlEpsgDatabase.java:88)

I could not find the EPSG.sql stream referenced, there is an
EPSG.zip (which does not contain EPSG.sql) and an
EPSg_Indexes.sql.

There is an DatabaseCreationScript which a zipped file
containing several sql files, none of which are named EPSG.sql.



--
Jody Garnett

On 28 April 2015 at 22:02, Jim Hughes jn...@ccri.com
mailto:jn...@ccri.com wrote:

Jody,

Thanks!  Shout if there's something else I can help with.

Jim


On 04/28/2015 11:36 AM, Jody Garnett wrote:

Thanks Jim, I have merged your pull request - and rebased
that branch (so you have a tricky git fetch to perform).
I am going to go retrace tyler's steps on removing all
references to Vecmath and cleaning up any stray class
references.



--
Jody Garnett

On 20 April 2015 at 15:12, Jody Garnett
jody.garn...@gmail.com mailto:jody.garn...@gmail.com
wrote:

I think we have implemented XMatrix now, placed
checkboxes on the tasks to indicate how far we are along.

--
Jody Garnett

On 20 April 2015 at 11:10, Jim Hughes jn...@ccri.com
mailto:jn

Re: [Geotools-devel] Replacing Vecmath

[Jim Hughes]

Jody,

Thanks!  Shout if there's something else I can help with.

Jim

On 04/28/2015 11:36 AM, Jody Garnett wrote:
Thanks Jim, I have merged your pull request - and rebased that branch 
(so you have a tricky git fetch to perform). I am going to go retrace 
tyler's steps on removing all references to Vecmath and cleaning up 
any stray class references.




--
Jody Garnett

On 20 April 2015 at 15:12, Jody Garnett jody.garn...@gmail.com 
mailto:jody.garn...@gmail.com wrote:


I think we have implemented XMatrix now, placed checkboxes on the
tasks to indicate how far we are along.

--
Jody Garnett

On 20 April 2015 at 11:10, Jim Hughes jn...@ccri.com
mailto:jn...@ccri.com wrote:

Jody,

Thanks.  If I'm reading it right, the code sprint and my PR
should finish up 3.

Jim


On 04/20/2015 02:07 PM, Jody Garnett wrote:

We have tracked down the other vecmath use (see earlier email).

I have updated the tasks section on the proposal:
https://github.com/geotools/geotools/wiki/change-from-vecmath-to-ejml

--
Jody Garnett

On 20 April 2015 at 11:04, Jim Hughes jn...@ccri.com
mailto:jn...@ccri.com wrote:

Hi all,

I know the release and moving from Codehaus are likely
taking lots of folks' cycles...  I wanted to bump this
thread.  What kind of timeline are we on for getting the
matrix branch merged into master?

I've got a pr for fixing the outstanding build issues
right around GeneralMatrix here:
https://github.com/geotools/geotools/pull/813. Jody
mentioned that Tyler might have some time to track down
other vecmath usage.

After that, is there anything outstanding?

Thanks in advance,

Jim


On 04/10/2015 01:43 AM, Jody Garnett wrote:

IllegalArgumentException is good (and the message is
descriptive enough).

We are generally okay with adding new features to
master, and then after a month back porting (as long as
the change is additive and does not effect
compatibility). This is however a pretty major change,
which does effect API for downstream apps.

I have no ability to predict LocationTech IP timelines
(sigh) - if needed we can make a 14-M1 milestone release
to the maven repository.



--
Jody Garnett

On 9 April 2015 at 15:08, Jim Hughes jn...@ccri.com
mailto:jn...@ccri.com wrote:

Hi Jody,

Yeah, I'm glad that I'm caught up enough to work on
this again!

EJML throws an IllegalArgumentException rather than
a MismatchedSizeException when trying to invert an n
x m matrix (where n != m).  If that change is
sensible, we can use it; otherwise, there's already
a NoninvertibleTransformException.

Do we think we're in a good spot for this to make it
into GeoTools 13.1?

Cheers,

Jim


On 04/09/2015 05:56 PM, Jody Garnett wrote:

Great to hear from you Jim :) Proposal is being
transferred to github

(https://github.com/geotools/geotools/wiki/change-from-vecmath-to-ejml)
so you could not update it right now anyways :)

There were a couple more tasks in the proposal,
Tyler was working on removing the vecmath
dependency (providing us with some exceptions and
data structures to replace).

From earlier in this email thread - we need to find
replacements for the following:

* Point3d - can probably use one of the java
classes or roll our own here
* MismatchedSizeException - IllegalStateException
may work or perhaps we have a CRS exception we can
recycle...

Example of broken files:

* org.geotools.math.Line
* org.geotools.math.Plane



--
Jody Garnett

On 9 April 2015 at 13:50, Jim Hughes
jn...@ccri.com mailto:jn...@ccri.com wrote:

Hi all,

I finally grabbed a few minutes to bang away on
the vecmath transition; check out a PR into the
matrix branch here:
https://github.com/geotools/geotools/pull/813.
I tracked down the existing test failures; 3
changes to GeneralMatrix.java and an update of
the exception to catch in
ProjectiveTransform.java made

Re: [Geotools-devel] JAI Logging once more

[Jim Hughes]

Jody,

No problem, it looks great now.  As a sanity check, I did the following:

I pulled the head of GeoTools master, verified that it had the 
reflection fix, and built GeoTools locally.


Then, I built a branch of GeoMesa which pointed to 14-SNAPSHOT.  I 
sorted a build error related to an updated interface and noted a failing 
test where we were doing something with GML without a proper namespace.


From our unit tests, I noted the following messages:

Logging JAI messages: javax.media.jai logger redirected
Logging JAI messages: Unable to redirect to javax.media.jai

Since we expect to 'work with' JAI at times, the first message is 
expected in our modules which have jai_core as a test dependency. 
Otherwise, without JAI as a test dependency, the other message was observed.


I think this covers things thoroughly; there may be some more GeoMesa 
work to help users know when they are doing something which requires JAI.


Thanks,

Jim

On 04/22/2015 10:40 PM, Jody Garnett wrote:
Cleaned up to be pure reflection based, merged in and a note added to 
the docs: 
http://docs.geotools.org/latest/userguide/library/metadata/geotools.html


(I would of course love you to test this one, you should be able to 
watch it redirect by adding -DLOGGING_TRACE=true on the command line)


--
Jody Garnett

On 21 April 2015 at 12:37, Jody Garnett jody.garn...@gmail.com 
mailto:jody.garn...@gmail.com wrote:


I would like to revert to the reflection based solution, just so
it stays solved. I know it is not pretty, but since when is
logging code pretty.

--
Jody Garnett

On 21 April 2015 at 11:44, Jim Hughes jn...@ccri.com
mailto:jn...@ccri.com wrote:

Hi all,

As a background, I'm trying to use GeoTools while excluding
jai-core as
a dependency.  This will likely be the required state of
LocationTech
projects using GeoTools.

I goofed testing Jody's initial solution to the problem after
a final
commit was added which used less reflection.  I definitely
appreciate
the goal of relying on reflection as little as possible, but
it may be
one of the easier solutions to this problem.

Anyhow, as I was testing, if a call to jai.setImagingListener
is present
without jai-core on the class path, then any calls into
org.geotools.util.logging.Logging will throw an ClassNotFound for
javax.media.jai.util.ImagingListener.*

As I see it, there are two options:

1.  Use reflection.  I pushed up a PR which does that here:
https://github.com/geotools/geotools/pull/826.**

2.  Construct a second class (say JAILogging) which will have
a static
method to configure JAI.  In Logging.java, if Class.forName
succeeds,
delegate to a static method in that makes the actual call to
setImagingListener.  I haven't tried this out, but it may work
since the
ClassLoader will never try to look for ImagingListener. Plus
side, less
reflection.  Downside, we'd have to be careful about calling
that class
from outside of a 'safe' scope.

Anyhow, I apologize for not testing correctly in the first place.

Thoughts?

Jim

*

https://github.com/geotools/geotools/blob/master/modules/library/metadata/src/main/java/org/geotools/util/logging/Logging.java#L122

NB: I tried adding
Class.forName(javax.media.jai.util.ImagingListener)
and that didn't help.

Original PR: https://github.com/geotools/geotools/pull/731






--
BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT
Develop your own process in accordance with the BPMN 2 standard
Learn Process modeling best practices with Bonita BPM through
live exercises
http://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual-
event?utm_
source=Sourceforge_BPM_Camp_5_6_15utm_medium=emailutm_campaign=VA_SF
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
mailto:GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel





--
BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT
Develop your own process in accordance with the BPMN 2 standard
Learn Process modeling best practices with Bonita BPM through live exercises
http://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual- event?utm_
source=Sourceforge_BPM_Camp_5_6_15utm_medium=emailutm_campaign=VA_SF___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists

[Geotools-devel] JAI Logging once more

[Jim Hughes]

Hi all,

As a background, I'm trying to use GeoTools while excluding jai-core as 
a dependency.  This will likely be the required state of LocationTech 
projects using GeoTools.

I goofed testing Jody's initial solution to the problem after a final 
commit was added which used less reflection.  I definitely appreciate 
the goal of relying on reflection as little as possible, but it may be 
one of the easier solutions to this problem.

Anyhow, as I was testing, if a call to jai.setImagingListener is present 
without jai-core on the class path, then any calls into 
org.geotools.util.logging.Logging will throw an ClassNotFound for 
javax.media.jai.util.ImagingListener.*

As I see it, there are two options:

1.  Use reflection.  I pushed up a PR which does that here: 
https://github.com/geotools/geotools/pull/826.**

2.  Construct a second class (say JAILogging) which will have a static 
method to configure JAI.  In Logging.java, if Class.forName succeeds, 
delegate to a static method in that makes the actual call to 
setImagingListener.  I haven't tried this out, but it may work since the 
ClassLoader will never try to look for ImagingListener. Plus side, less 
reflection.  Downside, we'd have to be careful about calling that class 
from outside of a 'safe' scope.

Anyhow, I apologize for not testing correctly in the first place.

Thoughts?

Jim

* 
https://github.com/geotools/geotools/blob/master/modules/library/metadata/src/main/java/org/geotools/util/logging/Logging.java#L122

NB: I tried adding Class.forName(javax.media.jai.util.ImagingListener) 
and that didn't help.

Original PR: https://github.com/geotools/geotools/pull/731





--
BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT
Develop your own process in accordance with the BPMN 2 standard
Learn Process modeling best practices with Bonita BPM through live exercises
http://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual- event?utm_
source=Sourceforge_BPM_Camp_5_6_15utm_medium=emailutm_campaign=VA_SF
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] Replacing Vecmath

[Jim Hughes]

Jody,

Thanks.  If I'm reading it right, the code sprint and my PR should 
finish up 3.


Jim

On 04/20/2015 02:07 PM, Jody Garnett wrote:

We have tracked down the other vecmath use (see earlier email).

I have updated the tasks section on the proposal: 
https://github.com/geotools/geotools/wiki/change-from-vecmath-to-ejml


--
Jody Garnett

On 20 April 2015 at 11:04, Jim Hughes jn...@ccri.com 
mailto:jn...@ccri.com wrote:


Hi all,

I know the release and moving from Codehaus are likely taking lots
of folks' cycles...  I wanted to bump this thread.  What kind of
timeline are we on for getting the matrix branch merged into master?

I've got a pr for fixing the outstanding build issues right around
GeneralMatrix here: https://github.com/geotools/geotools/pull/813.
Jody mentioned that Tyler might have some time to track down other
vecmath usage.

After that, is there anything outstanding?

Thanks in advance,

Jim


On 04/10/2015 01:43 AM, Jody Garnett wrote:

IllegalArgumentException is good (and the message is descriptive
enough).

We are generally okay with adding new features to master, and
then after a month back porting (as long as the change is
additive and does not effect compatibility). This is however a
pretty major change, which does effect API for downstream apps.

I have no ability to predict LocationTech IP timelines (sigh) -
if needed we can make a 14-M1 milestone release to the maven
repository.



--
Jody Garnett

On 9 April 2015 at 15:08, Jim Hughes jn...@ccri.com
mailto:jn...@ccri.com wrote:

Hi Jody,

Yeah, I'm glad that I'm caught up enough to work on this again!

EJML throws an IllegalArgumentException rather than a
MismatchedSizeException when trying to invert an n x m matrix
(where n != m).  If that change is sensible, we can use it;
otherwise, there's already a NoninvertibleTransformException.

Do we think we're in a good spot for this to make it into
GeoTools 13.1?

Cheers,

Jim


On 04/09/2015 05:56 PM, Jody Garnett wrote:

Great to hear from you Jim :) Proposal is being transferred
to github
(https://github.com/geotools/geotools/wiki/change-from-vecmath-to-ejml)
so you could not update it right now anyways :)

There were a couple more tasks in the proposal, Tyler was
working on removing the vecmath dependency (providing us
with some exceptions and data structures to replace).

From earlier in this email thread - we need to find
replacements for the following:

* Point3d - can probably use one of the java classes or roll
our own here
* MismatchedSizeException - IllegalStateException may work
or perhaps we have a CRS exception we can recycle...

Example of broken files:

* org.geotools.math.Line
* org.geotools.math.Plane



--
Jody Garnett

On 9 April 2015 at 13:50, Jim Hughes jn...@ccri.com
mailto:jn...@ccri.com wrote:

Hi all,

I finally grabbed a few minutes to bang away on the
vecmath transition; check out a PR into the matrix
branch here:
https://github.com/geotools/geotools/pull/813. I tracked
down the existing test failures; 3 changes to
GeneralMatrix.java and an update of the exception to
catch in ProjectiveTransform.java made everything go.

Additionally, I started a GeneralMatrixTest file.  It
isn't complete, and I'm hoping that its current state is
beneficial.  I am quite happy for someone else to pitch
in some more test cases; it was a helpful exercise is
understanding some of the details of the constructors
and various functions.

As an amusement, I noticed that getRow is private and
getCol is public.

Anyhow, apologies for not updating the proposal to
reflect the idea of adding test coverage for GeneralMatrix.

What else remains to get all this merged in?  If there
are any formatting concerns, lemme know.

Thanks,

Jim


On 03/14/2015 09:24 PM, Jody Garnett wrote:

I think you are doing the coordination right now on
this email thread ( the tasks section of the proposal
is also good).

I like your idea to make a general matrix testing
master - add that to the tasks section of the proposal
and stick your initials next to it if you are in
position to volunteer.

I hope other PMC members can review the plan and vote.
On Sat, Mar 14, 2015 at 3:12 PM Jim Hughes
jn...@ccri.com mailto:jn...@ccri.com wrote:

Hi all

Re: [Geotools-devel] Replacing Vecmath

[Jim Hughes]

Hi all,

I know the release and moving from Codehaus are likely taking lots of 
folks' cycles...  I wanted to bump this thread.  What kind of timeline 
are we on for getting the matrix branch merged into master?


I've got a pr for fixing the outstanding build issues right around 
GeneralMatrix here: https://github.com/geotools/geotools/pull/813. Jody 
mentioned that Tyler might have some time to track down other vecmath 
usage.


After that, is there anything outstanding?

Thanks in advance,

Jim

On 04/10/2015 01:43 AM, Jody Garnett wrote:

IllegalArgumentException is good (and the message is descriptive enough).

We are generally okay with adding new features to master, and then 
after a month back porting (as long as the change is additive and does 
not effect compatibility). This is however a pretty major change, 
which does effect API for downstream apps.


I have no ability to predict LocationTech IP timelines (sigh) - if 
needed we can make a 14-M1 milestone release to the maven repository.




--
Jody Garnett

On 9 April 2015 at 15:08, Jim Hughes jn...@ccri.com 
mailto:jn...@ccri.com wrote:


Hi Jody,

Yeah, I'm glad that I'm caught up enough to work on this again!

EJML throws an IllegalArgumentException rather than a
MismatchedSizeException when trying to invert an n x m matrix
(where n != m).  If that change is sensible, we can use it;
otherwise, there's already a NoninvertibleTransformException.

Do we think we're in a good spot for this to make it into GeoTools
13.1?

Cheers,

Jim


On 04/09/2015 05:56 PM, Jody Garnett wrote:

Great to hear from you Jim :) Proposal is being transferred to
github
(https://github.com/geotools/geotools/wiki/change-from-vecmath-to-ejml)
so you could not update it right now anyways :)

There were a couple more tasks in the proposal, Tyler was working
on removing the vecmath dependency (providing us with some
exceptions and data structures to replace).

From earlier in this email thread - we need to find replacements
for the following:

* Point3d - can probably use one of the java classes or roll our
own here
* MismatchedSizeException - IllegalStateException may work or
perhaps we have a CRS exception we can recycle...

Example of broken files:

* org.geotools.math.Line
* org.geotools.math.Plane



--
Jody Garnett

On 9 April 2015 at 13:50, Jim Hughes jn...@ccri.com
mailto:jn...@ccri.com wrote:

Hi all,

I finally grabbed a few minutes to bang away on the vecmath
transition; check out a PR into the matrix branch here:
https://github.com/geotools/geotools/pull/813. I tracked down
the existing test failures; 3 changes to GeneralMatrix.java
and an update of the exception to catch in
ProjectiveTransform.java made everything go.

Additionally, I started a GeneralMatrixTest file.  It isn't
complete, and I'm hoping that its current state is
beneficial.  I am quite happy for someone else to pitch in
some more test cases; it was a helpful exercise is
understanding some of the details of the constructors and
various functions.

As an amusement, I noticed that getRow is private and getCol
is public.

Anyhow, apologies for not updating the proposal to reflect
the idea of adding test coverage for GeneralMatrix.

What else remains to get all this merged in?  If there are
any formatting concerns, lemme know.

Thanks,

Jim


On 03/14/2015 09:24 PM, Jody Garnett wrote:

I think you are doing the coordination right now on this
email thread ( the tasks section of the proposal is also good).

I like your idea to make a general matrix testing master -
add that to the tasks section of the proposal and stick your
initials next to it if you are in position to volunteer.

I hope other PMC members can review the plan and vote.
On Sat, Mar 14, 2015 at 3:12 PM Jim Hughes jn...@ccri.com
mailto:jn...@ccri.com wrote:

Hi all,

Jody, Tyler, and I hacked at this on Friday during a
code sprint.  The partial results are here:
https://github.com/geotools/geotools/commits/matrix

There are a few remaining errors to track down, and the
cautious part of me wants to go back and cook up a set
of unit tests for GeneralMatrix.

Anyhow, since we are all traveling for the next few
days, I wanted to ask what the best way to coordinate
future effort is?

If anyone else is interested pitching in, let us know.

Thanks,

Jim

p.s.  The commits from an 'Alex Tatusko' are from me.  I
goofed up and didn't change the Git info on a loaner laptop.


On 03/13/2015 03:55 PM

Re: [Geotools-devel] Replacing Vecmath

[Jim Hughes]

Hi all,

I finally grabbed a few minutes to bang away on the vecmath transition; 
check out a PR into the matrix branch here: 
https://github.com/geotools/geotools/pull/813.  I tracked down the 
existing test failures; 3 changes to GeneralMatrix.java and an update of 
the exception to catch in ProjectiveTransform.java made everything go.


Additionally, I started a GeneralMatrixTest file.  It isn't complete, 
and I'm hoping that its current state is beneficial.  I am quite happy 
for someone else to pitch in some more test cases; it was a helpful 
exercise is understanding some of the details of the constructors and 
various functions.


As an amusement, I noticed that getRow is private and getCol is public.

Anyhow, apologies for not updating the proposal to reflect the idea of 
adding test coverage for GeneralMatrix.


What else remains to get all this merged in?  If there are any 
formatting concerns, lemme know.


Thanks,

Jim

On 03/14/2015 09:24 PM, Jody Garnett wrote:
I think you are doing the coordination right now on this email thread 
( the tasks section of the proposal is also good).


I like your idea to make a general matrix testing master - add that to 
the tasks section of the proposal and stick your initials next to it 
if you are in position to volunteer.


I hope other PMC members can review the plan and vote.
On Sat, Mar 14, 2015 at 3:12 PM Jim Hughes jn...@ccri.com 
mailto:jn...@ccri.com wrote:


Hi all,

Jody, Tyler, and I hacked at this on Friday during a code sprint. 
The partial results are here:

https://github.com/geotools/geotools/commits/matrix

There are a few remaining errors to track down, and the cautious
part of me wants to go back and cook up a set of unit tests for
GeneralMatrix.

Anyhow, since we are all traveling for the next few days, I wanted
to ask what the best way to coordinate future effort is?

If anyone else is interested pitching in, let us know.

Thanks,

Jim

p.s.  The commits from an 'Alex Tatusko' are from me.  I goofed up
and didn't change the Git info on a loaner laptop.


On 03/13/2015 03:55 PM, Jody Garnett wrote:

A bit more research shows we should support the geotools Matrix
and XMatrix methods (giving us a limited set of methods to
implement). Proposal updated accordingly ...

--
Jody Garnett

On 13 March 2015 at 12:31, Andrea Aime
andrea.a...@geo-solutions.it
mailto:andrea.a...@geo-solutions.it wrote:

On Fri, Mar 13, 2015 at 7:57 PM, Jody Garnett
jody.garn...@gmail.com mailto:jody.garn...@gmail.com wrote:

At the foss4gna code sprint, here is a proposal to make
this change:

http://docs.codehaus.org/display/GEOTOOLS/Change+from+vecmath+to+EJML


Looking at this with Jim, the library has DenseMatrix64F
(which does the work) and an example of wrapping this up
for normal use called SimpleMatrix.

Each operation in SimpleMatrix makes a copy, an overhead
we could avoid by using DenseMatrix64F ourself. So the
proposal is to delegate to DenseMatrix64F, using
SimpleMatrix (and its friend BaseMatrix) as a guide.

Alternatively we could kill our GeneralMatrix class and
use SimpleMatrix directly in the rest of the GeoTools
codebase, it would be more an an API change - which is
okay if people only see this stuff inside MathTransforms.
Anyone got an idea on how visible GeneralMatrix is?


I don't... and guess we should try to keep it around.
Anyways, the direction appears to be sound, no objections.

Cheers
Andrea


-- 
==

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.
==

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054  Massarosa (LU)
Italy
phone: +39 0584 962313 tel:%2B39%200584%20962313
fax: +39 0584 1660272 tel:%2B39%200584%201660272
mob: +39  339 8844549 tel:%2B39%20%C2%A0339%208844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta
elettronica e/o nel/i file/s allegato/i sono da considerarsi
strettamente riservate. Il loro utilizzo è consentito
esclusivamente al destinatario del messaggio, per le finalità
indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo
cortesemente di darcene notizia via e-mail e di procedere
alla distruzione del messaggio stesso, cancellandolo dal
Vostro sistema. Conservare il messaggio stesso, divulgarlo
anche in parte

Re: [Geotools-devel] ECQL Irregularities

[Jim Hughes]

Kevin,

Hmm... does the Barnes Surface process create an ad hoc, intermediate 
feature type?


I think the issue may be here: 
https://github.com/geotools/geotools/blob/master/modules/library/cql/src/main/java/org/geotools/filter/text/commons/ExpressionToText.java#L215-221. 
PropertyNames are not escaped.  I fiddled with cooking up filters with 
Literals rather than PropertyNames, and those were escaped by code above.


Cheers,

Jim

On 03/30/2015 09:16 PM, Kevin Smith wrote:
On 30 March 2015 at 18:04, Jim Hughes jn...@ccri.com 
mailto:jn...@ccri.com wrote:


The EBNF does reference key words/reserved words.  That list
includes wkt keywords like POINT, POLYGON and predicates like
INTERSECTS and DURING.

A property name appears to get worked out as 'character string
literal', and would need to be quoted.  I think that would do the
trick.  Since the literal *point* is written sans qutoes, I'd
guess that the preference for keyword identification is too high? 
If that's the case, something in the parser may need a little nudge.


In general, I'd suggest that reserved words in a query language
make poor attribute names.;)


Yes but the CQL encoder should be able to deal with it when it 
happens.  What's particularly strange is that the specific example 
that brought this to my attention, the query is actually wrong in that 
the attribute in question doesn't even exist.  It was just being added 
to a raster symbolizer in a style using Barnes Surface for no apparent 
reason and when I converted it to CQL and then tried to parse the CQL, 
it broke.



Cheers,

Jim


On 03/30/2015 07:55 PM, Jody Garnett wrote:

So what is needed here ... list of reserved words that must be
escaped when encoded?

I think the square brackets are just to introduce precedence,
isolating point won't help since it will start trying to treat it
as geometry.

For reference the grammar is here:
http://old.geotools.org/ECQL-Parser-Design_110493908.html

--
Jody Garnett

On 30 March 2015 at 16:46, Kevin Smith ksm...@boundlessgeo.com
mailto:ksm...@boundlessgeo.com wrote:

I've noticed that ECQL.toCQL doesn't escape WKT keywords.  So
PropertyNamepoint/PropertyName is encoded to just point
rather than point or [point]. When this is parsed in turn,
it is interpreted as a WKT literal, although it's lacking the
rest of the expression and so fails.

The bracket notation also fails to resolve this when parsing,
it still interprets [point] as being the start of a WKT
literal rather than an attribute name.  When double quoted
(point) it escapes against the WKT interpretation as expected.

I may be missing something.  When I looked up the formal CQL
spec hidden in the CSW spec, i couldn't find any indication
of either bracket or double quote escaping of property
names.  Were these added for ECQL?

-- 


Kevin Smith

Software Engineer | Boundless http://boundlessgeo.com/

ksm...@boundlessgeo.com mailto:ksm...@boundlessgeo.com

+1-778-785-7459 tel:%2B1-778-785-7459

@boundlessgeo http://twitter.com/boundlessgeo/


http://twitter.com/boundlessgeo/

http://boundlessgeo.com/
http://boundlessgeo.com/



--
Dive into the World of Parallel Programming The Go Parallel
Website, sponsored
by Intel and developed in partnership with Slashdot Media, is
your hub for all
things parallel software development, from weekly thought
leadership blogs to
news, videos, case studies, tutorials and more. Take a look
and join the
conversation now. http://goparallel.sourceforge.net/
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
mailto:GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel





--
Dive into the World of Parallel Programming The Go Parallel Website, 
sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for 
all
things parallel software development, from weekly thought leadership blogs 
to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now.http://goparallel.sourceforge.net/


___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net  
mailto:GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel




--
Dive into the World of Parallel

Re: [Geotools-devel] New JIRA account request

[Jim Hughes]

Are the previous accounts being transferred?  Or will we all need to 
request new accounts?


Thanks for handling the transition!

Jim

On 03/27/2015 02:28 PM, Jody Garnett wrote:
There is an option to let users sign up, but I am scared to turn it on 
due to spammers. I will set up PSC members (and committers?) with the 
ability to add users.


Signing you up now.


--
Jody Garnett

On 27 March 2015 at 11:24, Travis Brundage tbrund...@boundlessgeo.com 
mailto:tbrund...@boundlessgeo.com wrote:


Hello,

With the migration from codehaus about to occur, I wanted to
request an account on the new JIRA (
https://osgeo-org.atlassian.net/ ). Can you please hook me up,
Jody? I don't see a way to sign up there myself.

Thanks,
Travis Brundage

-- 
Travis Brundage

Software Engineer | Boundless
tbrund...@boundlessgeo.com mailto:tbrund...@boundlessgeo.com
250.888.2820 tel:250.888.2820
@boundlessgeo



--
Dive into the World of Parallel Programming The Go Parallel
Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your
hub for all
things parallel software development, from weekly thought
leadership blogs to
news, videos, case studies, tutorials and more. Take a look and
join the
conversation now. http://goparallel.sourceforge.net/
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
mailto:GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel




--
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/


___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel


--
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the 
conversation now. http://goparallel.sourceforge.net/___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] Another ECQL.toFilter question

[Jim Hughes]

Jody,

I'm thinking of this one: https://github.com/geotools/geotools/pull/731

I tried starting a separate thread for it, but there's been a little 
more traffic on the devel list today.


Cheers,

Jim

On 03/17/2015 02:26 PM, Jody Garnett wrote:

Um, which were you thinking of Jim?

--
Jody Garnett

On 17 March 2015 at 10:36, Jim Hughes jn...@ccri.com 
mailto:jn...@ccri.com wrote:


Hi Jody,

I just confirmed briefly: Who != 'Bierce' does not parse with
ECQL.toFilter; on the other hand, Who  'Bierce' is just fine.

Thanks for the merge and JIRA ticket.

This one happens to be convenient to get into the release.  I
didn't realize that your fix from the OSGeo code sprint didn't get
merged.  If the community can reach a happy place with that before
13.0 is cut, I'd be grateful.  If not, that's fine as well.

Thanks,

Jim


On 03/17/2015 01:24 PM, Jody Garnett wrote:

Hi Jim,

Well we should certainly be correct - how was it working before?
Does the parser accept both != and  ?

I am content to grab this pull request on to master, but this is
very close.

Here is your jira ticket for the release notes:
https://jira.codehaus.org/browse/GEOT-5051



--
Jody Garnett

On 17 March 2015 at 09:35, Jim Hughes jn...@ccri.com
mailto:jn...@ccri.com wrote:

Hi all,

Last Friday, some of my fellow GeoMesans noticed that
ECQL.toFilter
writes out PropertyIsNotEqual with != rather than  (which
is what the
EBNF specifies).  Anyhow, Chris tossed up a PR here:
https://github.com/geotools/geotools/pull/772.

Anyhow, I wanted to start a discussion of the 'correct'
behavior of
toFilter and also to bump the PR.

Thanks in advance,

Jim


--
Dive into the World of Parallel Programming The Go Parallel
Website, sponsored
by Intel and developed in partnership with Slashdot Media, is
your hub for all
things parallel software development, from weekly thought
leadership blogs to
news, videos, case studies, tutorials and more. Take a look
and join the
conversation now. http://goparallel.sourceforge.net/
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
mailto:GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel







--
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the 
conversation now. http://goparallel.sourceforge.net/___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] JAI Logging PR

[Jim Hughes]

Thanks, I am happy to test the PR.

I think we'll be able to help test releases more in the future.

Thanks again,

Jim

On 03/17/2015 05:20 PM, Jody Garnett wrote:
I think I can clean up that pull request based on feedback, if you are 
in position to test it would be good.


(Having such a long gap between the beta and the release is not doing 
us much good - we really need to get RC1 tested)



--
Jody Garnett

On 17 March 2015 at 10:21, Jim Hughes jn...@ccri.com 
mailto:jn...@ccri.com wrote:


Hi all,

Back at the OSGeo/LocationTech code sprint, I tried out GeoMesa with
GeoTools 13.x.  LocationTech is not allowing us to ship JAI, and
we ran
into some issues as GeoTools tried to configure JAI logging
without JAI.

Jody had submitted a PR to fix an issue when JAI is not present
(https://github.com/geotools/geotools/pull/731/commits). Since the
configuration is now a big try block, would it make sense to skip the
reflection?  (I think that's what Andrea was getting at in his
comment.)

Thanks all,

Jim


--
Dive into the World of Parallel Programming The Go Parallel
Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your
hub for all
things parallel software development, from weekly thought
leadership blogs to
news, videos, case studies, tutorials and more. Take a look and
join the
conversation now. http://goparallel.sourceforge.net/
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
mailto:GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel




--
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the 
conversation now. http://goparallel.sourceforge.net/___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] Another ECQL.toFilter question

[Jim Hughes]

Hi Jody,

I just confirmed briefly: Who != 'Bierce' does not parse with 
ECQL.toFilter; on the other hand, Who  'Bierce' is just fine.


Thanks for the merge and JIRA ticket.

This one happens to be convenient to get into the release.  I didn't 
realize that your fix from the OSGeo code sprint didn't get merged. If 
the community can reach a happy place with that before 13.0 is cut, I'd 
be grateful.  If not, that's fine as well.


Thanks,

Jim

On 03/17/2015 01:24 PM, Jody Garnett wrote:

Hi Jim,

Well we should certainly be correct - how was it working before? Does 
the parser accept both != and  ?


I am content to grab this pull request on to master, but this is very 
close.


Here is your jira ticket for the release notes: 
https://jira.codehaus.org/browse/GEOT-5051




--
Jody Garnett

On 17 March 2015 at 09:35, Jim Hughes jn...@ccri.com 
mailto:jn...@ccri.com wrote:


Hi all,

Last Friday, some of my fellow GeoMesans noticed that ECQL.toFilter
writes out PropertyIsNotEqual with != rather than  (which is
what the
EBNF specifies).  Anyhow, Chris tossed up a PR here:
https://github.com/geotools/geotools/pull/772.

Anyhow, I wanted to start a discussion of the 'correct' behavior of
toFilter and also to bump the PR.

Thanks in advance,

Jim


--
Dive into the World of Parallel Programming The Go Parallel
Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your
hub for all
things parallel software development, from weekly thought
leadership blogs to
news, videos, case studies, tutorials and more. Take a look and
join the
conversation now. http://goparallel.sourceforge.net/
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
mailto:GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel




--
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the 
conversation now. http://goparallel.sourceforge.net/___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

[Geotools-devel] JAI Logging PR

[Jim Hughes]

Hi all,

Back at the OSGeo/LocationTech code sprint, I tried out GeoMesa with 
GeoTools 13.x.  LocationTech is not allowing us to ship JAI, and we ran 
into some issues as GeoTools tried to configure JAI logging without JAI.

Jody had submitted a PR to fix an issue when JAI is not present 
(https://github.com/geotools/geotools/pull/731/commits). Since the 
configuration is now a big try block, would it make sense to skip the 
reflection?  (I think that's what Andrea was getting at in his comment.)

Thanks all,

Jim

--
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the 
conversation now. http://goparallel.sourceforge.net/
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

[Geotools-devel] Another ECQL.toFilter question

[Jim Hughes]

Hi all,

Last Friday, some of my fellow GeoMesans noticed that ECQL.toFilter 
writes out PropertyIsNotEqual with != rather than  (which is what the 
EBNF specifies).  Anyhow, Chris tossed up a PR here: 
https://github.com/geotools/geotools/pull/772.

Anyhow, I wanted to start a discussion of the 'correct' behavior of 
toFilter and also to bump the PR.

Thanks in advance,

Jim

--
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the 
conversation now. http://goparallel.sourceforge.net/
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] Replacing Vecmath

[Jim Hughes]

Hi all,

Jody, Tyler, and I hacked at this on Friday during a code sprint. The 
partial results are here: 
https://github.com/geotools/geotools/commits/matrix


There are a few remaining errors to track down, and the cautious part of 
me wants to go back and cook up a set of unit tests for GeneralMatrix.


Anyhow, since we are all traveling for the next few days, I wanted to 
ask what the best way to coordinate future effort is?


If anyone else is interested pitching in, let us know.

Thanks,

Jim

p.s.  The commits from an 'Alex Tatusko' are from me.  I goofed up and 
didn't change the Git info on a loaner laptop.


On 03/13/2015 03:55 PM, Jody Garnett wrote:
A bit more research shows we should support the geotools Matrix and 
XMatrix methods (giving us a limited set of methods to implement). 
Proposal updated accordingly ...


--
Jody Garnett

On 13 March 2015 at 12:31, Andrea Aime andrea.a...@geo-solutions.it 
mailto:andrea.a...@geo-solutions.it wrote:


On Fri, Mar 13, 2015 at 7:57 PM, Jody Garnett
jody.garn...@gmail.com mailto:jody.garn...@gmail.com wrote:

At the foss4gna code sprint, here is a proposal to make this
change:
http://docs.codehaus.org/display/GEOTOOLS/Change+from+vecmath+to+EJML


Looking at this with Jim, the library has DenseMatrix64F
(which does the work) and an example of wrapping this up for
normal use called SimpleMatrix.

Each operation in SimpleMatrix makes a copy, an overhead we
could avoid by using DenseMatrix64F ourself. So the proposal
is to delegate to DenseMatrix64F, using SimpleMatrix (and its
friend BaseMatrix) as a guide.

Alternatively we could kill our GeneralMatrix class and use
SimpleMatrix directly in the rest of the GeoTools codebase, it
would be more an an API change - which is okay if people only
see this stuff inside MathTransforms. Anyone got an idea on
how visible GeneralMatrix is?


I don't... and guess we should try to keep it around. Anyways, the
direction appears to be sound, no objections.

Cheers
Andrea


-- 
==

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.
==

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054  Massarosa (LU)
Italy
phone: +39 0584 962313 tel:%2B39%200584%20962313
fax: +39 0584 1660272 tel:%2B39%200584%201660272
mob: +39  339 8844549 tel:%2B39%20%C2%A0339%208844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica
e/o nel/i file/s allegato/i sono da considerarsi strettamente
riservate. Il loro utilizzo è consentito esclusivamente al
destinatario del messaggio, per le finalità indicate nel messaggio
stesso. Qualora riceviate questo messaggio senza esserne il
destinatario, Vi preghiamo cortesemente di darcene notizia via
e-mail e di procedere alla distruzione del messaggio stesso,
cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti,
copiarlo, od utilizzarlo per finalità diverse, costituisce
comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended
solely for the attention and use of the named addressee(s) and may
be confidential or proprietary in nature or covered by the
provisions of privacy act (Legislative Decree June, 30 2003,
no.196 - Italy's New Data Protection Code).Any use not in accord
with its purpose, any disclosure, reproduction, copying,
distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please
contact immediately the sender by telephone, fax or e-mail and
delete the information in this message that has been received in
error. The sender does not give any warranty or accept liability
as the content, accuracy or completeness of sent messages and
accepts no responsibility  for changes made after they were sent
or for other risks which arise as a result of e-mail transmission,
viruses, etc.


---




--
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/

Re: [Geotools-devel] GeoTools / GeoServer Meeting 2015-03-03

[Jim Hughes]

When/where is the mini-sprint going to be?

Jody floated the idea of doing something on Friday, and I'd be up for 
hanging out and working on the various GeoTools/GeoServer things I'm 
interested in.

Cheers,

Jim

On 03/03/2015 01:47 PM, Ben Caradoc-Davies wrote:
 foss4gna mini-sprint
 

 Who is going: Andrea, Jody, Travis, Torbien

 Going to focus on running CITE tests and cleaning up the mess.

 Goal:
 - This time get one CITE test running on the build server with new CITE
 Engine (don't care if passing or not)
 - Way to specify which version of CITE test to run .. github revision or
 something.
 - A few people running CITE test locally for live debugging


--
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the 
conversation now. http://goparallel.sourceforge.net/
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] distributed rendering

[Jim Hughes]

Hi all,

I'd like to try and summarize the discussion so far to make sure that I 
understand.


As a request comes into GeoServer, it would either be handled by a 
custom GetMapCallback or by a distributed-render-aware Renderer by 
GetMap.  From there, the distributed GeoTools data/feature store would 
accept sufficient rendering hints so that it could return artificial 
SimpleFeatures containing a RenderedImage.  On GeoServer, those multiple 
RenderedImages would be combined into the final image to satisfy the WMS 
request.


In terms of how the initial request is handled, I am a little unclear on 
the choices here.  I believe we can and should restrict ourselves to 
SLDs which are 'easy to distribute'.  I think trading some style options 
for fast WMS responses with data sets involving billions of features is 
reasonable.  As we get a better handle on system preformance, I'd be 
interested in supporting more styling as we zoom into a range which we 
know we can range render quickly enough.


At the moment, can we ask the Style classes how they can be simplified 
to be 'distributable'?


In terms of the distribution of work, I do like the idea of pushing tile 
rendering to the cloud nodes.  From the GeoTools side, is there anything 
we can/should be doing to make the process of passing hints and getting 
back synthetic SimpleFeatures more clear?  GeoMesa does this in a few 
places to help generate heatmaps and time series.  (I mention that since 
inserting ourselves in the GetMap step will likely let us call out to 
the database directly and return RenderedImages more cleanly.)


Andrea, I'll support Jody's multi-threaded/parallel rendering ideas.  
Chris and Rich have shared results from a 5 node cluster. If I am 
understanding correctly, each of those servers will produce a 
RenderedImage with part of the final image.  In particular, I don't 
expect the rendered images to overlap all that much.  One server would 
have data for the Americas and another would paint the ranges for 
Europe.  GeoMesa uses a random sharding approach, so when we do this, 
our tiles may look very similar.


Anyhow, on the GeoServer side, GeoWave and GeoMesa would both be 
providing at least one RenderedImage per cloud node.  If there are 
exactly 5 images, whatever approach would be fine.  If there are 
hundreds of images to merge, would having a concurrent merging process 
help?  I'd conjecture that a local process which would help uDig use 
multiple cores may also help our distributed rendering pipeline.


That's my understanding of the distributed rendering architecture so 
far.  I'm excited to see this come together!


Jim

On 02/15/2015 09:24 AM, Andrea Aime wrote:
On Tue, Feb 10, 2015 at 8:23 PM, Rich Fecher rfec...@gmail.com 
mailto:rfec...@gmail.com wrote:


Yeah, so on the GeoWave project we started out with a simple
sub-sampling approach that used a render transform to pass the
pixel dimensions of the map request to our data store so that we
could skip within our tablet servers at the equivalent of a pixel
resolution on our space filling curve.  This works well for basic
point data but for polygons and lines - data with more complex
stylization that does not well map what is actually rendered to
the gridded representation on our space filling curve, we take
advantage of actually doing the rendering within the tablet servers.


Hi,
so I've looked a bit into the code you have setup and tried to get a 
grip on it.


Here is what I've gathered so far.
You have setup a modified renderer that has a distributed rendering 
mode.
This is not really a different mode of operation of its own, but more 
of a way to offload the

entire rendering process of a layer to the underlying store.

I see you basically setup a helper object that contains all the 
information needed to
render the layer, styles, all rendering options, and so on, and you 
put such object into

a query hint that is given to the store.

The store can handle this special hint, and in case it's found, it 
will take the object
in the hint, and distribute it to all the nodes in your cluster, using 
a typical distributed
processing pattern, where you move your processing algorithm towards 
your data,

which is already suitably split among the various nodes.

Each node computes a RenderedImage using a local StreamingRenderer 
(which in
your case, it's really the object you put in the hint) and sends it 
back to the store, which in turn
generates fake features with a special attribute containing one of 
images we
got back from the nodes, the modified renderer will recognize them and 
basically just
paint this fake feature collection as a set of images to blend on top 
of the current map.


This sort of follows along existing ideas that stores willing to 
optimize rendering
can actively participate in some steps of it, right now we have hints 
to offload
geometry generalization 

Re: [Geotools-devel] failure building on ares (jjtree-javacc:jar:11-SNAPSHOT)

[Jim Hughes]

Hi all,

I'm generally interested in the CQL parser.  Is switching to the Maven 
mojo project something I could contribute in a few weeks? (I'd be doing 
it on personal time.)


Jim

On 08/16/2014 03:57 AM, Andrea Aime wrote:
On Sat, Aug 16, 2014 at 7:46 AM, Jody Garnett jody.garn...@gmail.com 
mailto:jody.garn...@gmail.com wrote:


I assume we have tried
http://mojo.codehaus.org/javacc-maven-plugin/


Don't know we did. Afaik the javacc plugin we use was developed
before that mojo project existed.

The mojo one works btw, I use it in GeoServer for some bits of the WCS 1.1
parsing, but I'm not sure how much work would be involved in using
it in GeoTools also.
Here is the GeoServer usage, for reference:

plugin
groupIdorg.codehaus.mojo/groupId
artifactIdjavacc-maven-plugin/artifactId
version2.3/version
executions
  execution
idjjtree/id
goals
  goaljjtree/goal
/goals
configuration
sourceDirectorysrc/main/java/org/geoserver/wcs/kvp/rangesubset/sourceDirectory
outputDirectorytarget/generated-sources/jjtree/outputDirectory
/configuration
  /execution
  execution
idjavacc/id
goals
  goaljavacc/goal
/goals
configuration
sourceDirectorytarget/generated-sources/jjtree//sourceDirectory
outputDirectorytarget/generated-sources/javacc/outputDirectory
/configuration
  /execution
/executions
  /plugin


Mauricio, do you know if we have any custom behavior that would make
it hard to switch to the mojo one? As far as I can see the jjtree-javacc
plugin is used only to build cql

Cheers
Andrea

--
==
GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.
==

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054  Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39  339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---


--


___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel


--
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

[Geotools-devel] Issue with ID filter serialization

[Jim Hughes]

Hi all,

I think I've found a small issue with how ID filters are serialized by 
ECQL.toCQL.  From what I can gather from the ECQL grammar, an ID filter 
should either be ID (number) or ID('string').  The existent quote 
check in FilterToECQL checks to see if there are .'s in the string which 
I works when the ID scheme is fid.num.

I've put in a ticket and pushed up a PR.

Ticket: https://jira.codehaus.org/browse/GEOT-4870
PR:  https://github.com/geotools/geotools/pull/524

Thanks in advance for any feedback,

Jim


--
Infragistics Professional
Build stunning WinForms apps today!
Reboot your WinForms applications with our WinForms controls. 
Build a bridge from your legacy apps to the future.
http://pubads.g.doubleclick.net/gampad/clk?id=153845071iu=/4140/ostg.clktrk
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] ECQL temporal filter question

[Jim Hughes]

Jody,

I tried to report the issue in Jira; do I need an account?  I setup a 
Codehaus as 'jnh5y' with this e-mail address.


If there were a general test, I imagine one could test the DataStore 
interface rather than that ContentDataStore interface/abstract 
implementation.  (For GeoMesa, we ended up going with the Abstract** 
implementations since some class element had too restrictive of 
permissions at one point.)


Anyhow, is a back-port to the 11.x series possible?  In reality, that 
likely only matters if the 11.3 release will come before the 12.0 release...


Thanks,

Jim

On 08/01/2014 02:38 PM, Jody Garnett wrote:
Actually there is something you could do to help. Create a bug ticket 
so this change can show up in our release notes :)


There is examples of how to do a common integration test - see how 
JDBCDataStore does it. Other than that we had a set of tests form 
MemoryDatastore that have been copied into each implementation and 
customised.


There was a bug about setting up a common ContentDataStore test 
suite - but I cannot seem to find it. It is a great idea if you would 
like to work on it.




Jody Garnett


On Fri, Aug 1, 2014 at 11:29 AM, James Hughes jn...@ccri.com 
mailto:jn...@ccri.com wrote:


Hi Jody,

Thanks for the fast turn away; a back port to stable branches
would be great.  Lemme know if I can help with that.

I was actually asking about something a little different from
reprojection.  Since each Accumulo table effectively has one
index, I'm working on tearing apart ECQL filters so that I can use
their spatial, temporal, and attribute predicates differently.  We
ran into this issue since I pulled out temporal filters on the
client-side and needed to send them to the server-side.

As a general follow-up, since there are a number of DataStore
implementations, has anyone cooked up a general DataStore
'integration test'?  I could imagine such a test being really,
really helpful to see the status of a project like GeoMesa.

Thanks again!

Jim


- Original Message -
From:
Jody Garnett jody.garn...@gmail.com
mailto:jody.garn...@gmail.com

To:
Jim Hughes jn...@ccri.com mailto:jn...@ccri.com
Cc:
GeoTools Developers list
geotools-devel@lists.sourceforge.net
mailto:geotools-devel@lists.sourceforge.net
Sent:
Fri, 1 Aug 2014 10:42:26 -0700
Subject:
Re: [Geotools-devel] ECQL temporal filter question



Hi Jim,

Thanks for the pull request, looks fine to me (the test cases
help), think you made a good call about printing the extra
info if present.

Serialising by ECQL is fine, the XML format is more verbose,
but can be used to smuggle additional information along for
the ride.

As for a better way: I had my heart set on a transform
addition to GeoTools Query for the GeoMesa
http://geomesa.github.io project. This would express a bit
of the processing involved in reprojection more clearly in the
filter syntax. The result is an expression for each attribute
in the final data product, making it easier to split work up.
Sadly that development team did not come back geotools-devel
before the 12.x code freeze. Hopefully with time/talent/money
the idea will be put back on the table.


Jody Garnett


On Fri, Aug 1, 2014 at 5:22 AM, Jim Hughes jn...@ccri.com
mailto:jn...@ccri.com wrote:

Thanks Jody,

I've put up https://github.com/geotools/geotools/pull/519.
I'm a little unsure how much of the tests it made sense to
duplicate.  Also, given that the current behavior is to
truncate Dates to the second, I figured it was reasonable
to print the extra info if it is present, but otherwise to
act the same.

By the way, my use case involves splitting up ECQL filters
and giving pieces of the filter to various Accumulo
iterators.  If there is a better way to be serializing and
deserializing parts of an ECQL query, I'm open to doing
things a better way.

Thanks in advance,

Jim


On 07/31/2014 07:35 PM, Jody Garnett wrote:

My three questions are

1.  Is it reasonable to expect that
ECQL.toCQL/toFilter should be inverses?


If it is reasonable if you supply a patch :) I figured
it was reasonable and patched a couple of gaps here:
https://jira.codehaus.org/browse/GEOT-4783

In particular you can review the patch

https://github.com/geotools/geotools/commit/649853a304a4779ed86121151193ea0fe049f367
to see how I handled recognising part of a filter

[Geotools-devel] [jira] (GEOT-4867) Add millisecond resolution to ECQL.toCQL.

[Jim Hughes (JIRA)]

Title: Message Title










 

 Jim Hughes created an issue


















 GeoTools /  GEOT-4867



  Add millisecond resolution to ECQL.toCQL. 










Issue Type:

  Bug




Assignee:


 Unassigned




Created:


 06/Aug/14 3:26 PM




Fix Versions:


 12-RC1




Priority:

  Major




Reporter:

 Jim Hughes










When serializing ECQL Filters with ECQL.toCQL, millis data information was been dropped.












   

 Add Comment






















 This message was sent by Atlassian JIRA (v6.1.6#6162-sha1:7af547c

Re: [Geotools-devel] ECQL temporal filter question

[Jim Hughes]

Jody,

Here's the ticket: http://jira.codehaus.org/browse/GEOT-4867

More inline.

On 08/06/2014 04:23 PM, Jody Garnett wrote:

Additional comments inline:

If there were a general test, I imagine one could test the
DataStore interface rather than that ContentDataStore
interface/abstract implementation.  (For GeoMesa, we ended up
going with the Abstract** implementations since some class element
had too restrictive of permissions at one point.)


Darn, I am trying to kill AbstractDataStore classes. Would of loved to 
know what permissions were getting in your way. The geotools-devel 
list here is for exactly these kind of questions.


It was over a year ago...  IIRC, it was the final modifier on 
ContentFeatureStore.addFeatures.  I actually remember visiting the 
GitHub page and seeing that the top commit fixed my problem on master.  
Sadly, I had to make it go with a release version that day/week:(


I'll see if we can spend a few minutes and switch over to Content** 
implementations, but it might take a little while.



Anyhow, is a back-port to the 11.x series possible?  In reality,
that likely only matters if the 11.3 release will come before the
12.0 release...


The release schedule is available: 
https://github.com/geoserver/geoserver/wiki/Release-Schedule


Awesome; I think I had run across another schedule/release info which 
didn't have 12.x on it.


Thanks again,

Jim
--
Infragistics Professional
Build stunning WinForms apps today!
Reboot your WinForms applications with our WinForms controls. 
Build a bridge from your legacy apps to the future.
http://pubads.g.doubleclick.net/gampad/clk?id=153845071iu=/4140/ostg.clktrk___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] ECQL temporal filter question

[Jim Hughes]

Thanks Jody,

I've put up https://github.com/geotools/geotools/pull/519.  I'm a little 
unsure how much of the tests it made sense to duplicate. Also, given 
that the current behavior is to truncate Dates to the second, I figured 
it was reasonable to print the extra info if it is present, but 
otherwise to act the same.


By the way, my use case involves splitting up ECQL filters and giving 
pieces of the filter to various Accumulo iterators.  If there is a 
better way to be serializing and deserializing parts of an ECQL query, 
I'm open to doing things a better way.


Thanks in advance,

Jim

On 07/31/2014 07:35 PM, Jody Garnett wrote:

My three questions are

1.  Is it reasonable to expect that ECQL.toCQL/toFilter should be
inverses?


If it is reasonable if you supply a patch :) I figured it was 
reasonable and patched a couple of gaps here: 
https://jira.codehaus.org/browse/GEOT-4783


In particular you can review the patch 
https://github.com/geotools/geotools/commit/649853a304a4779ed86121151193ea0fe049f367 
to see how I handled recognising part of a filter as being 
representable as ECQL, and then creating a method to output the ECQL.


   if (isInFilter(filter)) {
return buildIN(filter, extraData);
   }
   // default to normal OR output
   return FilterToTextUtil.buildBinaryLogicalOperator(OR, this, 
filter, extraData);


2.  If my issue is a bug, can I submit a PR, etc?  Is
there documentation about that process?


Yep, there is a CONTRIBUTING.md 
https://github.com/geotools/geotools/blob/master/CONTRIBUTING.mdfile 
(which is shown as a link when you issue a github pull request).  Or 
see the home page about getting involved 
http://geotools.org/getinvolved.html.


3.  Has anyone else run into this?  Is there a well-known work-around?


I don't think many people rely on ECQL.toCQL/toFilter being 
invertible. So lets just get it done.


--
Jody


--
Want fast and easy access to all the code in your enterprise? Index and
search up to 200,000 lines of code with a free copy of Black Duck
Code Sight - the same software that powers the world's largest code
search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

[Geotools-devel] ECQL temporal filter question

[Jim Hughes]

Hi all,

I'm running into an issue with serializing/deserializing ECQL filters 
with ECQL.toCQL and ECQL.toFilter.  In particular, I'm noticing that 
these functions are not inverses when I pass in a temporal filter 
containing Dates with milliseconds information specified.  The millis 
info is being dropped as a Filter is turned into a String.  I believe 
that is happening in org.geotools.filter.text.commons.ExpressionToText.

My three questions are

1.  Is it reasonable to expect that ECQL.toCQL/toFilter should be inverses?

2.  If my issue is a bug, can I submit a PR, etc?  Is there 
documentation about that process?

3.  Has anyone else run into this?  Is there a well-known work-around?

Thanks,

Jim

--
Infragistics Professional
Build stunning WinForms apps today!
Reboot your WinForms applications with our WinForms controls. 
Build a bridge from your legacy apps to the future.
http://pubads.g.doubleclick.net/gampad/clk?id=153845071iu=/4140/ostg.clktrk
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel