Re: [Geotools-devel] gt-geojsondatastore GeoJSONReader should specify encoding as UTF-8?

[Ian Turton]

I just assumed that everything was going to be utf8. Happy to review a pull
request with a test.

Ian

On Tue, 10 May 2022, 13:41 Tobias Gerdin, 
wrote:

> Hello,
>
>
>
> I was puzzled by the behaviour of org.geotools.data.geojson.GeoJSONReader 
> when I was using it to read a feature collection containing non-ascii 
> strings. It complains that the JSON string contains invalid UTF-8 data.
>
>
>
> Due to client mandate I need to develop on a Windows 11 machine. The
> default platform encoding is windows-1252 (for archeological reasons, I
> guess), not UTF8. I noticed that GeoJSONReader uses plain String.getBytes()
> to read the JSON data (
> https://github.com/geotools/geotools/blob/f416fcc3763b2db020c54a9323601fbdd49388e7/modules/unsupported/geojson-core/src/main/java/org/geotools/data/geojson/GeoJSONReader.java#L179
> ).
>
>
>
> When I change the JVM charset encoding (which needs to be done at startup) 
> using -Dfile.encoding=”UTF-8” my code works, but I rather not have to do 
> this. I am not an expert on JSON but I recall the spec mandates that JSON 
> data is encoded in UTF-8. So I believe that the above linked line should do 
> jsonString.getBytes(StandardCharsets.*UTF_8*) (and in all other locations 
> where JSON data is read).
>
>
>
> Apparently Java is slated to go UTF-8 by default in the future, but until
> then we need to deal with this mess I guess.
>
>
> [image: Havochvatten]
>
> *Tobias Gerdin*
>
> Systemutvecklare, Konsult
>
> Enheten för systemutveckling
>
>
>
> Gullbergs Strandgata 15, 411 04 Göteborg
>
> Box 11930, SE-404 39 Göteborg
>
> tobias.ger...@havochvatten.se
>
> www.havochvatten.se
> [image: Havochvatten]
> Havs- och vattenmyndigheten behandlar dina personuppgifter i enlighet med
> dataskyddsförordningen och myndighetens dataskyddspolicy, läs mer på
> www.havochvatten.se/sa-behandlar-hav-dina-personuppgifter
> SwAM processes your personal data in accordance with the General Data
> Protection Regulation (GDPR) and our Data Protection Policy, see
> www.havochvatten.se/sa-behandlar-hav-dina-personuppgifter
> ___
> GeoTools-Devel mailing list
> GeoTools-Devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/geotools-devel
>
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

[Geotools-devel] PMC meeting notes, May 10th 2022

[Andrea Aime]

GeoTools / GeoServer PMC meeting - 2022-05-10Attending

   -

   Torben Barsballe
   -

   Stacy Acosta
   -

   Jody Garnett
   -

   Andrea Aime
   -

   Jukka Rahkonen


Actions from prior meetings:

   -

   AA: review Jody’s Log4j PR
   -

   AA: setup HTTPS and www.geoserver.org (AA to talk with SAC)

Agenda

   1.

   GeoServer Domain
   2.

   Build server care and feeding and upgrade
   3.

   GeoServer 2.21-RC
   4.

   GeoServer 2.21.0 release
   5.

   FeatureCollection.query(Query)
   6.

   GeoTools Extra Samples

Actions

   -

   andrea: email devel list for release volunteer
   -

   torben and jody: try and clean up a committed m2 repo




GeoServer Domain

Action follow-up for setup HTTPS and www.geoserver.org (AA to talk with SAC)

   -

   We require zone records to do this - follow-up with Planet again.
   -

   Also would like geowebcache domain


Build server care and feeding and upgrade

   -

   Ran out of disk space, see gitter for troubleshooting
   -

   each workspace is taking 1-2G in space (including doc builds)
   -

   2.21-M0 tag includes an m2 repo - making every git checkout massive
   -


  
https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/removing-sensitive-data-from-a-repository

  -

  Force push on a tag (so not terrible rewrite of history)
  -

  action: torben and jody to try the above out (breakout session)
  -

   Jody proposed AX101 machine for new build server
   -

  Jeroen and Simone have offered to split costs


New Machine AX101 below proposed:

€65 AX61-NVME 

   -

   CPU: AMD Ryzen 3900 12 cores / 24 threads
   -

   RAM: 128 GB DDR4 ECC
   -

   Disk: 2x 1.92 TB NVMe SSD
   -

   Bandwidth: 1 Gbits, Unlimited


€115 SYS-6-SSD-256 

   -

   CPU: AMD  Epyc 7351p 16 cores / 32 threads
   -

   RAM: 256GB DDR4 ECC 2400MHz
   -

   Disk: SoftRAID  2x512GB NVME SSD
   -

   Bandwidth: 250 Mbps Unlimited


€113 AX101 

   -

   CPU: AMD 5950X 16 cores / 32 threads
   -

   RAM: 128 GB DDR4 ECC
   -

   Disk: 2x 3.84 TB NVMe SSD
   -

   Bandwidth: 1 Gbits, Unlimited


€136 SYS-7-SSD-128 

   -

   CPU: AMD  Epyc 7451 24 cores / 48 threads
   -

   RAM: 128GB DDR4 ECC 2400MHz
   -

   Disk: SoftRAID 2 x 512GB  NVME SSD


   -

   Bandwidth: 250 Mbps Unlimited


€165 AX161  (was €131
base + €33 for disk)

   -

   CPU: AMD 5950X 32 cores / 64 threads
   -

   RAM: 128 GB DDR4 ECC
   -

   Disk: 2x 1.92 TB NVMe SSD
   -

   Bandwidth: 1 Gbits


€168 SYS-7-SSD-256

   -

   CPU: AMD  Epyc 7451 24 cores / 48 threads
   -

   RAM: 256GB DDR4 ECC 2400MHz
   -

   Disk: SoftRAID 2 x 512GB  NVME SSD


   -

   Bandwidth: 250 Mbps Unlimited


GeoServer 2.21-RC

Release Candidate is out
https://geoserver.org/announcements/2022/05/09/geoserver-2-21-RC-released.html


   -

   broken link noticed (new WPS and KML output settings, external output
   directory)
   -

   we want to thank implementer / customer for the final announcement
   -

   please take care with issue FixedFor and provide any indication of what
   was fixed

Asked the user list to test, need some testing please :)

This is a really good release, love feature type customization.

GeoServer 2.21.0 release

We need a volunteer
https://github.com/geoserver/geoserver/wiki/Release-Schedule

   -

   This is just a normal release, all the work for new branches is the RC
   -

   Andrea is asking the devel list for a release manager


Round up of known issues:

   -

   Dispatcher is willing to log request body? And the new request header
   settings duplicate this functionality. Joseph is going to make a PR to fix
   before release :)
   -

   Anything else? Please test log4j configurations
   -

   A couple hard coded fonts in the UI to cleanup

FeatureCollection.query(Query)

GeoTools has some hooks for optimizations:

   -

   Propose FeatureCollection.query(Query)
   -

   Feature collection has optimizations for Filter, Sort (single
   attribute), …
   -

  intended to be a fluent api for production of an internal query :)
  -

   Most of the wrappers are doing one thing in memory, and lose track of
   the query.

Andrea will make a proposal, the goal is the ability to pass
FeatureCollection around.
GeoTiff Extra Samples

GeoTiff extra samples field bug (PR pending)

https://osgeo-org.atlassian.net/browse/GEOT-6452

Stacy working on creating a test case

The only currently available data source that reflects the issue is
internal proprietary data and not suitable for inclusion in a test case.

Consider using gdal_create can often be used for creating a test file that
is alike from the essential parts but with dummy contents

https://gdal.org/programs/gdal_create.html

An 

[Geotools-devel] gt-geojsondatastore GeoJSONReader should specify encoding as UTF-8?

[Tobias Gerdin]

Hello,


I was puzzled by the behaviour of org.geotools.data.geojson.GeoJSONReader when 
I was using it to read a feature collection containing non-ascii strings. It 
complains that the JSON string contains invalid UTF-8 data.


Due to client mandate I need to develop on a Windows 11 machine. The default 
platform encoding is windows-1252 (for archeological reasons, I guess), not 
UTF8. I noticed that GeoJSONReader uses plain String.getBytes() to read the 
JSON data 
(https://github.com/geotools/geotools/blob/f416fcc3763b2db020c54a9323601fbdd49388e7/modules/unsupported/geojson-core/src/main/java/org/geotools/data/geojson/GeoJSONReader.java#L179).


When I change the JVM charset encoding (which needs to be done at startup) 
using -Dfile.encoding="UTF-8" my code works, but I rather not have to do this. 
I am not an expert on JSON but I recall the spec mandates that JSON data is 
encoded in UTF-8. So I believe that the above linked line should do 
jsonString.getBytes(StandardCharsets.UTF_8) (and in all other locations where 
JSON data is read).

Apparently Java is slated to go UTF-8 by default in the future, but until then 
we need to deal with this mess I guess.

[Havochvatten]
Tobias Gerdin
Systemutvecklare, Konsult
Enheten för systemutveckling

Gullbergs Strandgata 15, 411 04 Göteborg
Box 11930, SE-404 39 Göteborg
tobias.ger...@havochvatten.se
www.havochvatten.se
[Havochvatten]
Havs- och vattenmyndigheten behandlar dina personuppgifter i enlighet med 
dataskyddsförordningen och myndighetens dataskyddspolicy, läs mer på 
www.havochvatten.se/sa-behandlar-hav-dina-personuppgifter
SwAM processes your personal data in accordance with the General Data 
Protection Regulation (GDPR) and our Data Protection Policy, see 
www.havochvatten.se/sa-behandlar-hav-dina-personuppgifter
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] Migrate tests in gt-jdbc from JUnit3 to JUnit4

[Uhrig, Stefan via GeoTools-Devel]

Hi Andrea,

Thanks for the hint. That's helpful. I considered migrating OnlineTest first, 
but not only the JDBC tests derive from that class. Using OnlineTestSupport as 
base for JDBCTestSupport reduces the amount of code that has to be migrated. 
It's still a lot, but not as much as with the OnlineTest migration...

I'll create a Jira item and will then prepare a pull request.

Cheers,
Stefan

From: Andrea Aime 
Sent: Friday, May 6, 2022 6:52 PM
To: Uhrig, Stefan 
Cc: GeoTools Developers 
Subject: Re: [Geotools-devel] Migrate tests in gt-jdbc from JUnit3 to JUnit4

Hi Stefan,
personally I don't have objections, as long as the tests for all databases
are still working after the migration. I believe Ben some years ago prepared an 
OnlineTest replacement JUnit4 .. here:
https://github.com/geotools/geotools/blob/73051a745a647c134ae7c2be26978e0968bfbb03/modules/library/sample-data/src/main/java/org/geotools/test/OnlineTestSupport.java#L44

Only 3 classes seem to be extending it... not sure if it's gonna help or 
hinder, just letting you know it's there.

Cheers
Andrea


On Fri, May 6, 2022 at 6:16 PM Uhrig, Stefan via GeoTools-Devel 
mailto:geotools-devel@lists.sourceforge.net>>
 wrote:
Hi all,

I'd like to migrate the tests in gt-jdbc (and all derived tests) from JUnit3 to 
JUnit4 (or at least attempt it). Are there any objections against that?

Background is that I'd like to enable parallel test execution for the HANA JDBC 
Plugin. That seems to work only for JUnit4 tests.

Best regards,
Stefan

___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel


--

Regards,

Andrea Aime

==
GeoServer Professional Services from the experts!

Visit 
http://bit.ly/gs-services-us
 for more information.
==

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions Group
phone: +39 0584 962313

fax: +39 0584 1660272

mob:   +39  333 8128928


https://www.geosolutionsgroup.com/

http://twitter.com/geosolutions_it

---

Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 
2016/679 - Regolamento generale sulla protezione dei dati "GDPR"), si precisa 
che ogni circostanza inerente alla presente email (il suo contenuto, gli 
eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i 
destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per 
errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei 
comunque grato se potesse darmene notizia.

This email is intended only for the person or entity to which it is addressed 
and may contain information that is privileged, confidential or otherwise 
protected from disclosure. We remind that - as provided by European