Hello Jukka and Andrea,Jukka, apparently I misunderstood you. GeoServer's JSON is GeoJSON as its (backwards-) compatible with GeoJSON (the addition of the inofficial "crs" object does not prevent GeoJSON clients form reading the format). The same is true for OGC's JSON-FG.
The new Slim format is NOT compatible with GeoJSON as GeoJSON clients will fail reading it (expecting an object and not an array for the "properties" key).
Actually, I did not intend to invent a completely new format at all. The idea was just to address a shortcomming (for my mind) of GeoJSON and do some small optimizations. However, technically that is a new and incompatible format. I didn't really get this yesterday. Sorry for that.
As a Porsche *Cabrio* is still a Porsche, I'm sure that a name including the term GeoJSON will still be the best option. However, this assumes that the GeoJSON inventors actually specify that new variant (or support it, at least). Since we have several new formats like BrokJSON and JSON-FG *without* the term "Geo" in it, I guess that the GeoJSON guys are not really open to such a new variant. Jukka, you seem to know them quite a bit, what do you think about it? Is is worth asking them?
Otherwiese people have to deal with "meaningless" (sorry for that term) format names like "BrokJSON". What is it? Storing Broks in JSON? Is it about geodata? What are Broks? Do you know what people are behind BrokJSON?
Andrea recommended to also maintain a publicly visible site with some specifications for the new format like https://www.brokjson.dev/. That would be cool, of course. However, I'm first of all running a software company and have a couple of B2B projects using such huge GetFeature responses. I'm just looking for a *simple* solution which is available in the next days (not weeks or months). You know what I mean?
I think about just implementing BrokJSON with the new module. However, I guess that there are more changes in the client's reader required than with my current format. Also, BrokJSON seems not to preserve/include the actual "geometry_name", which may be required if the client has to filter some requests by geometries. Instead one has to relay on what DescribeFeatureTypes returned, only.
One last thing: currently the new module "falls back" to GeoServer's normal response for complex features. With a really new format (whether it's BrokJSON or something new), the format should just reject/fail if someone requests complex features in that new format, right?
What are your suggestions? Cheers Carsten Am 06.10.2022 um 21:05 schrieb Rahkonen Jukka:
Hi, I suggest to use something/anything else than "GeoJSON" in the name of the community module or as a common name of the format because this JSON variant is not valid GeoJSON https://www.rfc-editor.org/rfc/rfc7946. In my previous mail I used the OGC draft about the "JSON for features and geometries" as an example because it does not contain "GeoJSON" in the name for making it clear that it is not written by the authors of GeoJSON. And that despite the fact that actually JSON-FG (or whatever it will be called) is also valid GeoJSON because it only adds new features into the GeoJSON specification. I did not mean that Geoserver is delivering JSON-FG, it does not. Sorry for the confusion. If you definitely want to use the name "Slim GeoJSON" as a name of the format it would be polite to contact the authors of the GeoJSON specification and ask their opinion. All the authors are named in the beginning of the RFC document but maybe Howard Butler or Sean Gilles are the best first contacts. Otherwise it is fine for me if you prefer to write your format just by yourself. -Jukka Rahkonen- -----Alkuperäinen viesti----- Lähettäjä: Carsten Klein <c.kl...@datagis.com> Lähetetty: torstai 6. lokakuuta 2022 20.09 Vastaanottaja: Rahkonen Jukka <jukka.rahko...@maanmittauslaitos.fi>; Andrea Aime <andrea.a...@geosolutionsgroup.com> Kopio: geoserver-devel@lists.sourceforge.net; bj...@wololo.org Aihe: Re: [Geoserver-devel] Enhancement: WFS Simple Feature Response in new more compact JSON/JSONP format Hi Jukka, you suggest a different name (e. g. "OGC Features and geometries JSON"). What parts of the module/format would you like to be renamed? Module name? Class name? outputFormat name? Name and description of the format/module in README.md? GeoServer actually has a class named GeoJSONGetFeatureResponse for emitting JSON. My class SlimGeoJSONGetFeatureResponse is just an extension adding the slim capability. Also, in Format-DropDown on the Layer Preview page, JSON is announced as ... GeoJSON GeoJSON(JSONP) ... On the other hand, after having read only some parts of the OGC draft document, does GeoServer really return JSON-FG? At first glance, GeoServer extends GeoJSON (it's no official GeoJSON) but it's still far from JSON-FG, isn't it? Maybe naming it "OGC Features and geometries JSON" isn't much more apt? I just wanted to stay as close as possible with GeoServer as I'm extending *its* current JSON capabilities. GeoServer docs state: https://eur06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.geoserver.org%2Flatest%2Fen%2Fuser%2Fservices%2Fwfs%2Foutputformats.html%23json-and-jsonp-output&data=05%7C01%7Cjukka.rahkonen%40maanmittauslaitos.fi%7C7e9bb4668f3048d0800908daa7bd8852%7Cc4f8a63255804a1c92371d5a571b71fa%7C0%7C0%7C638006729727251264%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ec1%2BnAsT0UIx4fdO0CQN3Hox0wrLjHF%2Fnz0Mg3SYOYY%3D&reserved=0The JSON output format (and JSONP if enabled) return feature content as GeoJSON document. Here is an example of a simple GeoJSON file;The "GeoJSON" is a link to https://eur06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgeojson.org%2F&data=05%7C01%7Cjukka.rahkonen%40maanmittauslaitos.fi%7C7e9bb4668f3048d0800908daa7bd8852%7Cc4f8a63255804a1c92371d5a571b71fa%7C0%7C0%7C638006729727251264%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=rGQjaaiJWl89NvfMs8t%2BFJqJF4Gc2MDJgmrOjdkBgHw%3D&reserved=0 I do not know how GeoServer serializes complex features in JSON. Maybe that is closer to JSON-FG. For me, however, simple features encoded in JSON seem quite close to GeoJSON with the additon of a "crs" object (as initially recommended by GeoJSON in [GJ2008] and removed later due do interoperability concerns). Although I'm open for changes, I believe that this small extension should likely not change much of the class it derives from (including terms and names), except that it removes space wasting redundancy (the "Slim" aspect). Asking the gdal-dev list may be an option. However, this could start a very long running process with really lots of different opinions. For my mind, if someone is concerned whether GeoServer delivers GeoJSON, CRS-extended GeoJSON or even JSPN-FG - this should first be clairified (and possibly be changed) in GeoServer core. Later, extensions and modules could adapt these changes in a second step. Cheers Carsten Am 06.10.2022 um 16:26 schrieb Rahkonen Jukka:Hi, I would consider some other name because this format is not GeoJSON. OGC is using name "OGC Features and geometries JSON" in its own project for enhancing GeoJSON https://eur06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.ogc.org%2FDRAFTS%2F21-045.html&data=05%7C01%7Cjukka.rahkonen%40maanmittauslaitos.fi%7C7e9bb4668f3048d0800908daa7bd8852%7Cc4f8a63255804a1c92371d5a571b71fa%7C0%7C0%7C638006729727251264%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=7KYdte0FMaaaUNIZ7mqVV6nwkzBfMnljjVHD%2FO%2FrazE%3D&reserved=0. Perhaps your format could be something like "Slim JSON for features". Maybe you could contact also other open source projects for collecting more opinions and suggestions. A short announcement on the gdal-dev mailing list might be a good start. -Jukka Rahkonen- -----Alkuperäinen viesti----- Lähettäjä: Carsten Klein <c.kl...@datagis.com> Lähetetty: torstai 6. lokakuuta 2022 16.47 Vastaanottaja: Andrea Aime <andrea.a...@geosolutionsgroup.com> Kopio: geoserver-devel@lists.sourceforge.net; bj...@wololo.org Aihe: Re: [Geoserver-devel] Enhancement: WFS Simple Feature Response in new more compact JSON/JSONP format Hello Andrea, I've just issued a PR on GitHub. The format and community module has been renamed to "Slim GeoJSON". "Slim" is shorter and likely easier to pronounce than compact (together with GeoJSON). Sorry for struggling and messing around with Git. However, the final version of the PR should be OK for merging. There are still some issues with QA checks and build checks (a test failed on Mac). I've written some more remarks on that in the PR on GitHub. Cheers Carsten On Thu, Sep 29, 2022 at 10:17 AM Andrea Aime wroteOn Thu, Sep 22, 2022 at 10:08 AM Carsten Klein <c.kl...@datagis.com <mailto:c.kl...@datagis.com>> wrote: Hello there, the new WFS output format CompactJSON, implemented as a community extension, is ready and works as expected. With our mentioned WFS responses (1.000+ features, 450 columns each), compared to standard GeoJSON, savings are between ~30% and ~70% (uncompressed) and between ~20% and ~50% with gzip content encoding. (see ´Statistics´ and ´Conclusion´ below) What are the next steps in order to contribute/publish the new community extension? Shall I commit all the stuff to my forked ´geoserver´ repository and issue a CR including all changes required in existing core code (that is the existing JSON producer)? Yes, make a Pull Request and send a CLA to OSGeo. When opening the PR you have a checklist that needs to be satisfied, with links to the relevant docs. For reference, here it is: https://eur06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit h ub.com%2Fgeoserver%2Fgeoserver%2Fblob%2Fmain%2F.github%2FPULL_REQUEST _ TEMPLATE.md&data=05%7C01%7Cjukka.rahkonen%40maanmittauslaitos.fi% 7 Cbe7e5c20e1084a56a2b608daa7a15866%7Cc4f8a63255804a1c92371d5a571b71fa% 7 C0%7C0%7C638006608655183714%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwM D AiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C& s data=uEICQlG9TNers9r7xIDHBWy%2BeqR%2BPQqL2juv84XbkUY%3D&reserved= 0 <https://eur06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgi t hub.com%2Fgeoserver%2Fgeoserver%2Fblob%2Fmain%2F.github%2FPULL_REQUES T _TEMPLATE.md&data=05%7C01%7Cjukka.rahkonen%40maanmittauslaitos.fi % 7Cbe7e5c20e1084a56a2b608daa7a15866%7Cc4f8a63255804a1c92371d5a571b71fa % 7C0%7C0%7C638006608655183714%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAw M DAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C& ; sdata=uEICQlG9TNers9r7xIDHBWy%2BeqR%2BPQqL2juv84XbkUY%3D&reserved = 0> One last question: some of the WFS format extensions have one or more GeoServerApplication.properties files, which seem to provide human readable names for the actual MIME types or formats. Should I provide these, too? Where are these texts used? What are the rules? It would be nice if you could add it, yes. They are used in the format dropdown you can find in the layer preview page. We don't have a list of rules, checking what the other formats do and mimicking it, is the de-facto approach when adding a new output format. Statistics ========== Given two layers A and B, both having ~450 columns. Layer A's geometry contains polygons (~300 vertices each). Layer B has point geometries (created as a view of layer A with ST_Centroid(the_geom) AS the_geom). Querying only ~1,000 and all ~10,000 features/rows from both layers: Layer A (heavy polygons) ------------------------ ~1,000 rows bytes raw bytes gzip GeoJSON: 19,098,924 (100%) 5,361,536 (100%) CompactJSON: 12,586,758 ( 66%) 4,234,995 ( 79%) FlatGeobuf: 10,093,224 ( 53%) 7,006,365 (130%) Layer A (heavy polygons) ------------------------ ~10,000 rows bytes raw bytes gzip GeoJSON: 174,686,637 (100%) 45,414,569 (100%) CompactJSON: 109,700,539 ( 63%) 35,677,456 ( 78%) FlatGeobuf: 86,031,048 ( 49%) 58,857,835 (130%) Layer B (lightweight points) ---------------------------- ~1,000 rows bytes raw bytes gzip GeoJSON: 9,443,980 (100%) 2,236,172 (100%) CompactJSON: 2,931,814 ( 31%) 1,174,942 ( 53%) FlatGeobuf: 3,710,902 ( 39%) 2,044,614 ( 91%) Layer B (lightweight points) ---------------------------- ~10,000 rows bytes raw bytes gzip GeoJSON: 92,394,763 (100%) 18,923,302 (100%) CompactJSON: 27,408,665 ( 30%) 9,720,941 ( 51%) FlatGeobuf: 32,588,230 ( 35%) 16,824,845 ( 89%) Conclusion ========== Savings mainly depend on the ratio between schema information and data. That is, savings are small if features have much more data than schema information. Typically, geometries may get quite large, so savings depend on the size/complexity of the feature's geometries. (e. g. tests where done using complex polygons vs. simple point objects) Also remarkable is that CompactJSON produces not much bigger responses (aka is not much worse) than FlatGeobuf. However, FlatGeobuf responses do not compress very well and are bigger than compressed CompactJSON in all (and even bigger than compressed GeoJSON if geometries are heavy in some) cases. Interesting finding indeed. Cheers Andrea == GeoServer Professional Services from the experts! Visit https://eur06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fbit. l%2F&data=05%7C01%7Cjukka.rahkonen%40maanmittauslaitos.fi%7C7e9bb 4668f3048d0800908daa7bd8852%7Cc4f8a63255804a1c92371d5a571b71fa%7C0%7C 0%7C638006729727251264%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLC JQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdat a=vvvpwIKpew14rnJvNcg5Z63FaDnxVtSIK8lRKmMBS2o%3D&reserved=0 y%2Fgs-services-us&data=05%7C01%7Cjukka.rahkonen%40maanmittauslai t os.fi%7Cbe7e5c20e1084a56a2b608daa7a15866%7Cc4f8a63255804a1c92371d5a57 1 b71fa%7C0%7C0%7C638006608655183714%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC 4 wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C% 7 C&sdata=aixKL%2FMfJ78fvCvg9icTQH%2FRTvr7SMuaEX3ejZWzBm0%3D&re s erved=0 <https://eur06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fbit. ly%2Fgs-services-us&data=05%7C01%7Cjukka.rahkonen%40maanmittausla i tos.fi%7Cbe7e5c20e1084a56a2b608daa7a15866%7Cc4f8a63255804a1c92371d5a5 7 1b71fa%7C0%7C0%7C638006608655183714%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiM C 4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C % 7C&sdata=aixKL%2FMfJ78fvCvg9icTQH%2FRTvr7SMuaEX3ejZWzBm0%3D&r e served=0>for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions Group phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 https://eur06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww. geosolutionsgroup.com%2F&data=05%7C01%7Cjukka.rahkonen%40maanmitt a uslaitos.fi%7Cbe7e5c20e1084a56a2b608daa7a15866%7Cc4f8a63255804a1c9237 1 d5a571b71fa%7C0%7C0%7C638006608655183714%7CUnknown%7CTWFpbGZsb3d8eyJW I joiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000% 7 C%7C%7C&sdata=vVpcWGDkUsUGZiuizBkzoVXyw2r7C3g2yo3WlWqVeTg%3D& r eserved=0 <https://eur06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fww w .geosolutionsgroup.com%2F&data=05%7C01%7Cjukka.rahkonen%40maanmit t auslaitos.fi%7Cbe7e5c20e1084a56a2b608daa7a15866%7Cc4f8a63255804a1c923 7 1d5a571b71fa%7C0%7C0%7C638006608655183714%7CUnknown%7CTWFpbGZsb3d8eyJ W IjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000 % 7C%7C%7C&sdata=vVpcWGDkUsUGZiuizBkzoVXyw2r7C3g2yo3WlWqVeTg%3D& ; reserved=0> https://eur06.safelinks.protection.outlook.com/?url=http%3A%2F%2Ftwit t er.com%2Fgeosolutions_it&data=05%7C01%7Cjukka.rahkonen%40maanmitt a uslaitos.fi%7Cbe7e5c20e1084a56a2b608daa7a15866%7Cc4f8a63255804a1c9237 1 d5a571b71fa%7C0%7C0%7C638006608655183714%7CUnknown%7CTWFpbGZsb3d8eyJW I joiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000% 7 C%7C%7C&sdata=0nUuMX0boyNwqQQ0XUsPUmKkGFiV5dIrHo08ifFLkGg%3D& r eserved=0 <https://eur06.safelinks.protection.outlook.com/?url=http%3A%2F%2Ftwi t ter.com%2Fgeosolutions_it&data=05%7C01%7Cjukka.rahkonen%40maanmit t auslaitos.fi%7Cbe7e5c20e1084a56a2b608daa7a15866%7Cc4f8a63255804a1c923 7 1d5a571b71fa%7C0%7C0%7C638006608655183714%7CUnknown%7CTWFpbGZsb3d8eyJ W IjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000 % 7C%7C%7C&sdata=0nUuMX0boyNwqQQ0XUsPUmKkGFiV5dIrHo08ifFLkGg%3D& ; reserved=0> ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati "GDPR"), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 "GDPR" - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail_______________________________________________ Geoserver-devel mailing list Geoserver-devel@lists.sourceforge.net https://eur06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flist s.sourceforge.net%2Flists%2Flistinfo%2Fgeoserver-devel&data=05%7C0 1%7Cjukka.rahkonen%40maanmittauslaitos.fi%7C7e9bb4668f3048d0800908daa7 bd8852%7Cc4f8a63255804a1c92371d5a571b71fa%7C0%7C0%7C638006729727251264 %7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6I k1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=xYL6KJHWeYrwrxGIwW4LNi x4C7yYF6wW62xUsQ%2FpZDE%3D&reserved=0
_______________________________________________ Geoserver-devel mailing list Geoserver-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/geoserver-devel
