No offence taken on my side ;-) Do you think sedona-common could broaden its scope to non cluster technologies? In Baremaps, we assumed that a few TBs of vector data can now easily be processed efficiently on a single node with a lot of RAM and NVMe SSDs. Our current implementation uses Postgis and is good enough for our use case. I turned to Calcite ST functions with the hope to process large chunks of memory-mapped data faster directly in the JVM. I didn’t turned to Sedona as it looked closely tied to spark and flink (I may be wrong). That being said, I believe the implementation of the ST functions might be shared and independent of a cluster technology (via a module?) and then bindings could be developed on a case by case basis.
Best, Bertil > On 2 Feb 2024, at 10:39, Jia Yu <ji...@apache.org> wrote: > > Hi all (removing non-geospatial projects from the loop), > > Just to clarify about the mission of the Apache Sedona project and make sure > nobody gets offended by my words before :-) > > First of all, sedona-common is not a replacement of JTS at all. I also > contribute to JTS. Instead, sedona-common is more like a enhancer for JTS, > GeoTools, (and Apache SIS soon) when it comes to Spatial SQL. > > Secondly, the mission of the Sedona project itself is to bring PostGIS-style > Spatial SQL to all cluster computation engines. Sedona also makes sure that > ST functions behave the same as PostGIS. Eventually we want Sedona ST > functions to achieve 100% parity with PostGIS. This requires lots of work in > the sedona-common module. Some people might think PostGIS is just a wrapper > around GEOS. In fact, it is not. > > Thanks, > Jia > > On Thu, Feb 1, 2024 at 11:15 PM Bertil Chapuis <bchap...@gmail.com > <mailto:bchap...@gmail.com>> wrote: >> Hi Jia, >> >>> This is awesome. However, although Flink depends on Calcite, the >>> SpatialType does not seem to be supported by Flink (based on my test >>> last year). Not sure if this works for Drill. Please correct me if I'm >>> wrong. >> >> This may be related to the version of calcite used by Flink (1.32) [1]. If I >> recall correctly, this version implemented the changes related to JTS and >> most of the functions came later. >> >> >> flink/flink-table/pom.xml at 0779c91e581dc16c4aef61d6cc27774f11495907 · >> apache/flink >> github.com >> >> <https://github.com/apache/flink/blob/0779c91e581dc16c4aef61d6cc27774f11495907/flink-table/pom.xml#L81>flink/flink-table/pom.xml >> at 0779c91e581dc16c4aef61d6cc27774f11495907 · apache/flink >> <https://github.com/apache/flink/blob/0779c91e581dc16c4aef61d6cc27774f11495907/flink-table/pom.xml#L81> >> github.com >> <https://github.com/apache/flink/blob/0779c91e581dc16c4aef61d6cc27774f11495907/flink-table/pom.xml#L81> >>> On a side note, why don't we implement all geometric computation >>> logics in Sedona and Calcite simply wraps around Sedona-common (of >>> course JTS too)? Sedona implements many more functions in addition to >>> JTS: >>> https://github.com/apache/sedona/tree/master/common/src/main/java/org/apache/sedona/common >>> >>> This way, we don't reinvent the wheel. Moreover, Sedona is a >>> spatial-as-the-first-class-citizen project. We can receive way more >>> contributions and make a huge impact. >> >> Thank a lot for the pointer. We should definitively find ways to collaborate >> more on this topic. I’m not sure I will be able to attend the codesprint in >> person, but I would be really interested to exchange remotely if it is >> possible. >> >> Best, >> >> Bertil >> >>> Thanks, >>> Jia >>> >>> On Thu, Feb 1, 2024 at 10:25 PM Bertil Chapuis <bchap...@gmail.com >>> <mailto:bchap...@gmail.com>> wrote: >>>> >>>> Hello Jia and Charles, >>>> >>>> I'm really interested in this topic as well. Apache Calcite transitionned >>>> from ESRI Geometry to JTS, and many ST functions have been implemented >>>> there as well [1, 2, 3]. Sharing experiences and code could benefit all >>>> projects. >>>> >>>> I haven’t looked into the details of each project, but from what I >>>> understand, Sedona depends on Calcite through Flink, and Drill depends >>>> directly on Calcite. Is that correct? Since these functions are available >>>> in Calcite’s core, it means they may already be available in the >>>> respective class paths. >>>> >>>> Best regards, >>>> >>>> Bertil >>>> >>>> [1] >>>> https://github.com/apache/calcite/blob/main/core/src/main/java/org/apache/calcite/runtime/SpatialTypeFunctions.java >>>> [2] >>>> https://github.com/apache/calcite/blob/main/core/src/test/resources/sql/spatial.iq >>>> [3] >>>> https://calcite.apache.org/docs/reference.html#geometry-conversion-functions-2d >>>> >>>> >>>>> On 2 Feb 2024, at 06:47, Jia Yu <ji...@apache.org >>>>> <mailto:ji...@apache.org>> wrote: >>>>> >>>>> Hi Charles, >>>>> >>>>> This is Jia Yu from Apache Sedona. I think what you did is fantastic. >>>>> As a project of this Joint codespring, I am proposing to implement a >>>>> comprehensive set of spatial functions to Apache Drill using Apache >>>>> Sedona. >>>>> >>>>> Apache Sedona has implemented over 130 ST functions and a >>>>> high-performance geometry serializer in pure Java. All these functions >>>>> have been ported to Apache Spark, Apache Flink and Snowflake. They are >>>>> being downloaded over 1.5 million times per month. >>>>> >>>>> This porting process is fairly simple. Let's take Sedona on Apache >>>>> Flink as an example: >>>>> >>>>> 1. Call a Sedona java function in a UDF template: >>>>> https://github.com/apache/sedona/blob/master/flink/src/main/java/org/apache/sedona/flink/expressions/Functions.java >>>>> 2. Register this function in a catalog file: >>>>> https://github.com/apache/sedona/blob/master/flink/src/main/java/org/apache/sedona/flink/Catalog.java >>>>> >>>>> What do you think? >>>>> >>>>> Thanks, >>>>> Jia >>>>> >>>>> On Thu, Feb 1, 2024 at 2:44 PM Charles Givre <cgi...@gmail.com >>>>> <mailto:cgi...@gmail.com>> wrote: >>>>>> >>>>>> Hi Martin, >>>>>> Thanks for sending. I'd love for Drill to be included in this. I have >>>>>> a question for you. A while ago, I started work on a collection of UDFs >>>>>> for interacting with H3 Geo Indexes. I'm not an expert on this but >>>>>> would this be useful? Here's the repo: >>>>>> https://github.com/datadistillr/drill-h3-udf If someone would like to >>>>>> collaborate to complete this and get it integrated, I'm all for that. >>>>>> Best, >>>>>> -- C >>>>>> >>>>>> >>>>>> >>>>>>> On Jan 31, 2024, at 10:20, Martin Desruisseaux >>>>>>> <martin.desruisse...@geomatys.com >>>>>>> <mailto:martin.desruisse...@geomatys.com>> wrote: >>>>>>> >>>>>>> Hello all >>>>>>> >>>>>>> The Open Geospatial Consortium (OGC), The Apache Software Foundation >>>>>>> (ASF) and The Open Source Geospatial Foundation (OSGeo) hold a join >>>>>>> code sprint on February 26 to 28 [1]. The main goals are to support the >>>>>>> development of open standards for geospatial information and to support >>>>>>> the development of free and open source software which implements those >>>>>>> standards, as well as creating awareness about the standards and >>>>>>> software projects. This is the fourth year that this joint code sprint >>>>>>> is organized, and this year will be physically located in Évora >>>>>>> (Portugal). The event can also be attended on-line. Registration is >>>>>>> free [2]. >>>>>>> >>>>>>> Apache SIS, Sedona, Baremaps, Parquet, Drill and Camel projects >>>>>>> participated in the past. It would be great if participation was >>>>>>> possible this year too. Some ideas could be: >>>>>>> >>>>>>> * Experiment the use of Apache SIS in Sedona for referencing and grid >>>>>>> coverage services (could be a join effort between Sedona and SIS >>>>>>> developers). >>>>>>> * Any work related to Geoparquet [3] (an incubating OGC standard based >>>>>>> on Apache Parquet). >>>>>>> * Any work related to Drill GIS functions [4]. >>>>>>> * Any work related to Camel Geocoder [5]. For example, exploring the >>>>>>> pertinence of using the ISO 19112 standard (could be a join effort >>>>>>> between Camel and GeoAPI developers). >>>>>>> >>>>>>> If anyone is interested, the wiki page [1] can be edited directly. If >>>>>>> particular you can add your project in the "Which Apache projects are >>>>>>> going to participate?" section. If an introduction to a project can be >>>>>>> presented as a tutorial, it can also be added in the "Mentor streams" >>>>>>> section of [1]. >>>>>>> >>>>>>> Martin >>>>>>> >>>>>>> [1]https://github.com/opengeospatial/developer-events/wiki/2024-Joint-OGC-%E2%80%93-OSGeo-%E2%80%93-ASF-Code-Sprint >>>>>>> [2]https://developer.ogc.org/sprints/23/ >>>>>>> [3]https://geoparquet.org/ >>>>>>> [4]https://drill.apache.org/docs/gis-functions/ >>>>>>> [5]https://camel.apache.org/components/4.0.x/geocoder-component.html >>>>>> >>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe, e-mail: dev-unsubscr...@baremaps.apache.org >>>>> <mailto:dev-unsubscr...@baremaps.apache.org> >>>>> For additional commands, e-mail: dev-h...@baremaps.apache.org >>>>> <mailto:dev-h...@baremaps.apache.org> >>>>> >>>> >>
signature.asc
Description: Message signed with OpenPGP
