SteNicholas opened a new issue, #578:
URL: https://github.com/apache/paimon-cpp/issues/578

   ### Search before asking
   
   - [x] I searched in the 
[issues](https://github.com/apache/paimon-cpp/issues) and found nothing similar.
   
   ### Motivation
   
   Paimon Java supports `GEOMETRY` and `GEOGRAPHY` as logical types backed by 
OGC Well-Known Binary (WKB). Paimon C++ currently does not recognize these 
types in its schema parser, which prevents C++ applications from using tables 
containing geospatial columns.
   
   Treating these columns as ordinary `BYTES` is insufficient: the 
geometry/geography distinction, coordinate reference system (CRS), geography 
edge interpolation algorithm, and Parquet logical annotations must survive 
schema and data round trips.
   
   The goal is to support geospatial columns in Paimon C++ and enable Java/C++ 
interoperability, following the Java 
[GeometryType](https://github.com/apache/paimon/blob/master/paimon-api/src/main/java/org/apache/paimon/types/GeometryType.java)
 and 
[GeographyType](https://github.com/apache/paimon/blob/master/paimon-api/src/main/java/org/apache/paimon/types/GeographyType.java)
 implementations.
   
   ### Solution
   
   Implement type and storage support, initially through Parquet.
   
   1. **Type definitions and schema round trips**
   
      Add public type identification and schema parsing/serialization for 
`GEOMETRY`, `GEOMETRY(crs)`, `GEOGRAPHY`, `GEOGRAPHY(crs)`, and `GEOGRAPHY(crs, 
algorithm)`. Preserve nullability, field IDs, quoted CRS parameters, and nested 
occurrences in ROW, ARRAY, and MAP.
   
      Match Java defaults: CRS `OGC:CRS84` and geography algorithm `spherical`. 
Support `spherical`, `vincenty`, `thomas`, `andoyer`, and `karney`, with 
matching parameter validation.
   
   2. **Arrow representation and Parquet interoperability**
   
      Reuse the existing Binary/WKB data path while preserving logical type 
metadata. Align Arrow field metadata with Java's `geoarrow.wkb` representation, 
including `crs` and `edges`.
   
      Read and write Parquet BYTE_ARRAY columns with native `GEOMETRY` or 
`GEOGRAPHY` logical annotations. Metadata on an ordinary Binary field alone 
must not be assumed to produce these annotations; the writer needs the 
appropriate GeoArrow extension-type or native Parquet schema mapping. Preserve 
logical metadata through projection and nested reads.
   
   3. **Logical type semantics**
   
      Make type comparison, casts, and schema evolution aware of geospatial 
metadata. Distinguish BYTES, GEOMETRY, GEOGRAPHY, different CRS values, and 
different edge algorithms, while matching Java's case-insensitive CRS 
comparison.
   
      Review existing metadata-insensitive comparisons, including 
`DataField::operator==`, so a shared Binary physical representation does not 
imply logical compatibility.
   
      Align Java's restrictions on primary keys, partition keys, bucket keys, 
sequence fields, and clustering/sort columns. Reject unsupported file formats 
explicitly, including configured per-level and changelog formats.
   
   4. **Statistics and filtering**
   
      Preserve null counts without treating WKB byte ordering as a valid 
geospatial min/max ordering. Apply this to Paimon statistics, Parquet 
statistics, and pruning paths, including nested columns. Optional bounding-box 
statistics can be added separately.
   
   5. **Tests and documentation**
   
      Add schema, metadata, validation, and Parquet round-trip coverage, plus 
Java-to-C++ and C++-to-Java interoperability tests. Document the supported 
formats and type restrictions.
   
   ### Anything else?
   
   **Dependency consideration**
   
   Paimon C++ currently pins [Arrow 
17.0.0](https://github.com/apache/paimon-cpp/blob/main/third_party/versions.txt),
 whose Parquet implementation lacks GEOMETRY/GEOGRAPHY logical types. [Arrow 
21.0.0 introduced 
support](https://arrow.apache.org/blog/2025/07/17/21.0.0-release/). Evaluate a 
compatible Arrow upgrade or backport the required support, accounting for the 
project's C++17 requirement and existing Arrow patches. Extending only the 
Paimon schema parser will not resolve the native Parquet compatibility gap.
   
   **Acceptance criteria**
   
   - [ ] Both types round-trip through table schema JSON and Arrow schema 
metadata, including nested fields, custom CRS, algorithms, and nullability.
   - [ ] Java-written geospatial Parquet data is readable by C++, and 
C++-written data is readable by Java, preserving WKB values and logical type 
parameters.
   - [ ] Parquet footers contain the correct logical annotations, including 
nested columns.
   - [ ] Nulls and valid empty geometries round-trip correctly.
   - [ ] Unsupported key/order usages, incompatible type changes, and 
unsupported file formats fail with clear errors.
   - [ ] Statistics and filtering do not apply ordinary binary min/max 
semantics to geospatial values.
   - [ ] Existing BYTES behavior remains covered by regression tests.
   
   Reference: Java 
[ParquetSchemaConverter](https://github.com/apache/paimon/blob/master/paimon-format/src/main/java/org/apache/paimon/format/parquet/ParquetSchemaConverter.java)
 and 
[ArrowSchemaMetadata](https://github.com/apache/paimon/blob/master/paimon-format/src/main/java/org/apache/paimon/format/ArrowSchemaMetadata.java).
   
   The initial scope is logical types, storage, and interoperability. Spatial 
functions, coordinate transformations, and spatial indexes can be tracked 
separately.
   
   ### Are you willing to submit a PR?
   
   - [ ] I'm willing to submit a PR!
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to