SteNicholas opened a new issue, #578: URL: https://github.com/apache/paimon-cpp/issues/578
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon-cpp/issues) and found nothing similar. ### Motivation Paimon Java supports `GEOMETRY` and `GEOGRAPHY` as logical types backed by OGC Well-Known Binary (WKB). Paimon C++ currently does not recognize these types in its schema parser, which prevents C++ applications from using tables containing geospatial columns. Treating these columns as ordinary `BYTES` is insufficient: the geometry/geography distinction, coordinate reference system (CRS), geography edge interpolation algorithm, and Parquet logical annotations must survive schema and data round trips. The goal is to support geospatial columns in Paimon C++ and enable Java/C++ interoperability, following the Java [GeometryType](https://github.com/apache/paimon/blob/master/paimon-api/src/main/java/org/apache/paimon/types/GeometryType.java) and [GeographyType](https://github.com/apache/paimon/blob/master/paimon-api/src/main/java/org/apache/paimon/types/GeographyType.java) implementations. ### Solution Implement type and storage support, initially through Parquet. 1. **Type definitions and schema round trips** Add public type identification and schema parsing/serialization for `GEOMETRY`, `GEOMETRY(crs)`, `GEOGRAPHY`, `GEOGRAPHY(crs)`, and `GEOGRAPHY(crs, algorithm)`. Preserve nullability, field IDs, quoted CRS parameters, and nested occurrences in ROW, ARRAY, and MAP. Match Java defaults: CRS `OGC:CRS84` and geography algorithm `spherical`. Support `spherical`, `vincenty`, `thomas`, `andoyer`, and `karney`, with matching parameter validation. 2. **Arrow representation and Parquet interoperability** Reuse the existing Binary/WKB data path while preserving logical type metadata. Align Arrow field metadata with Java's `geoarrow.wkb` representation, including `crs` and `edges`. Read and write Parquet BYTE_ARRAY columns with native `GEOMETRY` or `GEOGRAPHY` logical annotations. Metadata on an ordinary Binary field alone must not be assumed to produce these annotations; the writer needs the appropriate GeoArrow extension-type or native Parquet schema mapping. Preserve logical metadata through projection and nested reads. 3. **Logical type semantics** Make type comparison, casts, and schema evolution aware of geospatial metadata. Distinguish BYTES, GEOMETRY, GEOGRAPHY, different CRS values, and different edge algorithms, while matching Java's case-insensitive CRS comparison. Review existing metadata-insensitive comparisons, including `DataField::operator==`, so a shared Binary physical representation does not imply logical compatibility. Align Java's restrictions on primary keys, partition keys, bucket keys, sequence fields, and clustering/sort columns. Reject unsupported file formats explicitly, including configured per-level and changelog formats. 4. **Statistics and filtering** Preserve null counts without treating WKB byte ordering as a valid geospatial min/max ordering. Apply this to Paimon statistics, Parquet statistics, and pruning paths, including nested columns. Optional bounding-box statistics can be added separately. 5. **Tests and documentation** Add schema, metadata, validation, and Parquet round-trip coverage, plus Java-to-C++ and C++-to-Java interoperability tests. Document the supported formats and type restrictions. ### Anything else? **Dependency consideration** Paimon C++ currently pins [Arrow 17.0.0](https://github.com/apache/paimon-cpp/blob/main/third_party/versions.txt), whose Parquet implementation lacks GEOMETRY/GEOGRAPHY logical types. [Arrow 21.0.0 introduced support](https://arrow.apache.org/blog/2025/07/17/21.0.0-release/). Evaluate a compatible Arrow upgrade or backport the required support, accounting for the project's C++17 requirement and existing Arrow patches. Extending only the Paimon schema parser will not resolve the native Parquet compatibility gap. **Acceptance criteria** - [ ] Both types round-trip through table schema JSON and Arrow schema metadata, including nested fields, custom CRS, algorithms, and nullability. - [ ] Java-written geospatial Parquet data is readable by C++, and C++-written data is readable by Java, preserving WKB values and logical type parameters. - [ ] Parquet footers contain the correct logical annotations, including nested columns. - [ ] Nulls and valid empty geometries round-trip correctly. - [ ] Unsupported key/order usages, incompatible type changes, and unsupported file formats fail with clear errors. - [ ] Statistics and filtering do not apply ordinary binary min/max semantics to geospatial values. - [ ] Existing BYTES behavior remains covered by regression tests. Reference: Java [ParquetSchemaConverter](https://github.com/apache/paimon/blob/master/paimon-format/src/main/java/org/apache/paimon/format/parquet/ParquetSchemaConverter.java) and [ArrowSchemaMetadata](https://github.com/apache/paimon/blob/master/paimon-format/src/main/java/org/apache/paimon/format/ArrowSchemaMetadata.java). The initial scope is logical types, storage, and interoperability. Spatial functions, coordinate transformations, and spatial indexes can be tracked separately. ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
