SteNicholas opened a new pull request, #580:
URL: https://github.com/apache/paimon-cpp/pull/580

   ### Purpose
   
   Linked issue: close #578
   
   Support GEOMETRY and GEOGRAPHY columns in Paimon C++, following Paimon Java. 
Values remain OGC Well-Known Binary (WKB) in Arrow `Binary` arrays, with the 
logical type, coordinate reference system (CRS) and geography edge 
interpolation algorithm preserved through schema and data round trips.
   
   - Add `FieldType::GEOMETRY` and `FieldType::GEOGRAPHY`, and parse and 
serialize their schema declarations, including quoted CRS parameters, 
nullability and nested ROW, ARRAY and MAP fields. Match Java's default 
`OGC:CRS84` CRS, case-insensitive CRS comparison and five geography edge 
algorithms.
   - Preserve `geoarrow.wkb` field metadata, field IDs and nullability through 
the Arrow C Data Interface, nested projections, field mapping, row tracking, 
data evolution and MAP shredding. Preserve declared MAP key nullability in 
table schemas while keeping stored keys non-null.
   - Make type comparison, casts and schema evolution distinguish BYTES, 
GEOMETRY, GEOGRAPHY, different CRS values and different edge algorithms. Reject 
geospatial primary, partition and bucket keys, sequence fields and 
sequence-group ordering keys, and clustering/sort columns.
   - Backport native Parquet GEOMETRY/GEOGRAPHY annotations to Arrow 17 while 
retaining C++17. Store WKB in BYTE_ARRAY columns and preserve the annotations 
and parameters for nested columns, including Java-compatible LIST element and 
MAP key/value field IDs.
   - Require Parquet data and changelog files, including per-level formats and 
operation-specific overrides, and reject geospatial schemas at unsupported 
format readers and writers.
   - Retain null counts while suppressing WKB min/max statistics in Paimon, the 
realtime store and Parquet. Ignore WKB bounds in file, row-group and page 
pruning, treat incomplete null counts as unknown, and support null predicates 
without changing ordinary BYTES or partition predicate behavior.
   
   Support is initially limited to Parquet. Spatial functions, coordinate 
transformations and spatial indexes are outside this change.
   
   ### Tests
   
   - Schema and type tests for default and custom CRS values, quoted and 
Unicode CRS parameters, all five geography edge algorithms, invalid metadata, 
logical equality, incompatible casts and schema evolution, and nested C Data 
Interface round trips.
   - Metadata preservation tests for projections, field mapping, MAP shredding, 
data evolution, row tracking and BLOB view resolution; validation tests for 
unsupported key/order usages and file formats, including per-level and 
operation-specific overrides.
   - Statistics and predicate tests for retained null counts, absent geospatial 
min/max bounds, incomplete row-group statistics, null predicates, rejected 
comparisons, page pruning and unchanged BYTES behavior.
   - Parquet tests for native annotations and field IDs, nested ROW/ARRAY/MAP 
values, nulls and valid empty geometries, sliced array adaptation, incompatible 
read schemas, default native parameters and escaped CRS round trips.
   - Integration tests for table reads with reordered projections and null 
predicates, and Java-to-C++ and C++-to-Java Parquet interoperability covering 
WKB values, logical parameters, nested annotations and statistics.
   
   ### API and Format
   
   - API: adds `FieldType::GEOMETRY` and `FieldType::GEOGRAPHY` in 
`include/paimon/defs.h`. `Schema::GetFieldType` identifies these types from 
Arrow field metadata.
   - Format: table schemas preserve GEOMETRY/GEOGRAPHY declarations and 
parameters. Arrow fields carry `geoarrow.wkb` metadata, and Parquet BYTE_ARRAY 
columns carry native GEOMETRY/GEOGRAPHY logical annotations compatible with 
Paimon Java. Existing BYTES columns retain their binary semantics.
   
   ### Documentation
   
   Yes. `docs/source/user_guide/data_types.rst` documents the geospatial schema 
syntax, CRS and edge algorithms, Arrow metadata, Parquet storage, nested fields 
and MAP key nullability, key/order and cast restrictions, statistics and 
predicates, and the Java/C++ interoperability test.
   
   ### Generative AI tooling
   
   Generated-by: OpenAI Codex (GPT-6)
   
   🤖 Generated with [OpenAI Codex](https://openai.com/codex/)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to