davidchaava opened a new pull request, #28832: URL: https://github.com/apache/flink/pull/28832
## What is the purpose of the change This PR adds Parquet read/write support for the GEOGRAPHY logical type using GeoParquet-compatible metadata on top of Flink's existing Parquet version. GEOGRAPHY values are stored as Parquet `BINARY` values with WKB encoding. The writer emits GeoParquet `geo` key-value metadata with `encoding=WKB` and `edges=spherical`, allowing downstream systems such as BigQuery to recognize the column as GEOGRAPHY. This keeps the implementation compatible with Flink's current Parquet dependency (`parquet-mr 1.15.2`) and does not require upgrading to the native Parquet GEOGRAPHY logical type. ## Brief change log - Map Flink GEOGRAPHY to Parquet `BINARY`. - Emit GeoParquet 1.1.0 metadata for top-level GEOGRAPHY columns. - Add GEOGRAPHY support to Parquet row writing. - Add GEOGRAPHY support to Parquet vectorized reading, including nested values. - Add Parquet writer and reader coverage for GEOGRAPHY, nested GEOGRAPHY, and mixed schemas. ## Related specifications This PR follows the [GeoParquet 1.1.0 specification](https://geoparquet.org/releases/v1.1.0/) for file-level `geo` metadata. GEOGRAPHY values are stored as Parquet `BINARY` with GeoParquet metadata using `encoding=WKB` and `edges=spherical`. This keeps the implementation compatible with Flink's current Parquet dependency (`parquet-mr 1.15.2`) instead of relying on the native Parquet GEOGRAPHY logical type. ## Related changes The SQL ST functions are tracked separately in [apache/flink#28788](https://github.com/apache/flink/pull/28788). This PR focuses on Parquet format interoperability for GEOGRAPHY. The ST functions PR provides the SQL constructor/accessor functions used by user-facing SQL workflows. ## Verifying this change This change is covered by unit tests in `flink-parquet`. I also validated a generated GeoParquet file manually with a local Flink pipeline: - Wrote GEOGRAPHY values through the Flink filesystem connector with `format='parquet'`. - Read the same Parquet data back through Flink SQL. - Verified the generated file contains GeoParquet 1.1.0 metadata and was written with Flink's current Parquet version: ```text {"version":"1.1.0","primary_column":"location","columns":{"location":{"encoding":"WKB","geometry_types":[],"edges":"spherical"}}} parquet-mr version 1.15.2 ``` I additionally loaded the generated Parquet file into BigQuery and verified that BigQuery recognized the column as GEOGRAPHY. BigQuery test table: ```text healthy-skill-501608-t7.akvelon.legacy_geography_flink_20260727 ``` BigQuery schema: ```text case_id: integer subtype: string location: geography ``` Spatial query validation: ```sql SELECT case_id, subtype, ST_ASTEXT(location) AS wkt FROM `healthy-skill-501608-t7.akvelon.legacy_geography_flink_20260727` ORDER BY case_id ``` Result: ```text 1 POINT POINT(-122.4194 37.7749) 2 POLYGON POLYGON((-122.5 37.7, -122.4 37.7, -122.4 37.8, -122.5 37.8, -122.5 37.7)) 3 NULL NULL ``` ## Does this pull request potentially affect one of the following parts - Dependencies: no - The public API: yes, GEOGRAPHY can now be represented in Parquet format - The serializers: no - The runtime per-record code paths: yes, Parquet GEOGRAPHY read/write paths - Anything that affects deployment or recovery: no - The S3 file system connector: no ## Documentation This PR does not add end-user documentation. GEOGRAPHY SQL functions and broader user-facing SQL behavior are handled in the related ST functions PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
