JunRuiLee opened a new pull request, #496: URL: https://github.com/apache/paimon-rust/pull/496
### Purpose Linked issue: close #495 Spark + Java Paimon support case-insensitive column matching on reads (Spark resolves projection/filter columns case-insensitively by default and normalizes them to the schema names before pushdown). `paimon-rust` had no equivalent — column resolution was always exact-case — so engines reading Paimon through the Rust core / C bindings / DataFusion / Python (e.g. Apache Doris) could not offer the behavior users get from Spark. This PR adds a `case-sensitive` table option (**default `true`**, matching Java Paimon and preserving current exact-match behavior). When a table sets `case-sensitive = false`, read-path column resolution matches names by **ASCII case-folding** and errors when a name is **ambiguous** (multiple schema fields collide under folding), mirroring Spark's `AMBIGUOUS` behavior. This lets a reader control, per table, whether column matching ignores case. ### Brief change log - **`CoreOptions::case_sensitive()`** — new `case-sensitive` option, default `true` (only an explicit `false` disables it). - **Projection (`ReadBuilder` / `resolve_projected_fields`)** — case-aware field lookup; case-insensitive duplicate detection so `["Name", "name"]` is reported as a duplicate rather than returning the column twice. - **`PredicateBuilder`** — `new_with_case_sensitive(fields, case_sensitive)` (the existing `new` delegates with `true`); `resolve_field` folds case when disabled, errors on ambiguity, and stores the **canonical schema name** in the leaf so downstream name-based lookups (e.g. index pruning) keep matching. - **DataFusion** — `FilterTranslator` and `classify_filter_pushdown` thread the option (read from the table's `CoreOptions`), so both scan pushdown and `supports_filters_pushdown` classification agree; an ambiguous reference is left unpushed (residual), never mis-resolved. - **Python `with_filter`** — predicate conversion reads the option and resolves literal types + builds predicates case-aware. - **C binding predicates** — the three `PredicateBuilder` sites use `new_with_case_sensitive`, and integer-datum coercion resolves the target column with the same case sensitivity (covers Doris and the Go binding). ASCII-only folding throughout (`eq_ignore_ascii_case`); no Unicode case folding. **Scope**: read path, top-level columns only. Nested/dotted paths, the `_ROW_ID` system column (kept an exact reserved name), and the write path are out of scope for this change. ### Tests - `cargo test -p paimon --lib spec::core_options` — option default/explicit/unrecognized. - `cargo test -p paimon --lib spec::predicate` — case-insensitive match, default-sensitive rejection, ambiguity error, canonical-name storage. - `cargo test -p paimon --lib table::read_builder` — case-sensitive/insensitive projection resolution + duplicate detection. - `cargo test -p paimon-datafusion --lib filter_pushdown` — case-insensitive pushdown + classification. - Python predicate tests (pyo3, `PYO3_PYTHON=python3.12`). ### API and Format - New public API: `PredicateBuilder::new_with_case_sensitive`; `CoreOptions::case_sensitive()`. Existing `PredicateBuilder::new` is unchanged (delegates to case-sensitive). - New table option `case-sensitive` (default `true`). No storage format change. Default behavior is unchanged. ### Documentation New table option `case-sensitive`; behavior described above and in the linked issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
