JingsongLi opened a new pull request, #543: URL: https://github.com/apache/paimon-rust/pull/543
## What changed - expose Paimon `CHAR` / `VARCHAR` columns to DataFusion as `Utf8View` - strip storage-only Parquet field-id metadata from the DataFusion-facing schema - cast reader batches to the declared DataFusion schema while preserving zero-column row counts - normalize string scalars for equality, ordering, membership, range, and pattern residual filters - update MERGE, UPDATE, SQL context, table-provider, Parquet, and TPC-DS regression tests ## Why The TPC-DS benchmark uncovered a mismatch between the schema advertised to DataFusion and the physical string arrays returned by Paimon scans. String residual predicates also constructed `Utf8` scalars even when the decoded column used `Utf8View`, causing Arrow comparison kernels to reject otherwise valid predicates. This PR keeps the compatibility change self-contained. Runtime-filter, pruning, statistics, and reader batch-size work remain in later PRs. ## Validation - `cargo check -p paimon -p paimon-datafusion -p paimon-tpcds-bench` - `cargo test -p paimon test_evaluate_string_view_comparison_families` - `cargo test -p paimon-datafusion --lib test_datafusion_schema_hides_paimon_field_ids` - `cargo test -p paimon-datafusion --test pk_tables test_pk_aggregation_count_star_empty_projection -- --exact` - `cargo test -p paimon-tpcds-bench` - `cargo clippy -p paimon -p paimon-datafusion -p paimon-tpcds-bench --all-targets -- -D warnings` - `cargo fmt --all -- --check` A local full `paimon-datafusion --lib` run passed 278 tests; seven existing fixture-dependent tests reported `TableNotExist` because the external `/tmp/paimon-warehouse` tables were not populated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
