leaves12138 commented on PR #543:
URL: https://github.com/apache/paimon-rust/pull/543#issuecomment-5013870901
I found two blockers in the current revision (`94d76ed`):
1. **`Utf8View` leaks through the Python API and breaks common PyArrow
operations.**
`datafusion_arrow_schema` now advertises Paimon string columns as
`Utf8View`, and `bindings/python/src/context.rs` exports the resulting batches
to PyArrow without conversion. With the CI version of PyArrow (23.0.1), both
write round-trip tests fail because `Table.sort_by()` uses `take`, for which
PyArrow has no `string_view` kernel:
```text
ArrowNotImplementedError: Function 'array_take' has no kernel matching
input types (string_view, uint64)
```
This is not only a test expectation issue: existing Python consumers can
receive batches on which common PyArrow operations fail. Please preserve `Utf8`
at the Python boundary (or provide another explicit compatibility strategy).
Merely changing the tests to avoid `sort_by()` would leave the user-facing
regression in place.
2. **The DataFusion integration suite was not migrated to the new string
array type.**
The `integration (datafusion)` job currently reports 25 failures in
`append_merge_into`. Helpers such as `collect_int_int_str` and
`collect_int_str` in `crates/integrations/datafusion/tests/common/mod.rs` still
unconditionally downcast query results to `StringArray`, so they panic after
the schema change. There are additional direct `StringArray` downcasts in
`pk_tables.rs`, `read_tables.rs`, and other integration tests.
Please update the full integration suite. A shared helper that reads
`Utf8`, `LargeUtf8`, and `Utf8View` would be preferable to coupling every test
to one physical representation.
The focused residual-filter, DataFusion schema, zero-column projection, and
TPC-DS tests pass locally; I did not find another wrong-read in those
implementation paths.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
