leaves12138 commented on PR #543:
URL: https://github.com/apache/paimon-rust/pull/543#issuecomment-5014266150
There is still one Python compatibility path that is not covered by the
current fix (`7ef01dc0`).
`pyarrow_compatible_batch` only runs for the bundled
`pypaimon_rust.datafusion.SQLContext.sql()` method. `PaimonCatalog` is also a
public Python API intended to be registered into DataFusion's native Python
`SessionContext`, and that path bypasses this conversion entirely.
I reproduced it locally against this head with PyArrow 23.0.1 and DataFusion
54.0.0:
```python
internal = SQLContext()
internal.register_catalog("paimon", {"warehouse": warehouse})
internal.sql("CREATE TABLE paimon.default.t (id INT, name STRING)")
internal.sql("INSERT INTO paimon.default.t VALUES (2, 'b'), (1, 'a')")
external = datafusion.SessionContext()
external.register_catalog_provider(
"paimon", PaimonCatalog({"warehouse": warehouse})
)
table = pa.Table.from_batches(
external.sql("SELECT id, name FROM paimon.default.t").collect()
)
```
The bundled `SQLContext` now returns `name: string` and `sort_by` succeeds,
but the external `SessionContext` path still returns:
```text
id: int32
name: string_view
```
and `table.sort_by("id")` fails with:
```text
ArrowNotImplementedError: Function 'array_take' has no kernel matching input
types (string_view, uint64)
```
Please also preserve compatible `Utf8` batches for the `PaimonCatalog` FFI
provider path, or explicitly define why the two public Python query APIs should
expose different string types. A regression test using
`SessionContext.register_catalog_provider` plus `sort_by` would cover this path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
