XiaoHongbo-Hope opened a new pull request, #9821: URL: https://github.com/apache/paimon/pull/9821
### Purpose Support PyArrow 22 and exercise it in the Python 3.12+ CI lanes. The current CI briefly installs PyArrow 24 but later replaces it with 18.1 for Vortex, so the declared `<20` ceiling has no high-version coverage. PyArrow 22 also exposes three issues when its S3 filesystem is used against Alibaba Cloud OSS: - recent AWS SDKs send `STREAMING-UNSIGNED-PAYLOAD-TRAILER` by default, which OSS rejects with `NotImplemented`; - `create_dir` performs `HeadBucket`, which can be denied for roles that have the required object permissions; - recursive `DeleteObjects` fails with `MissingArgument` after selecting required-only request checksums. This change widens the tested dependency range to `<23`, selects PyArrow 22 in CI, and keeps Vortex on that selected version. The legacy OSS backend requests checksums only when required, represents empty logical directories with a child marker, and deletes recursively with individual object operations before removing empty directory markers. PyArrow `<16` handling and the Jindo backend are unchanged. ### Tests - PyArrow 19.0.1: `49 passed` in `oss_legacy_mode_test.py` and `file_io_test.py` - PyArrow 22.0.0: `49 passed` in the same tests - flake8 with `paimon-python/dev/cfg.ini` on the changed Python files - dependency resolution check for `vortex-data==0.70.0` with `pyarrow==22.0.0` - real OSS canary: object write/stat/list/read plus Paimon create database, create table, commit, scan, drop table, and drop database - LeRobot local reader validation on PyArrow 22: exact equality for single-frame and 8-frame samples, including decoded video pixels The same-machine LeRobot throughput medians changed from 273.13 to 273.78 samples/s for single-frame reads and from 29.47 to 29.65 samples/s for 8-frame reads, within normal run-to-run variation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
