XiaoHongbo-Hope opened a new pull request, #10450: URL: https://github.com/apache/paimon/pull/10450
### Purpose While adding newer PyArrow support in #9821 for LeRobot integration, we found that optional AWS SDK request checksums can break writes to OSS and custom S3 endpoints. Pickled FileIO objects also need the same configuration when clients are reconstructed in already-running workers. Temporarily set `AWS_REQUEST_CHECKSUM_CALCULATION=WHEN_REQUIRED` during compatible client construction, restore the previous environment setting even on failure, and rebuild those clients on deserialization using the worker's PyArrow version. Covers `oss://` with the PyArrow implementation and custom endpoints under `s3://`, `s3a://`, and `s3n://`. Extracted from #9821 onto master. **Independent of #10425**: no Catalog/deletion changes, Boto3 dependency, or PyArrow dependency-range changes. Batch-delete compatibility and dependency relaxation remain separate work. The environment setting is process-wide. A lock coordinates Paimon client construction only; unrelated libraries constructing AWS clients concurrently are not protected by this lock. ### Validation - PyArrow 19.0.1: **51 passed, 1 skipped**; PyArrow 23.0.0: **52 passed** across client, OSS legacy, and FileIO tests. - Real PyArrow 23 requests to a local HTTP endpoint verify compatible parent/spawn-worker writes omit checksum trailers while an ordinary client retains them. - Covers environment restoration, S3 scheme aliases, old pickle state, and worker version recomputation. - Flake8, Python 3.6 syntax, and `git diff --check` pass. No live OSS/DLF rerun for this extracted PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
