kerwin-zk opened a new pull request, #8029:
URL: https://github.com/apache/paimon/pull/8029
### Purpose
Make the Daft Paimon read source serializable when running with Ray.
Previously, `PaimonDataSource` and fallback read tasks could retain live
`FileStoreTable`, `FileIO`, `StorageConfig`, or `TableRead` objects. With
remote filesystems such as OSS/Jindo, Ray failed to serialize the execution
plan because those objects may contain non-picklable PyArrow filesystem
state.
```
RuntimeError: Failed to serialize: OtherString("TypeError: no default
__reduce__ due to non-trivial __cinit__")
```
### Tests
CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
