jerry-024 commented on PR #798: URL: https://github.com/apache/paimon-rust/pull/798#issuecomment-5596434511
This PR should remain on hold until paimon-rust can use an official OpenDAL release containing [apache/opendal#8224](https://github.com/apache/opendal/pull/8224) (or an equivalent backport). The change from concurrency 1 to 8 exposes a memory regression in the currently used OpenDAL implementation. Concurrent task results retain their original retry input, including the upload payload, even after a part succeeds. The multipart writer uses a hard-coded prefetch of 8192, so completed payloads can accumulate while more parts are submitted. Eight concurrent network uploads therefore do not bound retained payloads to eight 8 MiB parts, and large index builds can run out of memory. The previous concurrency-1 path does not accumulate completed inputs this way. The upstream fix releases successful task inputs before result collection while retaining failed inputs for retry, preserving concurrent uploads. The upstream PR is merged, but merging upstream is not sufficient for this PR: the selected published dependency must actually contain the fix. Once that release is available, update the dependency and verify buffer release, retry correctness, and peak memory during a large 8-way upload before merging. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
