SparksFyz opened a new pull request, #3868:
URL: https://github.com/apache/celeborn/pull/3868
### What changes were proposed in this pull request?
- Use the selected backend type when a MEMORY location falls back to
non-memory storage.
- Start from the first policy entry if the worker's creation policy
excludes MEMORY, avoiding an index of `-1`.
- Keep existing behavior for non-memory locations and eviction.
### Why are the changes needed?
When creating a file for a new MEMORY location (`evict=false`), the worker
skips memory if its memory file storage quota is exhausted. However, it still
passes MEMORY to `createDiskFile()`, so creation on an allowed remote backend
fails.
For example, a MEMORY location allowing `MEMORY,S3` cannot fall back to S3
when memory is unavailable, even though S3 is working. With local storage
disallowed, the worker returns `RESERVE_SLOTS_FAILED`.
The fix allows the worker to create the file on the selected fallback
backend. Replication is not required to trigger the issue.
### Does this PR resolve a correctness bug?
- [ ] Yes
This fixes partition reservation failures, not incorrect shuffle data
results.
### Does this PR introduce _any_ user-facing change?
- [x] Yes
Affected reservations can now succeed using an allowed fallback backend.
No new configuration is introduced.
### How was this patch tested?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
