eolivelli opened a new pull request, #3610: URL: https://github.com/apache/celeborn/pull/3610
NOTE: this PR is stacked on top of https://github.com/apache/celeborn/pull/3608 Please consider only 756d25e49ef5f0321b90002d319b72924b9f4196 ### What changes were proposed in this pull request? Handle the eviction to a different location type. ### Why are the changes needed? Because it may happen that a MEMORY file is to be evicted to another storage type (i.e. S3). This does not work. Usually, as described in tests in #3608 when you have tiered storage, the primary partition type is usually not MEMORY, but it may happen that during a partition split the client decides to use MEMORY. This patch fixes the problem on the worker side. An alternative fix would be to change the behavior of the client, and simulate what the master does when offering slots. Such a change would be more involved and it won't make the server side resilient to this scenario. ### Does this PR resolve a correctness bug? No ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? - New integration tests - Manual testing on real k8s cluster with S3 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
