Hi, This should be a nontrivial improvement to Celeborn imo, thanks Ethan !
I had a few queries: a) Are we viewing this enhancement as a cache or as a tiered storage layer ? When going over it, I felt the proposal might be doing both - though leaning more as a cache, but wanted to get clarity. b) If we are modelling it as a tiered storage layer, it would be good to also think about what the right abstractions should be and not special case it just for memory. For example: Memory -> NVME/SSD -> Spinning Disk -> HDFS/S3 (With one or more being missing in a deployment) This would unify the way we handle evictions from one level to the next with a tiered view of the storage layer. Complexity of the implementation is definitely a consideration here though. Note, this might be out of scope for this proposal and work for the future as well - wanted to get your thoughts if it was considered ! c) If modelling as a cache, we should change the abstractions in the proposal slightly and hide the details behind the cache implementation. Read and write path would not need to worry about how it is handled internally. Regards, Mridul On Tue, Sep 19, 2023 at 10:27 PM Ethan Feng <ethanf...@apache.org> wrote: > Hello Celeborn community, > > I have a proposal to support memory file storage in Celeborn: > > https://docs.google.com/document/d/1SM-oOM0JHEIoRHTYhE9PYH60_1D3NMxDR50LZIM7uW0/edit?usp=sharing > > Would really appreciate feedback from the community on this proposal. > > > Thanks > Ethan >
