Hi Houliang, Thanks for starting this discussion.
Maybe we can just introduce an option: `data-file.external-path`? I don't the usage of multi.locations. In DataFileMeta, yes, we need to add another field: external_path. About FileIO, I think you can implement an own hybrid FileIO created by catalog options. I think the general idea is fine, but we may need a POC code to observe its complexity. Best, Jingsong On Wed, Dec 11, 2024 at 7:15 PM Houliang Qi <[email protected]> wrote: > > Hi Paimon devs, > > > I’d like to initiate a discussion: Introduce Table Multi-Location > Management[1], currently, the table's data can only be persisted in catalog's > warehouse path, which can not be modified once it created, However, users may > wish to store data from a table on different storage devices, or even store > data from different partitions of a table on different storage devices based > on their level of activity. So, the topic of this proposal is how to enable > paimon to support multi-location management for a single table. > > > Any opinions are welcome, looking forward to your feedback, thanks. > > > [1] > https://docs.google.com/document/d/1NhmOyxM16QmY_rVb3KJtCKRrU_nogIJv532U59qW7EI/edit?tab=t.0#heading=h.xlrl29nlxwpo > > >
