Very thanks for Houliang Qi prepare this doc and PR. After reading the context, I have some questions. 1、Could you add the design implementation details of 'Support file migration between different storage locations' in the doc? 2、When and how execute the file migration between different storage locations? 3、What business scenarios need to execute the file migration between different storage locations? 4、Can ‘data-file.external-path’ be removed after configuring a table? For example, at the beginning, the data was written in HDFS and then written to OSS. After a period of time, if I want to write new data back to HDFS, can I just remove the ‘data-file.external-path’ configuration?
Best, Weijun Wang On Fri, Dec 13, 2024 at 2:56 PM Jingsong Li <jingsongl...@gmail.com> wrote: > > Hi Houliang, > > Thanks for starting this discussion. > > Maybe we can just introduce an option: `data-file.external-path`? I > don't the usage of multi.locations. > > In DataFileMeta, yes, we need to add another field: external_path. > > About FileIO, I think you can implement an own hybrid FileIO created > by catalog options. > > I think the general idea is fine, but we may need a POC code to > observe its complexity. > > Best, > Jingsong > > On Wed, Dec 11, 2024 at 7:15 PM Houliang Qi <neuyi...@163.com> wrote: > > > > Hi Paimon devs, > > > > > > I’d like to initiate a discussion: Introduce Table Multi-Location > > Management[1], currently, the table's data can only be persisted in > > catalog's warehouse path, which can not be modified once it created, > > However, users may wish to store data from a table on different storage > > devices, or even store data from different partitions of a table on > > different storage devices based on their level of activity. So, the topic > > of this proposal is how to enable paimon to support multi-location > > management for a single table. > > > > > > Any opinions are welcome, looking forward to your feedback, thanks. > > > > > > [1] > > https://docs.google.com/document/d/1NhmOyxM16QmY_rVb3KJtCKRrU_nogIJv532U59qW7EI/edit?tab=t.0#heading=h.xlrl29nlxwpo > > > > > >
