Re: [DISCUSS] Introduce Table Multi-Location Management

Houliang Qi Fri, 13 Dec 2024 02:52:22 -0800

Hi Jingsong,

Thank for your reply.

Initially, the design involved adding `multi.locations` and
`default.write.location` as table properties. However, upon further
consideration, it seems more efficient to move these filesystem options to the
catalog itself and modify the catalog to support multiple warehouses. This way,
the table properties would only need to include a `data-file.external-path`, If
the attribute `data-file.external-path` is not empty, the default written data
will be written to the storage specified by `data-file.external-path`.

Additionally, when migrating hot and cold data later, users could select the
destination address for the migration based on the filesystem options provided
in the catalog.

I will implement a POC code based on this new design and share it with the team
for feedback.

Best,
Houliang

Thanks for starting this discussion.

Maybe we can just introduce an option: `data-file.external-path`? I
don't the usage of multi.locations.

In DataFileMeta, yes, we need to add another field: external_path.

About FileIO, I think you can implement an own hybrid FileIO created
by catalog options.

I think the general idea is fine, but we may need a POC code to
observe its complexity.

Best,
Jingsong

On Wed, Dec 11, 2024 at 7:15 PM Houliang Qi <[email protected]> wrote:

Hi Paimon devs,

I’d like to initiate a discussion: Introduce Table Multi-Location
Management[1], currently, the table's data can only be persisted in catalog's
warehouse path, which can not be modified once it created, However, users may
wish to store data from a table on different storage devices, or even store
data from different partitions of a table on different storage devices based on
their level of activity. So, the topic of this proposal is how to enable paimon
to support multi-location management for a single table.

Any opinions are welcome, looking forward to your feedback, thanks.

[1]
https://docs.google.com/document/d/1NhmOyxM16QmY_rVb3KJtCKRrU_nogIJv532U59qW7EI/edit?tab=t.0#heading=h.xlrl29nlxwpo

Re: [DISCUSS] Introduce Table Multi-Location Management

Reply via email to