nsivabalan commented on issue #18135:
URL: https://github.com/apache/hudi/issues/18135#issuecomment-6356000305
Hey @shangxinli,
Thanks for the write up. before we dive into the approaches, I would like
to align on the requirements.
Can you confirm these are the requirements.
```
R1. No reads of cold files at registration/bootstrap. Only listing (names
and sizes) is allowed.
R2. One table, all tiers. Hot/warm partitions are native Hudi; cold
partitions stay at the old Hive path. One query sees all.
R3. A partition lives in exactly one place: under the Hudi table base path
or under the old base path, never both. in other words, if a partition is
tagged as cold (external path), it can never move to native hudi layer or new
target path. This goes hand in hand w/ R4.
R4. Old partitions are read-only. If one needs updates, the whole partition
is first moved under the table base path. But the move as such will not happen
seamlessly during a new write. Its an admin operation outside of ingestion.
R5: Any kind of writes to older partitions are rejected (ingestion (insert,
upsert, deletes), table services, insert overwrite, delete_partition etc). Only
reads are allowed based on partition predicates.
R6. Correct results or a clear failure, never silently missing rows: across
readers with and without the metadata table (MDT), atleast for readers who used
the patched version. For older readers who may not patch the support we plan to
develop, we can't guarantee fail fast.
R7: Supports only single partition column. preferrably only datestr style
partitions. So that during bootstrap, users can provide a predicate for list of
partitions to be kept in cold storage.
R8: As long as no one issues reads to older partitions, hudi should not
resurrect the data files for those cold partitions to hot tier. But if someone
were to trigger a read either w/o any partition predicate or by supplying a
predicate matching cold partitions, Hudi will serve data for the cold
partitions, on which case, data files will be resurrected to hot storage.
R9. No breaking storage-format change unless versioned and gated.
```
Clarification:
```
1: Would any of these be mutable streams? if so, global indexes? and whats
the configuration for `update.partition.path`? Irrespective of the config
value, if we are looking for global index, our indexing will result in reading
record keys from all partitions. How would we guard against cold data files
resurrecting to hot tier in this case?
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
