[
https://issues.apache.org/jira/browse/HUDI-309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16975525#comment-16975525
]
Raymond Xu commented on HUDI-309:
---------------------------------
[~vinoth]
Just trying to understand the new design..could you help clarify what's the
advantage of partitioning by action types? Is it to avoid redundant scanning?
Would that be better off if partitioning by datetime for even distribution?
> General Redesign of Archived Timeline for efficient scan and management
> -----------------------------------------------------------------------
>
> Key: HUDI-309
> URL: https://issues.apache.org/jira/browse/HUDI-309
> Project: Apache Hudi (incubating)
> Issue Type: New Feature
> Components: Common Core
> Reporter: Balaji Varadarajan
> Assignee: Vinoth Chandar
> Priority: Major
> Fix For: 0.5.1
>
> Attachments: Archive TImeline Notes by Vinoth 1.jpg, Archived
> Timeline Notes by Vinoth 2.jpg
>
>
> As designed by Vinoth:
> Goals
> # Archived Metadata should be scannable in the same way as data
> # Provides more safety by always serving committed data independent of
> timeframe when the corresponding commit action was tried. Currently, we
> implicitly assume a data file to be valid if its commit time is older than
> the earliest time in the active timeline. While this works ok, any inherent
> bugs in rollback could inadvertently expose a possibly duplicate file when
> its commit timestamp becomes older than that of any commits in the timeline.
> # We had to deal with lot of corner cases because of the way we treat a
> "commit" as special after it gets archived. Examples also include Savepoint
> handling logic by cleaner.
> # Small Files : For Cloud stores, archiving simply moves fils from one
> directory to another causing the archive folder to grow. We need a way to
> efficiently compact these files and at the same time be friendly to scans
> Design:
> The basic file-group abstraction for managing file versions for data files
> can be extended to managing archived commit metadata. The idea is to use an
> optimal format (like HFile) for storing compacted version of <commitTime,
> Metadata> pairs. Every archiving run will read <commitTime, Metadata> pairs
> from active timeline and append to indexable log files. We will run periodic
> minor compactions to merge multiple log files to a compacted HFile storing
> metadata for a time-range. It should be also noted that we will partition by
> the action types (commit/clean). This design would allow for the archived
> timeline to be queryable for determining whether a timeline is valid or not.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
