deepakpanda93 commented on issue #15779:
URL: https://github.com/apache/hudi/issues/15779#issuecomment-5253814013
Closing — this was documented in #19572 (merge commit 242ea80), now live on
the site.
**Scope note up front.** The issue title asks for log compaction in the
*Write Operations* docs, but `write_operations.md` had nowhere to link to: log
compaction shipped in 0.13.0 and had no user-facing prose anywhere. It appeared
only as a one-line `LOGCOMPACTION` action definition in `timeline.md`, passing
mentions in `hudi_stack.md` / `storage_layouts.md` / `metadata.md`,
auto-generated rows in `configurations.md`, and a paragraph in the 1.0 tech
spec. Critically, `compaction.md` — where a reader looking for compaction
actually lands — never mentioned it at all. So the fix writes the substantive
section where it belongs and points at it from the write path.
**What was added:**
- **`compaction.md`** — a new `## Log Compaction` section covering what it
does, what it costs, the timeline actions, the data-table configs, the
metadata-table configs, and a link to RFC-48.
- **`write_operations.md`** — **Write path** step 9 gains a pointer, gated
on `hoodie.log.compaction.inline` so it does not read as on-by-default.
- **`timeline.md`** — the `LOGCOMPACTION` bullet now links to the new
section instead of dead-ending.
**Five configs are documented:** `hoodie.log.compaction.inline` and
`hoodie.log.compaction.blocks.threshold` for the data table,
`hoodie.metadata.log.compaction.enable` and
`hoodie.metadata.log.compaction.blocks.threshold` for the metadata table, plus
a `:::caution` explaining that `hoodie.log.compaction.enable` — which does
appear in the configuration reference — is not a user switch at all. Names,
defaults, `Config Param` and `Since Version` values were all read from
`HoodieCompactionConfig` / `HoodieMetadataConfig` at `release-1.2.0`.
**Details the section deliberately calls out**, each of which is a trap for
someone reading the timeline or measuring the effect:
- **The completed instant is a `deltacommit`, not a `logcompaction`.**
`ActiveTimelineV2.transitionLogCompactionInflightToComplete` builds the
completed instant with `DELTA_COMMIT_ACTION`, so `logcompaction` covers only
`requested` and `inflight`. Someone listing `.hoodie/` for a completed
`logcompaction` instant would otherwise conclude it never ran.
- **It costs storage.** The stitched block is appended rather than replacing
anything, so superseded blocks survive until the next full compaction and
clean, and one run can emit more than one block if merged output exceeds the
block size (RFC-48). Existing material described only the read benefit.
- **`blocks.threshold` gates scheduling, not the trigger** — none of its
read sites are guarded by `inlineLogCompactionEnabled()` — and it fires on log
*file* count as well as block count.
- **Async exists for the metadata table from 1.2.0** via
`hoodie.metadata.table.service.manager.enabled` with `.actions=logcompaction`,
so a flat "no async" claim would have been wrong. Log compaction is also not
exposed through Flink options at all, which matters because the second half of
that page is Flink offline compaction.
- **A pending metadata-table log compaction blocks metadata-table major
compaction** (HUDI-7533).
**Version scope: `next` + `version-1.2.0` only.** This started out applied
to all 1.x copies; committer review found two reasons that was wrong, both
verified against the Java source at each tag:
1. **Inline log compaction is broken before 1.0.2.** `WriteOperationType`
serializes `LOG_COMPACT` as `"logcompact"` at 1.0.0 and 1.0.1 with no matching
case in `fromValue`, so it reaches `default: throw new HoodieException("Invalid
value of Type.")`. That is HUDI-9220, fixed in 1.0.2
(`LOG_COMPACT("log_compact")` with the case present). Documenting
`hoodie.log.compaction.inline` on 1.0.0/1.0.1 would have told users to enable
something that throws.
2. **The read-amplification benefit only holds from 1.2.0.** The
`COMPACTED_BLOCK_TIMES` skip lives in `scanInternalV2`, and `scanInternal`
picks the path at runtime from a config defaulting to `false` that is not
reachable on the Spark data-table read path in earlier releases. 1.0.0, 1.0.2
and 1.1.1 all still branch on `enableOptimizedLogBlocksScan`; 1.2.0 has zero
references to either branch, the skip having become unconditional (#17520).
`0.14.x` and `0.15.x` were already excluded, and there is no `version-1.1.0`
folder. Reverting `write_operations.md` on the dropped versions mattered as
much as reverting `compaction.md` — leaving the pointer behind would have left
a link to a `#log-compaction` anchor that no longer exists on those pages.
**One pre-existing problem found but not fixed here:** `metadata.md:151`
links to `compaction.md#delegating-mdt-compaction-to-an-external-platform`, a
heading that exists in no version. The anchor is dangling and `onBrokenAnchors`
is unset in `docusaurus.config.js` so it defaults to `warn`. It predates this
change and was left out rather than growing the scope — worth a follow-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
