voonhous commented on code in PR #19572:
URL: https://github.com/apache/hudi/pull/19572#discussion_r3757396435
##########
website/docs/compaction.md:
##########
@@ -283,6 +283,41 @@ Offline compaction needs to submit the Flink task on the
command line. The progr
The retry options (`--retry`, `--retry-last-failed-job`,
`--job-max-processing-time-ms`) are only effective in single-run mode, not in
service mode. Service mode has implicit retry semantics via its continuous
monitoring loop. A warning will be logged if `--retry-last-failed-job` is
enabled but `--job-max-processing-time-ms` is not set to a positive value.
:::
+## Log Compaction
+
+Log compaction is a minor compaction for Merge-on-Read tables. Rather than
merging log files into a new base file, it
+stitches several small log blocks into a larger one within the same file
group. A file group that receives frequent
+small updates can therefore be kept efficient without paying the cost of
rewriting its base file. Readers skip the log
+blocks that have already been stitched, so read amplification is reduced as
well. Log compaction appears on the timeline
+as a `logcompaction` action.
+
+| Config Name | Default | Description |
+|---|---|---|
+| `hoodie.log.compaction.inline` | `false` (Optional) | When set to true, the
log compaction service is triggered after each write. While being simpler
operationally, this adds extra latency on the write path.<br /><br />`Config
Param: INLINE_LOG_COMPACT`<br />`Since Version: 0.13.0` |
+| `hoodie.log.compaction.blocks.threshold` | `5` (Optional) | Log compaction
can be scheduled once the number of log blocks crosses this threshold.
Effective only when log compaction is enabled via
`hoodie.log.compaction.inline`.<br /><br />`Config Param:
LOG_COMPACTION_BLOCKS_THRESHOLD`<br />`Since Version: 0.13.0` |
+
+:::note
+`hoodie.log.compaction.inline` is the only built-in way to schedule log
compaction on a data table. There is no
+asynchronous log compaction service, SQL procedure, Hudi CLI command, or
standalone utility for it, unlike compaction.
+Programmatic scheduling is available through the write client's
`scheduleLogCompaction` and `logCompact` methods.
Review Comment:
The caution is a real improvement and the stall warning is right. Your Flink
correction is also right and mine was sloppy:
`CompactionUtil.scheduleMetadataCompaction` does try `scheduleCompaction` first
and only falls through to `scheduleLogCompaction` when that schedules nothing
and `isLogCompactionEnabled()` is true. "Flink does not need this" is accurate.
But chasing `hoodie.table.service.manager.uris` through the code turned up
something that changes the recommendation: **Hudi has no code path that hands a
log compaction to the table service manager.** Setting `actions=logcompaction`
suppresses inline execution and hands off to nothing, under any configuration.
The handoff site is reached. `scheduleLogCompaction` ->
`scheduleTableService(LOG_COMPACT)` -> `scheduleTableServiceInternal`, which
calls `delegateToTableServiceManager(tableServiceType, table)` at
`BaseHoodieTableServiceClient:765`. That method is the only place a
`HoodieTableServiceManagerClient` is ever constructed, so it is the only place
`hoodie.table.service.manager.uris` is ever read. It cannot dispatch a log
compaction, for two independent reasons:
```java
private Option<String> delegateToTableServiceManager(TableServiceType
tableServiceType, HoodieTable table) {
if
(!config.getTableServiceManagerConfig().isEnabledAndActionSupported(ActionType.compaction))
{
return Option.empty(); // <-- hardcoded to
`compaction`, not tableServiceType
}
HoodieTableServiceManagerClient tableServiceManagerClient = new
HoodieTableServiceManagerClient(...);
switch (tableServiceType) {
case COMPACT: return tableServiceManagerClient.executeCompaction();
case CLUSTER: return tableServiceManagerClient.executeClustering();
case CLEAN: return tableServiceManagerClient.executeClean();
default: // <-- LOG_COMPACT lands
here
log.info("Not supported delegate to table service manager,
tableServiceType : " + tableServiceType.getAction());
return Option.empty();
}
}
```
1. The guard asks `isEnabledAndActionSupported(ActionType.compaction)`
regardless of the type passed in. With exactly the config the paragraph
recommends (`actions=logcompaction`), the actions set is `{"logcompaction"}`,
which does not contain `compaction`, so it returns empty before a client is
built and the URI is never read.
2. Even if someone set `actions=compaction,logcompaction` to get past the
guard, `LOG_COMPACT` has no case in the switch and falls to `default`.
And the client has no log compaction operation to call anyway --
`HoodieTableServiceManagerClient` exposes only `executeCompaction()`,
`executeClean()` and `executeClustering()`.
Meanwhile `shouldDelegateToTableServiceManager(metadataWriteConfig,
ActionType.logcompaction)` is a pure config predicate (`RunsTableService:39` ->
`isEnabledAndActionSupported`), so it returns true and the writer skips
execution at `HoodieBackedTableMetadataWriter:2239`. That is the whole effect
of the setting.
So the net behaviour of following the paragraph is the stall you now warn
about, with no route out of it short of an external system independently
polling the MDT timeline for pending `logcompaction` instants -- which Hudi
neither performs nor signals. The `logcompaction` value listed as supported in
`hoodie.metadata.table.service.manager.actions`'s generated description is
aspirational; consistent with HUDI-3475 not having landed.
Given that, pointing readers at `hoodie.table.service.manager.uris` sends
them somewhere the code never looks. Suggest inverting the paragraph rather
than qualifying it -- the config combination is worth documenting precisely
because `configurations.md` advertises `logcompaction` as supported and someone
will try it:
> The metadata table runs its own log compaction, controlled by a separate
pair of configs. On Flink no extra configuration is needed: its compaction
pipeline schedules and executes metadata table log compaction directly.
and replace the caution with:
> `hoodie.metadata.table.service.manager.actions` lists `logcompaction` as a
supported action, but do not use it to move metadata table log compaction off
the writer. Setting it stops the writer from running log compaction inline
without handing the work anywhere: Hudi has no dispatch path for log compaction
to the table service manager, and does not ship a table service manager in any
case. The result is that pending `logcompaction` instants accumulate on the
metadata table and, per the note below, metadata table compaction stops being
scheduled as well.
That keeps the warning, drops the URI (which is only read on the data-table
compaction, clustering and clean paths), and stops presenting delegation as a
working option.
Happy to be wrong here if you find a dispatch path I missed -- I searched
for every construction of `HoodieTableServiceManagerClient` at `release-1.2.0`
and on `apache/master` and found the single site above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
