deepakpanda93 commented on issue #19693:
URL: https://github.com/apache/hudi/issues/19693#issuecomment-5414829269
We reproduced this end-to-end in a local Spark 3.5 environment, and
validated the workaround. Short answers first, then the evidence.
1. **Does `USE_TRANSITION_TIME` fix the hollow commit error?** Yes — the
error goes away and no data is lost.
2. **Is there a risk of data loss?** No, but there is a correctness catch in
the other direction: it silently **re-reads already-consumed commits** unless
you also change what you store as `extractId`.
3. **Can you switch without regenerating your lake?** Yes — no regeneration
needed. The only thing that changes is your checkpoint value, not your data.
---
## What we ran
* Spark 3.5.7, local filesystem, MOR table, 4 commits (`ids 1-10`, `11-20`,
`21-30`, `31-40`).
* Hollow commit created exactly as in your step 3 — a stuck
`<ts>.deltacommit.requested` on the timeline, positioned between commit 2 and
commit 3 so completed commits exist after it. It has no matching `.inflight` or
completed file and was present for every read below.
* Incremental read with `begin.instanttime` = commit 1, so a fully correct
answer is **ids 11-40 (30 rows)**.
**Version note:** Hudi 0.14.1 has no open-source Spark 3.5 bundle
(`hudi-spark3.5-bundle_2.12` starts at 0.15.0 — EMR 7.1.0 ships Amazon's own
build), so we used **0.15.1**. Before substituting we diffed
`TimelineUtils.handleHollowCommitIfNeeded` between `release-0.14.1` and
`release-0.15.1`: the only difference is a `String.format` →
parameterized-SLF4J logging change. The logic is identical.
## Result on the 0.14.x / 0.15.x code path (table version 6)
| `hoodie.read.timeline.holes.resolution.policy` | Result |
|---|---|
| *(default)* | **ERROR** — `Found hollow commit: '20260825173626458'` |
| `FAIL` | same error |
| `BLOCK` | OK — **10 rows, ids 11-20** |
| `USE_TRANSITION_TIME` | OK — **40 rows, ids 1-40** |
This confirms your reading of the options. `BLOCK` truncates the range at
the hollow commit, so nothing past it flows until the stuck producer finishes —
the multi-hour stall you cannot afford.
## The catch with `USE_TRANSITION_TIME`
It returned **40 rows, not the expected 30**. Nothing was lost — batch 1 was
re-delivered.
With this policy, `begin.instanttime` stops meaning *instant time* and
starts meaning *completion time*. Every commit completes strictly after its
instant time, so an instant-time checkpoint pulls the previous commit back into
range.
## Verified workaround for 0.14.1
Set the policy **and** switch `extractId` to a completion time in the same
change. On table version 6 the completion time is not in the instant filename
(`20260825173621649.deltacommit`) — it is the instant file's modification time,
exposed as `HoodieInstant.getStateTransitionTime()`. This ran successfully
against 0.15.1:
```python
jvm = spark._jvm
hconf = spark._jsc.hadoopConfiguration()
storage_conf =
jvm.org.apache.hudi.hadoop.fs.HadoopFSUtils.getStorageConf(hconf)
mc = (jvm.org.apache.hudi.common.table.HoodieTableMetaClient.builder()
.setConf(storage_conf).setBasePath(TABLE_PATH).build())
instants = list(mc.getActiveTimeline().getCommitsTimeline()
.filterCompletedInstants().getInstants())
# instant time -> completion time
for i in instants:
print(i.getTimestamp(), "->", i.getStateTransitionTime())
next_extract_id = max(i.getStateTransitionTime() for i in instants)
```
Producing, on our table:
```
20260825173621649 -> 20260825173625078
20260825173626457 -> 20260825173628563
20260825173629821 -> 20260825173630326
20260825173631586 -> 20260825173632003
```
Then read with that value:
```python
(spark.read.format("hudi")
.option("hoodie.datasource.query.type", "incremental")
.option("hoodie.read.timeline.holes.resolution.policy",
"USE_TRANSITION_TIME")
.option("hoodie.datasource.read.begin.instanttime", next_extract_id)
.load(TABLE_PATH))
```
Measured results, with the hollow commit still stuck on the timeline:
| `begin.instanttime` | Rows |
|---|---|
| C1 **instant** time `20260825173621649` | 40 rows, ids 1-40 — re-reads
batch 1 |
| C1 **completion** time `20260825173625078` | **30 rows, ids 11-40** —
correct |
| max completion `20260825173632003` (next run) | **0 rows** — clean steady
state |
The last row is the important one: the checkpoint advances and converges, so
consecutive runs neither stall nor re-read.
For the cutover run, take the current max completion time as your new
`extractId` baseline. No table rewrite or regeneration is required.
## On Hudi 1.x this failure mode does not exist
Incremental reads select their implementation by table version
(`DefaultSource.scala`): table version >= 8 routes to the **V2** incremental
relation, which is completion-time native and never consults hollow-commit
handling — there is no `HollowCommitHandling` reference anywhere in
`IncrementalRelationV2` / `MergeOnReadIncrementalRelationV2`.
On **1.2.0**, with the identical hollow commit in place, **all four policy
values returned 40 rows with no error.** The config is inert there.
We also verified your "no data loss between consecutive runs" requirement on
1.2.0 with the hollow commit present:
* **Run A** from commit 1's completion time -> 30 rows, ids 11-40 (spanned
the hollow commit, no error, no blocking)
* producer writes batch 5 (ids 41-50)
* **Run B** from commit 4's completion time -> 10 rows, ids 41-50
* Across ids 11-50: **`missing=[]`, `duplicated=[]`**
Table version 8 also puts the completion time directly in the instant
filename as `<requestedTime>_<completionTime>.deltacommit`, so no timeline API
call is needed to maintain the checkpoint. **Upgrading to 1.x is the cleaner
fix** — it removes the failure mode instead of working around it, and
`USE_TRANSITION_TIME` becomes unnecessary.
## Caveats on our reproduction
* The hollow commit was created by writing the `.deltacommit.requested` file
directly. A fabricated instant carries no heartbeat, so Hudi eventually
reclaims it: a continuity attempt failed with `HoodieRollbackException: Found
commits after time ..., please rollback greater commits first` under the
default EAGER failed-write cleanup, and even with
`hoodie.cleaner.policy.failed.writes=LAZY` the instant was gone by the
following write. A genuinely running producer holds a heartbeat and stays
in-flight. This affects how long the simulated hollow state survives, not the
read results above, all of which were measured while it was present.
* Local filesystem, single `local[2]` Spark — the cloud-storage rate
limiting you see in production is not exercised here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
