nsivabalan commented on code in PR #20194: URL: https://github.com/apache/hudi/pull/20194#discussion_r4280146909
########## hudi-ai-operator/skills/hudi-rca/references/mitigation-ladders.md: ########## @@ -0,0 +1,311 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> +# Mitigation ladders + +Thirteen operational procedures, each ordered. They are not tied to one failure signature — they are +the discipline that makes the difference between diagnosing a Hudi failure and making it worse. +`failure-catalog.md` rows reference them by number. + +Several of these say **do nothing**, **do not restart**, or **turn the number down**. Those steps are +not hedges. They are the steps most often got wrong, and the reason is usually that the obvious lever +points the other way. + +**The single most expensive mistake available is restarting before capturing evidence.** That is L9, +and it earns its place by frequency. + +--- + +## L1 — Locate the real exception in a multi-table writer + +When one driver writes many tables, the trace you find first is often not the trace you need. + +1. Search driver logs for `ERROR`, then search **within those results** for the table name. Keep + scrolling — one polling cycle can contain several tables' traces interleaved. +2. **Save the full trace somewhere durable the moment you find it.** Re-finding it is expensive, and + log rotation does not wait. +3. If you cannot attribute the failure to a table, temporarily move that table to a **single-table** + writer. One table per job makes the trace unambiguous. Move it back afterwards. +4. If the first pass (ERROR only) returns nothing, widen to INFO and WARN. Some stalls produce **no + ERROR line at all** — see L8 and HUDI-RCA-038. +5. Only then start root-causing. + +--- + +## L2 — Classify a Spark OOM before reacting + +An OOM is not one failure mode. These six have genuinely different remedies, and reaching for heap +first is right in only one of them. + +1. **Container OOM-kill (exit 137), Java heap fine** → native or off-heap memory, not heap. Raise + `spark.executor.memoryOverheadFactor` (commonly to 0.35, up to around 0.5). Suspect compression + codecs and shuffle transfer. +2. **JVM heap OOM (exit 52)** → a heap dump is usually written automatically. **Get it.** This is a + code or data-shape problem, not a sizing problem: memory buys time, not a fix. +3. **`OutOfDirectMemoryError` / `Direct buffer memory`** → raise overhead and direct-buffer memory. As + an emergency measure only, stop Spark preferring direct buffers for shuffle transfer, at a + throughput cost. **Do not leave that on.** +4. **Evicted mid-stage with `FetchFailedException` on shuffle files** → **this is not memory at all.** + It is **ephemeral local storage**. Evicted executors delete their shuffle files, surviving tasks + fail to fetch, Spark retries the whole stage, writes more shuffle, and evicts more executors. Raise + the executors' local-disk request and limit. **This does not show up in the Spark UI** — check pod + and node events instead. Treating a FetchFailed as a memory problem is a wasted tuning cycle, and + it is the most commonly misread entry in this list. +5. **Driver OOM requesting a large array** → the JVM cannot allocate a single contiguous array above + roughly 2 GB. This is a **structural ceiling**; more heap will not help. See HUDI-RCA-021. +6. **A few partitions consistently OOM while others are fine** → skew, or large rows. Check per-record + size in the stage detail. Above roughly 1 MB average, lower + `spark.shuffle.spill.numElementsForceSpillThreshold`. +7. **GC pressure rather than hard OOM** → **reduce** `spark.memory.fraction` and + `spark.memory.storageFraction`. Counter-intuitive: giving Spark's managed regions *less* leaves + more heap for user objects. Confirm G1GC and enable GC logging before tuning further. +8. Check for a pathological CPU-to-memory ratio. Powerful CPUs paired with small memory cause this + repeatedly, and no Hudi config will fix it. + +--- + +## L3 — Safe-deletion protocol + +**Never delete a file from a Hudi table outside this envelope.** Nine steps, in order, no exceptions. + +1. Inform stakeholders. +2. **Pause the writer.** +3. **Take a savepoint.** +4. Verify the deletion premise with **two independent checks** — for example, the file is untracked in + commit metadata **and** its instant is named by a completed rollback. One check is not enough. +5. **Copy the files you intend to delete to a backup prefix.** +6. Delete. +7. Validate: re-run the failing operation, or the validator. +8. Resume the writer. +9. After a few successful commits, pause, **delete the savepoint**, resume. + +**If any verification is ambiguous, stop.** Deleting a non-orphan corrupts the table silently — there +is no error, and you will find out later from query results. The ambiguity is the signal, not an +obstacle to work around. + +A human drives this procedure. An agent may explain it, point at the evidence that would satisfy +step 4, and say which files it believes are implicated — but **an agent must not perform, script, or +sequence the deletion.** See the explain-only block on HUDI-RCA-025. Review Comment: Both fixed in `576f14a4a46d`. L3 now matches the stricter rule: an agent may explain the procedure and point at the evidence that would satisfy step 4, but must not enumerate the files — a file list is the deletion one copy-paste removed, and the point of step 4 is that the human derives it from two independent verifications. The text says so, and references RCA-025 as the rule it must not contradict. RCA-020 step 3 has a gate now. It says a savepoint is the only thing standing between its file slices and the cleaner, that releasing it is durable and unrecoverable, and that confirming with whoever took it matters because "a savepoint nobody remembers taking is not the same as one nobody needs". On `hoodie.archive.beyond.savepoint` as the safer option — I did not take that one. The feature is not fully supported, archival will not move past the first savepoint regardless of the setting, and `restore` is refused outright while it is on (`BaseHoodieWriteClient.java:883`). Recommending it would trade one bad instruction for another. The ladder now says this explicitly so the next reader does not reach for it, and I removed it from that pattern's governing configs. The remaining mention in `workflow-catalog.md` is descriptive — it explains why archival stops where it does — and now carries the same caveat. ########## hudi-ai-operator/skills/hudi-rca/references/failure-catalog.md: ########## @@ -0,0 +1,1576 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> +# Hudi failure catalog + +38 failure patterns, each with the signature that identifies it, the mechanism behind it, and an +ordered mitigation ladder. Used by the `hudi-rca` skill to classify a pasted failure; readable on its +own as a troubleshooting reference. + +## How to read a row + +| Field | Meaning | +|---|---| +| **ID** | Stable identifier. Never renumber; retire instead. | +| **Category** | `input/config` · `sizing/scale` · `environment` · `defect` · `not-a-failure` | +| **Discoverability** | `self-evident` (the message says what to do) · `needs-expertise` (the message is generic, misleading, or points at the wrong layer) · `silent` (no exception at all) | +| **Signature** | Exception classes, message regexes, and the Hudi frames that identify the pattern. | +| **Symptom** | What the operator sees, in their terms. | +| **Root cause** | The mechanism, in one or two sentences. | +| **Evidence required** | Which rungs of the evidence ladder are needed to reach a confident call. | +| **Disambiguation** | Present only where a signature maps to more than one pattern. The explicit discriminator. | +| **Mitigation ladder** | Ordered, cheapest and most likely first. Counter-intuitive steps are marked and justified. | +| **Governing configs** | Exact `hoodie.*` / `spark.*` keys. | +| **`fixedInVersion`** | The Hudi release that fixes the defect, where one exists. | +| **`preflightGuardExists`** | Whether a current Hudi release fails fast on the condition instead of corrupting silently. | +| **`alsoAppliesWhenHealthy`** | `true` when the same advice is an optimization on a merely-slow job, not only a fix for a broken one. | +| **Provenance** | Public `apache/hudi` issue numbers. | + +## Three rules that govern every row + +**1. Match the innermost `Caused by`, never the wrapper.** In the public tracker the wrapper classes +dominate the counts — bare `HoodieException` 498 occurrences, `HoodieUpsertException` 189 — while the +classes that actually identify the fault barely appear: `SchemaCompatabilityException` 1, +`HoodieAvroSchemaException` 2, `HoodieKeyGeneratorException` 3. Users paste the wrapper. Every regex +below is written against the innermost cause plus the first Hudi frame, because a classifier keyed on +the top-level exception class will mostly see wrappers and learn nothing. + +**2. The stacktrace is the weakest signal, not the backbone.** For a large share of these the terminal +exception is generic and the real cause is table state or config sizing several steps removed. A +`413 Payload Too Large` on timeline init is really a forgotten savepoint blocking archival. A stalled +ingestion is really a rollback that never finished and still holds the lock. Where a row says the +evidence requires table state, the honest move is to run `HoodieTableHealthChecker` rather than parse +the trace harder. + +**3. Ask the Hudi version early.** The heavy issue clusters are 0.9 through 0.15, and several +high-traffic patterns here are fixed outright in 1.0 or 1.2. For this population "upgrade" is a +genuinely good first-line remedy in a way it usually is not. Rows carry `fixedInVersion` where known. + +## Disambiguation index + +Four signatures map to more than one pattern. Resolve them here before attributing. + +| Signature | Candidates | Discriminator | +|---|---|---| +| `ConcurrentModificationException` / `HoodieWriteConflictException` | HUDI-RCA-006 healthy OCC retry **vs** HUDI-RCA-006 no lock provider | **Writer config, not the trace** — the stacktrace is identical. If `hoodie.write.lock.provider` is set and two writers are intended, this is a by-design rejection: retry the application attempt. If no lock provider is set and two writers exist, the table is being corrupted silently and the conflict exception is the lucky case. | +| `HoodieRollbackException` | HUDI-RCA-027 rollback wedge · HUDI-RCA-026 metadata-table divergence · HUDI-RCA-025 dangling files | **The frame below `rollbackFailedWrites`.** `AsyncCleanerService.waitForCompletion` → cleaner/metadata-table path (026). A lock-provider frame (`LockManager.lock`, `*LockProvider.acquireLock`) → lock problem first (006). `HoodieMergeHandle.getLatestBaseFile` with `FileID ... does not exist` → 025. | +| `OutOfMemoryError` | HUDI-RCA-015 merge-side · HUDI-RCA-015 archival-side · HUDI-RCA-014 bloom index · HUDI-RCA-021 driver/timeline-server | **The Hudi frame.** `HoodieMergedLogRecordScanner` / `ExternalSpillableMap` → merge side (015a). `HoodieTimelineArchiveLog.writeToFile` / `HoodieAvroDataBlock.serializeRecords` → **archival side (015b), where adding executor memory is the wrong answer entirely** — the fix is to shrink commit metadata. Kryo / `ExternalSorter` frames under a shuffle read with no Hudi frame → bloom index (014). `Requested array size exceeds VM limit` in a driver Jetty thread → 021, a structural ceiling that more heap cannot move. | +| `FileNotFoundException` | HUDI-RCA-031 markers · HUDI-RCA-033 archiver race · HUDI-RCA-025 stale plan · HUDI-RCA-036 source file | **The path.** `MARKERS*` or `.hoodie/.temp/` → 031. A `.commit` / `.deltacommit` under `.hoodie/` → 033. A data or log file named in a compaction plan → 025. A path under the **source** bucket, not under the table → 036. A path under the table ending `.log.<n>` → treat as a compaction race, see 025's note. | + +## Cross-cutting trap: MOR defers the failure + +On Merge-on-Read an incompatible schema write **succeeds**, and compaction fails hours later in a +*different job* whose stacktrace names `HoodieCompactor` rather than the write. The table then cannot +compact and every later write fails. Cause and symptom are separated by both time and process. + +**Rule: a compaction frame plus a schema exception means the bad write happened earlier, in another +job. Investigate the writes from that window, not the job that just failed.** This applies to +HUDI-RCA-001, HUDI-RCA-003 and HUDI-RCA-004. + +--- + +# Group A — input/config + +The user asked Hudi to do something invalid, or configured it inconsistently. The remedy is to fix +data or config; nothing is wrong with Hudi or the cluster. + +--- + +## HUDI-RCA-001 — Schema evolution the writer rejects + +- **Category:** `input/config` · **Discoverability:** `needs-expertise` +- **Signature** (innermost cause): + - `org\.apache\.avro\.AvroTypeException: Cannot encode decimal with precision (\d+) as max precision (\d+)` + - `org\.apache\.parquet\.io\.InvalidRecordException: Parquet/Avro schema mismatch: Avro field '([^']+)' not found` + - Hudi frames: `HoodieAvroUtils.rewriteRecordWithNewSchema`, `HoodieAvroUtils.rewritePrimaryTypeWithDiffSchemaType`, `HoodieMergeHelper.runMerge`, `AbstractHoodieLogRecordReader.scanInternal`, `HoodieCompactor.compact` +- **Symptom:** A write that worked yesterday fails after an upstream column was retyped or a decimal + widened. On MOR the write succeeds and **compaction** fails hours later, permanently, for every + subsequent write. +- **Root cause:** The incoming Avro schema is not a backward-compatible evolution of the table schema + under Hudi's rules. Full schema-on-read evolution is off by default, and decimal precision widening + was rejected outright until the `SchemaChangeUtils.isTypeUpdateAllow` relaxation. +- **Evidence required:** stacktrace · writer props · `hoodie.properties` (table schema) · for the MOR + variant, table state to show which file slices span the change. +- **Disambiguation:** a compaction frame here means the offending write already committed — see the + MOR deferred-failure rule above. Distinguish from HUDI-RCA-003 by *direction*: a field the table has + and the batch lacks is 003; a field whose **type** changed is 001. +- **Mitigation ladder:** + 1. Upgrade if the signature is decimal precision — fixed in 1.0.x, and no config works around it on + 0.14/0.15. + 2. Set `hoodie.datasource.write.reconcile.schema=true`, with `hoodie.schema.on.read.enable=true`. + 3. Stop dropping or retyping the column in the writer: keep it and write typed nulls. + 4. If MOR and already wedged, the table cannot compact until the pending compaction is unscheduled + or the affected file groups are rewritten. Do not hand-delete the pending instant files. +- **Governing configs:** `hoodie.datasource.write.reconcile.schema`, `hoodie.schema.on.read.enable`, + `hoodie.avro.schema.validate`, `hoodie.datasource.write.schema.allow.auto.evolution.column.drop` +- **`fixedInVersion`:** 1.0.0 for decimal precision widening · **`preflightGuardExists`:** no +- **`alsoAppliesWhenHealthy`:** false +- **Provenance:** #8040, #5452, #11335, #8020, #2919 + +--- + +## HUDI-RCA-002 — Record key or precombine field missing, null, or wrong case + +- **Category:** `input/config` · **Discoverability:** `self-evident` for the key case, `needs-expertise` for precombine +- **Signature** (innermost cause): + - `org\.apache\.hudi\.exception\.HoodieKeyException: recordKey value: "null" for field: "([^"]+)" cannot be null or empty` + - `org\.apache\.hudi\.exception\.HoodieKeyException: Record key has to be non-null!` + - `org\.apache\.hudi\.exception\.HoodieException: \w+\(Part -\w+\) field not found in record\. Acceptable fields were :\[` + - Hudi frames: `HoodieAvroUtils.getNestedFieldVal`, `ComplexKeyGenerator`, often wrapped by + `SparkException: Failed to execute user defined function` so the top frame is Spark, not Hudi. +- **Symptom:** The write fails on the first batch, or only when one composite key field is null. +- **Root cause:** Three sub-cases sharing a remedy family: the configured field does not exist or is + null in some rows; the key config is **case-sensitive** and the user wrote `ID` for a column named + `id`; or HoodieStreamer defaulted `--source-ordering-field` to `ts`, producing a phantom `ts` + requirement that then persisted into `hoodie.properties`. +- **Evidence required:** stacktrace · writer props · `hoodie.properties` +- **Disambiguation:** if the named field is one the user never configured (classically `ts`), this is + the HoodieStreamer default, not a missing column. The message names a field the user has never heard + of, which is the whole confusion — say so explicitly rather than asking them to add a `ts` column. +- **Mitigation ladder:** + 1. Check the exact field name **and case** against the dataframe schema. Hudi does not normalize case. + 2. Filter or default nulls in the key columns before writing. + 3. For HoodieStreamer with no ordering field: set `--source-ordering-field` to a real column, or use + `--op INSERT` / `bulk_insert`, which skip precombine entirely. + 4. Multi-field keys with `bulk_insert` had a genuine bug before 0.13.1 — upgrade, or switch to + `upsert` to confirm. +- **Governing configs:** `hoodie.datasource.write.recordkey.field`, + `hoodie.datasource.write.precombine.field`, `hoodie.datasource.write.keygenerator.class`, + `hoodie.table.precombine.field`, `--source-ordering-field` +- **`fixedInVersion`:** 0.13.1 for the bulk_insert multi-field key bug · **`preflightGuardExists`:** partial — the key check fires at write time, not at config time +- **`alsoAppliesWhenHealthy`:** false +- **Provenance:** #3759, #9799, #11776, #10233 + +--- + +## HUDI-RCA-003 — Source drops a column the table still has + +- **Category:** `input/config` · **Discoverability:** `needs-expertise` +- **Signature** (innermost cause): + - `org\.apache\.parquet\.io\.InvalidRecordException: Parquet/Avro schema mismatch: Avro field '([^']+)' not found` + - Frames, in order: `AvroRecordConverter.getAvroField` → `ParquetReaderIterator.hasNext` → + `HoodieMergeHelper.runMerge` + - Wrapped by `HoodieUpsertException: Error upserting bucketType UPDATE for partition :\d+` +- **Symptom:** Every sync fails after an upstream schema change. The exception arrives on the **merge** + path while *reading existing Parquet*, so operators look at their table instead of their batch. +- **Root cause:** The incoming batch no longer carries a column the table's base files have. Hudi reads + existing records using the latest (incoming) schema, and the Parquet-to-Avro converter cannot find + the field. +- **Evidence required:** stacktrace · writer props · last commit metadata (`extraMetadata.schema`) for + the field's declared type +- **Disambiguation:** same exception class as HUDI-RCA-001's field-not-found variant. The + `HoodieMergeHelper.runMerge` frame and the fact that the field exists in the **table** but not the + **batch** is what makes it 003. Recover the field's declared type including its Avro `logicalType` + before re-adding it, or the re-add fails differently. +- **Mitigation ladder:** + 1. Re-add the column to the incoming batch as a **typed** null via a transformer — `cast(null as + string)`, or `cast(null as date)` where the logical type demands it. Cheapest and reversible. + 2. Set `hoodie.avro.schema.validate=false` only as a stopgap, not as the fix. + 3. **Config-layering trap:** if your config system overrides rather than merges list-valued settings, + adding a transformer silently drops the transformer that was already configured. Re-list all of + them, and verify the resolved config rather than the file you edited. +- **Governing configs:** `hoodie.avro.schema.validate`, + `hoodie.datasource.write.schema.allow.auto.evolution.column.drop`, + `hoodie.datasource.write.reconcile.schema` +- **`fixedInVersion`:** not a defect · **`preflightGuardExists`:** no +- **`alsoAppliesWhenHealthy`:** false +- **Provenance:** #8040 + +--- + +## HUDI-RCA-004 — Unvalidated source schema: records do not parse against the inferred type + +- **Category:** `input/config` · **Discoverability:** `self-evident` for the message, `needs-expertise` for the fix +- **Signature** (innermost cause): + - `org\.apache\.spark\.sql\.catalyst\.util\.BadRecordException` + - `java\.lang\.RuntimeException: Failed to parse a value for data type struct<.*> \(current token: \w+\)` +- **Symptom:** Records fail to parse and the whole job fails, rather than the bad rows being isolated. +- **Root cause:** A semi-structured source (CDC, JSON, document-store feed) emits a field whose actual + shape differs from what was inferred from earlier records — most often an array where a struct was + inferred. +- **Evidence required:** stacktrace · writer props (schema-provider config) · a raw source sample +- **Mitigation ladder:** + 1. Set an explicit **target schema** instead of relying on inference, and enable validation with a + quarantine destination so bad records are isolated rather than failing the commit. + 2. For CDC sources, make sure the target schema declares the envelope fields the transformer adds + (operation type, transaction id, LSN, event timestamps) — otherwise validation rejects every record. + 3. Loosen validation only as a last resort. It converts a loud failure into a silent one. +- **Governing configs:** `hoodie.streamer.schemaprovider.target.schema.file`, + `hoodie.streamer.schemaprovider.source.schema.file`, `hoodie.avro.schema.validate` +- **`fixedInVersion`:** not a defect · **`preflightGuardExists`:** no +- **`alsoAppliesWhenHealthy`:** false +- **Provenance:** none public — the mechanism is config hygiene and rarely filed + +--- + +## HUDI-RCA-005 — Timestamp key generator cannot parse the partition field + +- **Category:** `input/config` · **Discoverability:** `self-evident` +- **Signature** (innermost cause): + - `org\.apache\.hudi\.exception\.HoodieKeyGeneratorException: Unable to parse input partition field: (.+)` + - `java\.lang\.IllegalArgumentException: Invalid format: "(.+)" is malformed at "(.+)"` + - Hudi frame: `TimestampBasedAvroKeyGenerator.getPartitionPath` +- **Symptom:** Every write fails immediately, naming the exact offending value. +- **Root cause:** The configured **input** date format does not match the actual partition-field format. + Typical triggers: microsecond precision, a trailing `Z`, or a switch from epoch to ISO-8601 upstream. +- **Evidence required:** stacktrace alone suffices · writer props to confirm the configured format +- **Mitigation ladder:** + 1. Update the key generator's **input** format config to match the value printed in the exception. + 2. Verify the **output** partition format is unchanged. Changing the output format alters partition + paths and effectively forks the table — a far worse outcome than the parse failure. + 3. If the source format is now inconsistent across records, normalize it in a transformer before the + key generator sees it. +- **Governing configs:** `hoodie.keygen.timebased.input.dateformat`, + `hoodie.keygen.timebased.output.dateformat`, `hoodie.keygen.timebased.timestamp.type`, + `hoodie.keygen.timebased.input.timezone`, `hoodie.keygen.timebased.output.timezone` +- **`fixedInVersion`:** not a defect · **`preflightGuardExists`:** no +- **`alsoAppliesWhenHealthy`:** false +- **Provenance:** none public with this exact signature; included because it is trivially fixable and + almost certainly under-reported + +--- + +## HUDI-RCA-006 — Multi-writer without, or with a misconfigured, lock provider + +- **Category:** `input/config` · **Discoverability:** `needs-expertise`, and **`silent`** in the no-lock-at-all case +- **Signature** (innermost cause): + - `java\.util\.ConcurrentModificationException: Cannot resolve conflicts for overlapping writes`, + wrapped in `HoodieWriteConflictException`; frames + `SimpleConcurrentFileWritesConflictResolutionStrategy.resolveConflict`, + `TransactionUtils.resolveWriteConflictIfAny`, `SparkRDDWriteClient.preCommit` + - `org\.apache\.hudi\.exception\.HoodieLockException: Unable to acquire lock, lock object (null|LockResponse\(lockid:\d+, state:WAITING\))`; + frames `LockManager.lock`, `TransactionManager.beginTransaction` + - `java\.lang\.IllegalArgumentException: ALREADY_ACQUIRED` at `HiveMetastoreBasedLockProvider.acquireLock` + - `org\.apache\.hadoop\.fs\.FileAlreadyExistsException: File already exists: .*\.hoodie/\d+\.commit\.requested` + at `HoodieActiveTimeline.createImmutableFileInPath` +- **Symptom:** Intermittent failures that scale with writer count. With **no** lock provider there may + be no exception at all — just overwritten timeline state and lost data. +- **Root cause:** Two writers against one table without OCC properly configured, or OCC configured and + genuinely conflicting. The semantics users most often get wrong: **the OCC lock is table-level, not + partition-level**, and it serializes only the commit-validation phase. Writers on disjoint partitions + still serialize on the lock. +- **Evidence required:** stacktrace · **writer props — mandatory** · `hoodie.properties` +- **Disambiguation (the most important row in this catalog):** the conflict exception and the no-lock + corruption produce the **same stacktrace**. Only the writer config splits them. + - `hoodie.write.lock.provider` set, concurrency intended → by-design rejection of the later committer. + Expected. Add orchestrator-level retry of the whole application attempt. Severity: routine. + - No lock provider set, but two writers demonstrably exist → the table is being corrupted and this + exception is the *lucky* case where you found out. Severity: urgent. + - Treat "is a second writer intended?" as a question to ask the user, not something to infer. +- **Mitigation ladder:** + 1. If two writers exist and no lock provider is set, set one. Non-negotiable; absence is silent + corruption, not a performance choice. + 2. **Lower** `hoodie.write.lock.wait_time_ms` to around 5000. Counter-intuitive: a long wait converts + a fast, retryable rejection into a long stall that looks like a hang, and the later writer has to + retry regardless. + 3. For DynamoDB locks, set the endpoint URL explicitly and use on-demand billing; stale lock rows + occasionally need manual removal. + 4. Add retry of the whole application attempt at the orchestrator. The later committer is rejected by + design and must retry. + 5. Where lock serialization is itself the bottleneck on MOR, consider 1.x non-blocking concurrency + control, which pairs best with a bucket index. +- **Governing configs:** `hoodie.write.concurrency.mode`, `hoodie.write.lock.provider`, + `hoodie.write.lock.wait_time_ms`, `hoodie.write.lock.num_retries`, `hoodie.write.lock.dynamodb.*`, + `hoodie.write.lock.zookeeper.*`, `hoodie.cleaner.policy.failed.writes` +- **`fixedInVersion`:** n/a — NBCC added in 1.x as an alternative mode, not a fix +- **`preflightGuardExists`:** no — Hudi does not detect "two writers, no lock" at write time +- **`alsoAppliesWhenHealthy`:** true — lock wait tuning is an optimization on a merely-slow multi-writer setup +- **Provenance:** #7653, #3731, #4456, #6226, #9213 + +--- + +## HUDI-RCA-007 — Hive or Glue catalog sync failure: table written but invisible + +- **Category:** `input/config` · **Discoverability:** `needs-expertise` +- **Signature** (innermost cause): + - `java\.lang\.RuntimeException: Unable to instantiate org\.apache\.hadoop\.hive\.ql\.metadata\.SessionHiveMetaStoreClient`; + frames `HMSDDLExecutor.<init>`, `HiveSyncTool.initClient` + - `NoSuchObjectException: .* table not found` + - Wrapped in `HoodieHiveSyncException: Got runtime exception when hive syncing` or + `Could not sync using the meta sync class` + - Log giveaway: `Trying to connect to metastore with URI thrift://localhost:9083` when the user + believes they configured Glue +- **Symptom:** Data lands on storage correctly but the query engine cannot see it, or the write fails + only in the post-commit sync step. +- **Root cause:** Sync mode and catalog wiring mismatch, Hive metastore version incompatibility, or + table-name case sensitivity. The `localhost:9083` line is the tell that the intended catalog config + never took effect. +- **Evidence required:** stacktrace · writer props · driver log (for the metastore URI line) +- **Mitigation ladder:** + 1. Lowercase the table and database names everywhere. An uppercase table name causes a rename failure. + 2. Set `hoodie.datasource.hive_sync.mode` explicitly — `hms` for Glue and HMS. + 3. For Glue, set the AWS Glue sync tool class explicitly rather than relying on inference. + 4. Verify the Hive metastore version is 2.3 or newer for Spark 3 bundles. + 5. Confirm `hoodie.datasource.hive_sync.partition_extractor_class` matches the actual partitioning — + the non-partitioned extractor for unpartitioned tables, the multi-part extractor for multi-level. +- **Governing configs:** `hoodie.datasource.hive_sync.enable`, `hoodie.datasource.hive_sync.mode`, + `hoodie.datasource.hive_sync.database`, `hoodie.datasource.hive_sync.table`, + `hoodie.datasource.hive_sync.partition_extractor_class`, `hoodie.datasource.hive_sync.partition_fields`, + `hoodie.meta.sync.client.tool.class`, `hoodie.datasource.write.hive_style_partitioning` +- **`fixedInVersion`:** 0.11.1 for the Glue client-factory regression · **`preflightGuardExists`:** no +- **`alsoAppliesWhenHealthy`:** false +- **Provenance:** #5484, #8368, #2409, #954 + +--- + +## HUDI-RCA-008 — Config changed between runs against an existing table + +- **Category:** `input/config` · **Discoverability:** `self-evident` when the guard fires, **`silent`** when it does not +- **Signature:** + - `HoodieException: Config conflict \(key, current value, existing value\): ` — when Hudi's guard + catches it + - When the guard does not catch it: **no exception**. Duplicates, no-op deletes, or an unqueryable + table. +- **Symptom:** A pipeline that ran for months starts failing, or silently producing duplicates, after + someone edited the writer config or upgraded Hudi. +- **Root cause:** Several table-level properties are written into `hoodie.properties` on first write and + are effectively **immutable**: record key field, partition path field, key generator class, table + type, precombine field, hive-style partitioning. Changing them later either trips the conflict guard + or silently reinterprets existing records. +- **Evidence required:** `hoodie.properties` · writer props. **This pattern is definitionally a diff + between the two** — no stacktrace substitutes for it, and this is why the skill requests + `hoodie.properties` by default. +- **Disambiguation:** overlaps HUDI-RCA-013 (silent duplicates). If the properties diff shows a changed + key generator or record-key field, the duplicates are 008's mechanism and 013's symptom. Report 008 — + it names the cause. +- **Mitigation ladder:** + 1. Read `.hoodie/hoodie.properties` and diff it against the current writer config. This is step one of + nearly any Hudi triage, not just this pattern. + 2. Revert the changed property to match the table. + 3. If the change is genuinely required, rewrite the table into a new path with `bulk_insert` rather + than fighting the guard. Editing `hoodie.properties` by hand is not a migration. + 4. For upgrade-induced key-encoding changes, pin the old version or rewrite. A single-field record key + under `ComplexKeyGenerator` was encoded differently from 0.14.1 onward, which silently created + duplicates and silently no-op'd deletes. +- **Governing configs:** `hoodie.table.recordkey.fields`, `hoodie.table.partition.fields`, + `hoodie.table.keygenerator.class`, `hoodie.table.type`, `hoodie.table.precombine.field`, + `hoodie.table.version`, `hoodie.datasource.write.hive_style_partitioning` +- **`fixedInVersion`:** no — the 0.14.1 key-encoding change shipped without a migration path +- **`preflightGuardExists`:** partial — the conflict guard covers some keys, not all +- **`alsoAppliesWhenHealthy`:** false +- **Provenance:** #10508, #10587, #10233 + +--- + +## HUDI-RCA-009 — Wrong query type or engine wiring makes data look missing or stale + +- **Category:** `input/config` · **Discoverability:** **`silent`** +- **Signature:** **No exception.** Recognised by symptom, not by trace: an `INSERT OVERWRITE` wrote + files but `SELECT` returns old data; only a `_ro` table exists where the user expected `_rt`; + timestamps render as implausible far-future dates in the query engine. +- **Symptom:** Data is demonstrably on storage but the query returns stale, empty, or garbled results. +- **Root cause:** Query-side wiring, not write-side. Reading the read-optimized (`_ro`) view of a MOR + table and seeing pre-compaction state; missing `spark.sql.hive.convertMetastoreParquet=false`, so + Spark uses its own Parquet reader and bypasses Hudi's file-slice resolution; expecting two Hive tables + for a Copy-on-Write table, which only ever has one; or a timestamp type the catalog declares + differently from how it was written. +- **Evidence required:** `hoodie.properties` (table type) · the query and the engine · writer props +- **Proactive detection:** compare `COUNT(*)` on the snapshot view against the `_ro` view. A gap that + closes after compaction is this pattern, by design, not data loss. +- **Mitigation ladder:** + 1. Set `spark.sql.hive.convertMetastoreParquet=false`. Note that whitespace around `=` in a SQL `set` + has been enough to make this silently not apply — verify it took effect. + 2. Query the snapshot view, not `_ro`; set `hoodie.datasource.query.type=snapshot`. + 3. For MOR, confirm compaction has actually run if you need read-optimized freshness. + 4. For timestamp rendering, enable `hoodie.datasource.hive_sync.support_timestamp`. +- **Governing configs:** `hoodie.datasource.query.type`, + `hoodie.datasource.hive_sync.support_timestamp`, `spark.sql.hive.convertMetastoreParquet`, + `hoodie.datasource.write.drop.partition.columns` +- **`fixedInVersion`:** not a defect · **`preflightGuardExists`:** no +- **`alsoAppliesWhenHealthy`:** false +- **Provenance:** #4154, #2409, #2123, #2338 + +--- + +## HUDI-RCA-010 — HoodieStreamer source or schema-provider misconfiguration + +- **Category:** `input/config` · **Discoverability:** `needs-expertise` +- **Signature:** + - `org\.apache\.hudi\.exception\.HoodieException: Commit \d+ failed and rolled-back !` at + `DeltaSync.writeToSink` / `StreamSync.writeToSink` — **this message carries no diagnostic content + whatsoever** + - `java\.lang\.NullPointerException` inside `AvroDeserializer` with no Hudi frame + - Structurally wrong data: a whole CDC envelope ingested instead of the unwrapped row +- **Symptom:** The Streamer fails with a generic "commit failed and rolled-back", or ingests + structurally wrong data. +- **Root cause:** Schema provider mismatched to source. A provider returning a null target schema while + a transformer is configured; a generic Avro source used where a CDC-specific source and payload class + are required to unwrap the envelope; Confluent wire-format Avro needing registry-aware deserialization. +- **Evidence required:** stacktrace · **Streamer CLI args** · driver log — the real exception is in the + driver log *below* the "failed and rolled-back" line, not in the thrown one +- **Disambiguation:** `Commit ... failed and rolled-back` on its own is a **content-free wrapper**. Do + not attribute from it. Ask for the driver log and classify from the exception beneath it. If the user + can only supply this line, say so and request the driver log — that is the correct outcome, not a guess. +- **Mitigation ladder:** + 1. Read the driver log below the "failed and rolled-back" line and classify the real exception. + 2. For CDC sources, use the database-specific source class with its matching payload class, not a + generic Avro source. + 3. For Confluent Avro, let the Kafka Avro source fetch from the registry rather than hand-rolling a + schema provider. + 4. If using a transformer, do not return null from the target schema provider. Set a real schema, or + omit the provider so Hudi infers from the dataframe. +- **Governing configs:** `--source-class`, `--schemaprovider-class`, `--payload-class`, + `--transformer-class`, `--source-ordering-field`, `hoodie.streamer.schemaprovider.registry.url`, + `hoodie.streamer.source.kafka.topic` +- **`fixedInVersion`:** no · **`preflightGuardExists`:** no — and the uninformative message is itself + worth an upstream report +- **`alsoAppliesWhenHealthy`:** false +- **Provenance:** #2515, #2149, #2589, #8519, #10233, #5540 + +--- + +## HUDI-RCA-011 — Source initialization fails: credentials or schema registry + +- **Category:** `input/config` · **Discoverability:** `self-evident` for credentials, `needs-expertise` for the registry case +- **Signature:** Failures during writer startup, **before any batch is read**: + - `org\.apache\.kafka\.common\.errors\.\w+Exception`, SSL or keystore load failures, + connection-refused against brokers + - `org\.apache\.hudi\.utilities\.exception\.HoodieSchemaProviderException` +- **Symptom:** The job never ingests anything. No data lands, so there is no partial state to reason + about — which is itself the diagnostic. +- **Root cause:** Source credentials or TLS material are wrong or unreadable, or the source requires a + schema registry that was never configured. A common origin: a stream configuration **cloned** from an + older one created before registry support existed, carrying an empty registry reference that only + fails at writer init. +- **Evidence required:** stacktrace · writer props **as resolved, not as written** · the age of the + failure signal +- **Disambiguation:** distinguish an **init** failure from a **sync** failure. An init-failure counter + that never decrements can reflect a long-past startup failure on a driver that has since recovered. + Confirm the failure is recent before investigating it. +- **Mitigation ladder:** + 1. Fix the credential or registry configuration and restart. + 2. **Do not create a stream by cloning another** when the source type has gained required fields since + the original was created. Recreate from scratch — the clone silently carries empty required fields. + 3. Validate required source fields at creation time rather than discovering them at writer init. +- **Governing configs:** `hoodie.streamer.schemaprovider.registry.url`, + `hoodie.streamer.source.kafka.topic`, Kafka client security properties passed through + `hoodie.streamer.kafka.*` +- **`fixedInVersion`:** not a defect · **`preflightGuardExists`:** no +- **`alsoAppliesWhenHealthy`:** false +- **Provenance:** none public — onboarding-time failures are rarely filed + +--- + +## HUDI-RCA-012 — Checkpoint ahead of the source: offsets aged out or topic recreated + +- **Category:** `input/config` (persistent case) / `environment` (transient case) · **Discoverability:** **`silent`** in its most common form +- **Signature:** + - `org\.apache\.kafka\.clients\.consumer\.OffsetOutOfRangeException: Fetch position \S+ is out of range for partition` + - `Some data may have been lost because they are not available in Kafka any more` + - Or **no exception at all** — a healthy-looking job ingesting nothing +- **Symptom:** The stream looks perfectly healthy and ingests nothing. +- **Root cause — two causes with opposite correct responses:** + - **Persistent:** the topic was deleted and recreated, or retention aged past the committed + checkpoint. The checkpoint is permanently beyond the high watermark and will never recover. + - **Transient:** the broker's latest offset briefly moved backward (unclean leader election, replica + truncation, rebalance). Self-heals on the next produce. +- **Evidence required:** writer checkpoint from commit metadata · source topic offsets and metadata · + stacktrace where present +- **Disambiguation:** compare the committed checkpoint against current end offsets, then **watch the + drift for one interval**. Transient regressions close within minutes; persistent ones do not move at + all. A topic creation timestamp later than your last successful commit means recreation. Retention + settings alone do not delete a topic — if the topic is gone, something deleted it. +- **Mitigation ladder:** + 1. **Transient: do nothing.** Resetting the checkpoint here loses data for no reason. This is the + counter-intuitive step and the most common mistake on this pattern. + 2. **Persistent:** reset the checkpoint to the earliest available offset or a deliberately chosen + position, accepting that the gap is unrecoverable. Write down what was skipped. + 3. Confirm with the source owner before resetting anything. + 4. Prevent: set retention comfortably above your worst-case ingestion outage, and never delete a topic + a live stream is checkpointed against. +- **Governing configs:** `hoodie.streamer.source.kafka.topic`, + `auto.offset.reset` (the native Kafka consumer property, passed through — not a `hoodie.*` key), + `hoodie.streamer.checkpoint.provider.path` +- **`fixedInVersion`:** not a defect · **`preflightGuardExists`:** no +- **`alsoAppliesWhenHealthy`:** false +- **Provenance:** none public with this signature + +--- + +## HUDI-RCA-013 — Wrong operation or index choice produces silent duplicates + +- **Category:** `input/config` · **Discoverability:** **`silent`** — the highest-value row in the catalog +- **Signature:** **No exception.** Found only by running + `SELECT key, COUNT(*) FROM t GROUP BY key HAVING COUNT(*) > 1`, or noticed as unexplained table growth. +- **Symptom:** A primary key appears more than once. Often accompanied by table size growing far faster + than ingested volume. +- **Root cause:** A ladder of distinct mechanisms the user cannot tell apart without help: + `hoodie.datasource.write.operation` set to `insert` or `bulk_insert`, which **do not deduplicate + against existing data by design**; a non-global index with a record that moved partitions, so the key + legitimately exists twice; a global index on MOR where read-optimized queries show duplicates until + compaction runs; querying through Hive without `convertMetastoreParquet=false`; or a key-generator + encoding change across versions. +- **Evidence required:** `hoodie.properties` · writer props · the dedup query result · table type +- **Proactive detection:** `SELECT COUNT(1), COUNT(DISTINCT <record_key>)` on the **snapshot** view. + Run it after any index or key-generator change. +- **Disambiguation:** before calling this a defect, establish which mechanism applies. Cross-partition + duplicates with a non-global index are **working as configured**. Duplicates on a MOR `_ro` view that + disappear after compaction are **transient by design**. Only duplicates on the snapshot view with + `upsert` and a correct index are a real fault — and then check HUDI-RCA-008 for a key-encoding change. +- **Mitigation ladder:** + 1. Confirm `hoodie.datasource.write.operation=upsert`. This alone explains most reports. + 2. Determine whether the duplicates are cross-partition. If so, move to a global index and set + `hoodie.bloom.index.update.partition.path=true`. + 3. Re-run the dedup check against the **snapshot** view, not `_ro`, with + `convertMetastoreParquet=false`. + 4. If MOR, force a compaction and re-check — read-optimized duplicates may be transient by design. + 5. If the table crossed 0.14.1 with a single-field record key under `ComplexKeyGenerator`, compare + `_hoodie_record_key` encoding before and after: this is the known encoding regression, and the + remedy is to pin the old version or rewrite the table. +- **Governing configs:** `hoodie.datasource.write.operation`, `hoodie.index.type`, + `hoodie.bloom.index.update.partition.path`, `hoodie.datasource.write.keygenerator.class`, + `hoodie.combine.before.insert`, `hoodie.datasource.query.type` +- **`fixedInVersion`:** no · **`preflightGuardExists`:** no +- **`alsoAppliesWhenHealthy`:** false +- **Provenance:** #2338, #2255, #5777, #10508 + +--- + +# Group B — sizing/scale + +The config was right at 1,000 file groups and wrong at 500,000. Nobody's bug: the remedy is to re-tune +and add a guardrail. This is the category where a generic LLM most often gives the exactly wrong +advice, because several ladders here run *downward*. + +--- + +## HUDI-RCA-014 — Bloom index OOM from low parallelism or key skew Review Comment: Fixed in `576f14a4a46d`, and the catalog from #20195 confirmed your point precisely: `hoodie.index.type` has **no declared default**, and the effective one is written per engine in `HoodieIndexConfig.Builder.build()` — `SIMPLE` on Spark, `INMEMORY` on Flink, `SIMPLE` on Java. The ladder now opens with step 0, confirm the index type, and says why it is necessary: the signature names no Hudi class, so the pattern matches on shape alone and tells you nothing about which index produced the OOM. It then branches. BLOOM/GLOBAL_BLOOM keeps the existing four rungs. SIMPLE/GLOBAL_SIMPLE gets its own: source partitioning then `hoodie.simple.index.parallelism` (or the global variant), then `hoodie.simple.index.use.caching` / `.input.storage.level` if the pressure is on the input side rather than the join, then the `ExternalSorter` spill threshold. I kept that branch honest rather than padding it to match — a simple index joins incoming keys against base files, there is no bucketing to retune, so it genuinely has fewer levers. The last rung says so and treats migration to RLI or bucket index as the real answer rather than a last resort, flagged as a planned migration since it rewrites the metadata table. INMEMORY/BUCKET/FLINK_STATE get an explicit "this pattern does not apply". -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
