nsivabalan commented on code in PR #20194: URL: https://github.com/apache/hudi/pull/20194#discussion_r4279985326
########## hudi-ai-operator/skills/hudi-rca/references/failure-catalog.md: ########## @@ -0,0 +1,1576 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> +# Hudi failure catalog + +38 failure patterns, each with the signature that identifies it, the mechanism behind it, and an +ordered mitigation ladder. Used by the `hudi-rca` skill to classify a pasted failure; readable on its +own as a troubleshooting reference. + +## How to read a row + +| Field | Meaning | +|---|---| +| **ID** | Stable identifier. Never renumber; retire instead. | +| **Category** | `input/config` · `sizing/scale` · `environment` · `defect` · `not-a-failure` | +| **Discoverability** | `self-evident` (the message says what to do) · `needs-expertise` (the message is generic, misleading, or points at the wrong layer) · `silent` (no exception at all) | +| **Signature** | Exception classes, message regexes, and the Hudi frames that identify the pattern. | +| **Symptom** | What the operator sees, in their terms. | +| **Root cause** | The mechanism, in one or two sentences. | +| **Evidence required** | Which rungs of the evidence ladder are needed to reach a confident call. | +| **Disambiguation** | Present only where a signature maps to more than one pattern. The explicit discriminator. | +| **Mitigation ladder** | Ordered, cheapest and most likely first. Counter-intuitive steps are marked and justified. | +| **Governing configs** | Exact `hoodie.*` / `spark.*` keys. | +| **`fixedInVersion`** | The Hudi release that fixes the defect, where one exists. | +| **`preflightGuardExists`** | Whether a current Hudi release fails fast on the condition instead of corrupting silently. | +| **`alsoAppliesWhenHealthy`** | `true` when the same advice is an optimization on a merely-slow job, not only a fix for a broken one. | +| **Provenance** | Public `apache/hudi` issue numbers. | + +## Three rules that govern every row + +**1. Match the innermost `Caused by`, never the wrapper.** In the public tracker the wrapper classes +dominate the counts — bare `HoodieException` 498 occurrences, `HoodieUpsertException` 189 — while the +classes that actually identify the fault barely appear: `SchemaCompatabilityException` 1, +`HoodieAvroSchemaException` 2, `HoodieKeyGeneratorException` 3. Users paste the wrapper. Every regex +below is written against the innermost cause plus the first Hudi frame, because a classifier keyed on +the top-level exception class will mostly see wrappers and learn nothing. + +**2. The stacktrace is the weakest signal, not the backbone.** For a large share of these the terminal +exception is generic and the real cause is table state or config sizing several steps removed. A +`413 Payload Too Large` on timeline init is really a forgotten savepoint blocking archival. A stalled +ingestion is really a rollback that never finished and still holds the lock. Where a row says the +evidence requires table state, the honest move is to run `HoodieTableHealthChecker` rather than parse +the trace harder. + +**3. Ask the Hudi version early.** The heavy issue clusters are 0.9 through 0.15, and several +high-traffic patterns here are fixed outright in 1.0 or 1.2. For this population "upgrade" is a +genuinely good first-line remedy in a way it usually is not. Rows carry `fixedInVersion` where known. + +## Disambiguation index + +Four signatures map to more than one pattern. Resolve them here before attributing. + +| Signature | Candidates | Discriminator | +|---|---|---| +| `ConcurrentModificationException` / `HoodieWriteConflictException` | HUDI-RCA-006 healthy OCC retry **vs** HUDI-RCA-006 no lock provider | **Writer config, not the trace** — the stacktrace is identical. If `hoodie.write.lock.provider` is set and two writers are intended, this is a by-design rejection: retry the application attempt. If no lock provider is set and two writers exist, the table is being corrupted silently and the conflict exception is the lucky case. | +| `HoodieRollbackException` | HUDI-RCA-027 rollback wedge · HUDI-RCA-026 metadata-table divergence · HUDI-RCA-025 dangling files | **The frame below `rollbackFailedWrites`.** `AsyncCleanerService.waitForCompletion` → cleaner/metadata-table path (026). A lock-provider frame (`LockManager.lock`, `*LockProvider.acquireLock`) → lock problem first (006). `HoodieMergeHandle.getLatestBaseFile` with `FileID ... does not exist` → 025. | +| `OutOfMemoryError` | HUDI-RCA-015 merge-side · HUDI-RCA-015 archival-side · HUDI-RCA-014 bloom index · HUDI-RCA-021 driver/timeline-server | **The Hudi frame.** `HoodieMergedLogRecordScanner` / `ExternalSpillableMap` → merge side (015a). `HoodieTimelineArchiveLog.writeToFile` / `HoodieAvroDataBlock.serializeRecords` → **archival side (015b), where adding executor memory is the wrong answer entirely** — the fix is to shrink commit metadata. Kryo / `ExternalSorter` frames under a shuffle read with no Hudi frame → bloom index (014). `Requested array size exceeds VM limit` in a driver Jetty thread → 021, a structural ceiling that more heap cannot move. | +| `FileNotFoundException` | HUDI-RCA-031 markers · HUDI-RCA-033 archiver race · HUDI-RCA-025 stale plan · HUDI-RCA-036 source file | **The path.** `MARKERS*` or `.hoodie/.temp/` → 031. A `.commit` / `.deltacommit` under `.hoodie/` → 033. A data or log file named in a compaction plan → 025. A path under the **source** bucket, not under the table → 036. A path under the table ending `.log.<n>` → treat as a compaction race, see 025's note. | + +## Cross-cutting trap: MOR defers the failure + +On Merge-on-Read an incompatible schema write **succeeds**, and compaction fails hours later in a +*different job* whose stacktrace names `HoodieCompactor` rather than the write. The table then cannot +compact and every later write fails. Cause and symptom are separated by both time and process. + +**Rule: a compaction frame plus a schema exception means the bad write happened earlier, in another +job. Investigate the writes from that window, not the job that just failed.** This applies to +HUDI-RCA-001, HUDI-RCA-003 and HUDI-RCA-004. + +--- + +# Group A — input/config + +The user asked Hudi to do something invalid, or configured it inconsistently. The remedy is to fix +data or config; nothing is wrong with Hudi or the cluster. + +--- + +## HUDI-RCA-001 — Schema evolution the writer rejects + +- **Category:** `input/config` · **Discoverability:** `needs-expertise` +- **Signature** (innermost cause): + - `org\.apache\.avro\.AvroTypeException: Cannot encode decimal with precision (\d+) as max precision (\d+)` + - `org\.apache\.parquet\.io\.InvalidRecordException: Parquet/Avro schema mismatch: Avro field '([^']+)' not found` + - Hudi frames: `HoodieAvroUtils.rewriteRecordWithNewSchema`, `HoodieAvroUtils.rewritePrimaryTypeWithDiffSchemaType`, `HoodieMergeHelper.runMerge`, `AbstractHoodieLogRecordReader.scanInternal`, `HoodieCompactor.compact` +- **Symptom:** A write that worked yesterday fails after an upstream column was retyped or a decimal + widened. On MOR the write succeeds and **compaction** fails hours later, permanently, for every + subsequent write. +- **Root cause:** The incoming Avro schema is not a backward-compatible evolution of the table schema + under Hudi's rules. Full schema-on-read evolution is off by default, and decimal precision widening + was rejected outright until the `SchemaChangeUtils.isTypeUpdateAllow` relaxation. +- **Evidence required:** stacktrace · writer props · `hoodie.properties` (table schema) · for the MOR + variant, table state to show which file slices span the change. +- **Disambiguation:** a compaction frame here means the offending write already committed — see the + MOR deferred-failure rule above. Distinguish from HUDI-RCA-003 by *direction*: a field the table has + and the batch lacks is 003; a field whose **type** changed is 001. +- **Mitigation ladder:** + 1. Upgrade if the signature is decimal precision — fixed in 1.0.x, and no config works around it on + 0.14/0.15. + 2. Set `hoodie.datasource.write.reconcile.schema=true`, with `hoodie.schema.on.read.enable=true`. + 3. Stop dropping or retyping the column in the writer: keep it and write typed nulls. + 4. If MOR and already wedged, the table cannot compact until the pending compaction is unscheduled + or the affected file groups are rewritten. Do not hand-delete the pending instant files. +- **Governing configs:** `hoodie.datasource.write.reconcile.schema`, `hoodie.schema.on.read.enable`, + `hoodie.avro.schema.validate`, `hoodie.datasource.write.schema.allow.auto.evolution.column.drop` +- **`fixedInVersion`:** 1.0.0 for decimal precision widening · **`preflightGuardExists`:** no +- **`alsoAppliesWhenHealthy`:** false +- **Provenance:** #8040, #5452, #11335, #8020, #2919 + +--- + +## HUDI-RCA-002 — Record key or precombine field missing, null, or wrong case + +- **Category:** `input/config` · **Discoverability:** `self-evident` for the key case, `needs-expertise` for precombine +- **Signature** (innermost cause): + - `org\.apache\.hudi\.exception\.HoodieKeyException: recordKey value: "null" for field: "([^"]+)" cannot be null or empty` + - `org\.apache\.hudi\.exception\.HoodieKeyException: Record key has to be non-null!` + - `org\.apache\.hudi\.exception\.HoodieException: \w+\(Part -\w+\) field not found in record\. Acceptable fields were :\[` + - Hudi frames: `HoodieAvroUtils.getNestedFieldVal`, `ComplexKeyGenerator`, often wrapped by + `SparkException: Failed to execute user defined function` so the top frame is Spark, not Hudi. +- **Symptom:** The write fails on the first batch, or only when one composite key field is null. +- **Root cause:** Three sub-cases sharing a remedy family: the configured field does not exist or is + null in some rows; the key config is **case-sensitive** and the user wrote `ID` for a column named + `id`; or HoodieStreamer defaulted `--source-ordering-field` to `ts`, producing a phantom `ts` + requirement that then persisted into `hoodie.properties`. +- **Evidence required:** stacktrace · writer props · `hoodie.properties` +- **Disambiguation:** if the named field is one the user never configured (classically `ts`), this is + the HoodieStreamer default, not a missing column. The message names a field the user has never heard + of, which is the whole confusion — say so explicitly rather than asking them to add a `ts` column. +- **Mitigation ladder:** + 1. Check the exact field name **and case** against the dataframe schema. Hudi does not normalize case. + 2. Filter or default nulls in the key columns before writing. + 3. For HoodieStreamer with no ordering field: set `--source-ordering-field` to a real column, or use + `--op INSERT` / `bulk_insert`, which skip precombine entirely. + 4. Multi-field keys with `bulk_insert` had a genuine bug before 0.13.1 — upgrade, or switch to + `upsert` to confirm. +- **Governing configs:** `hoodie.datasource.write.recordkey.field`, + `hoodie.datasource.write.precombine.field`, `hoodie.datasource.write.keygenerator.class`, + `hoodie.table.precombine.field`, `--source-ordering-field` +- **`fixedInVersion`:** 0.13.1 for the bulk_insert multi-field key bug · **`preflightGuardExists`:** partial — the key check fires at write time, not at config time +- **`alsoAppliesWhenHealthy`:** false +- **Provenance:** #3759, #9799, #11776, #10233 + +--- + +## HUDI-RCA-003 — Source drops a column the table still has + +- **Category:** `input/config` · **Discoverability:** `needs-expertise` +- **Signature** (innermost cause): + - `org\.apache\.parquet\.io\.InvalidRecordException: Parquet/Avro schema mismatch: Avro field '([^']+)' not found` + - Frames, in order: `AvroRecordConverter.getAvroField` → `ParquetReaderIterator.hasNext` → + `HoodieMergeHelper.runMerge` + - Wrapped by `HoodieUpsertException: Error upserting bucketType UPDATE for partition :\d+` +- **Symptom:** Every sync fails after an upstream schema change. The exception arrives on the **merge** + path while *reading existing Parquet*, so operators look at their table instead of their batch. +- **Root cause:** The incoming batch no longer carries a column the table's base files have. Hudi reads + existing records using the latest (incoming) schema, and the Parquet-to-Avro converter cannot find + the field. +- **Evidence required:** stacktrace · writer props · last commit metadata (`extraMetadata.schema`) for + the field's declared type +- **Disambiguation:** same exception class as HUDI-RCA-001's field-not-found variant. The + `HoodieMergeHelper.runMerge` frame and the fact that the field exists in the **table** but not the + **batch** is what makes it 003. Recover the field's declared type including its Avro `logicalType` + before re-adding it, or the re-add fails differently. +- **Mitigation ladder:** + 1. Re-add the column to the incoming batch as a **typed** null via a transformer — `cast(null as + string)`, or `cast(null as date)` where the logical type demands it. Cheapest and reversible. + 2. Set `hoodie.avro.schema.validate=false` only as a stopgap, not as the fix. + 3. **Config-layering trap:** if your config system overrides rather than merges list-valued settings, + adding a transformer silently drops the transformer that was already configured. Re-list all of + them, and verify the resolved config rather than the file you edited. +- **Governing configs:** `hoodie.avro.schema.validate`, + `hoodie.datasource.write.schema.allow.auto.evolution.column.drop`, + `hoodie.datasource.write.reconcile.schema` +- **`fixedInVersion`:** not a defect · **`preflightGuardExists`:** no +- **`alsoAppliesWhenHealthy`:** false +- **Provenance:** #8040 + +--- + +## HUDI-RCA-004 — Unvalidated source schema: records do not parse against the inferred type + +- **Category:** `input/config` · **Discoverability:** `self-evident` for the message, `needs-expertise` for the fix +- **Signature** (innermost cause): + - `org\.apache\.spark\.sql\.catalyst\.util\.BadRecordException` + - `java\.lang\.RuntimeException: Failed to parse a value for data type struct<.*> \(current token: \w+\)` +- **Symptom:** Records fail to parse and the whole job fails, rather than the bad rows being isolated. +- **Root cause:** A semi-structured source (CDC, JSON, document-store feed) emits a field whose actual + shape differs from what was inferred from earlier records — most often an array where a struct was + inferred. +- **Evidence required:** stacktrace · writer props (schema-provider config) · a raw source sample +- **Mitigation ladder:** + 1. Set an explicit **target schema** instead of relying on inference, and enable validation with a + quarantine destination so bad records are isolated rather than failing the commit. + 2. For CDC sources, make sure the target schema declares the envelope fields the transformer adds + (operation type, transaction id, LSN, event timestamps) — otherwise validation rejects every record. + 3. Loosen validation only as a last resort. It converts a loud failure into a silent one. +- **Governing configs:** `hoodie.streamer.schemaprovider.target.schema.file`, + `hoodie.streamer.schemaprovider.source.schema.file`, `hoodie.avro.schema.validate` +- **`fixedInVersion`:** not a defect · **`preflightGuardExists`:** no +- **`alsoAppliesWhenHealthy`:** false +- **Provenance:** none public — the mechanism is config hygiene and rarely filed + +--- + +## HUDI-RCA-005 — Timestamp key generator cannot parse the partition field + +- **Category:** `input/config` · **Discoverability:** `self-evident` +- **Signature** (innermost cause): + - `org\.apache\.hudi\.exception\.HoodieKeyGeneratorException: Unable to parse input partition field: (.+)` + - `java\.lang\.IllegalArgumentException: Invalid format: "(.+)" is malformed at "(.+)"` + - Hudi frame: `TimestampBasedAvroKeyGenerator.getPartitionPath` +- **Symptom:** Every write fails immediately, naming the exact offending value. +- **Root cause:** The configured **input** date format does not match the actual partition-field format. + Typical triggers: microsecond precision, a trailing `Z`, or a switch from epoch to ISO-8601 upstream. +- **Evidence required:** stacktrace alone suffices · writer props to confirm the configured format +- **Mitigation ladder:** + 1. Update the key generator's **input** format config to match the value printed in the exception. + 2. Verify the **output** partition format is unchanged. Changing the output format alters partition + paths and effectively forks the table — a far worse outcome than the parse failure. + 3. If the source format is now inconsistent across records, normalize it in a transformer before the + key generator sees it. +- **Governing configs:** `hoodie.keygen.timebased.input.dateformat`, + `hoodie.keygen.timebased.output.dateformat`, `hoodie.keygen.timebased.timestamp.type`, + `hoodie.keygen.timebased.input.timezone`, `hoodie.keygen.timebased.output.timezone` +- **`fixedInVersion`:** not a defect · **`preflightGuardExists`:** no +- **`alsoAppliesWhenHealthy`:** false +- **Provenance:** none public with this exact signature; included because it is trivially fixable and + almost certainly under-reported + +--- + +## HUDI-RCA-006 — Multi-writer without, or with a misconfigured, lock provider + +- **Category:** `input/config` · **Discoverability:** `needs-expertise`, and **`silent`** in the no-lock-at-all case +- **Signature** (innermost cause): + - `java\.util\.ConcurrentModificationException: Cannot resolve conflicts for overlapping writes`, + wrapped in `HoodieWriteConflictException`; frames + `SimpleConcurrentFileWritesConflictResolutionStrategy.resolveConflict`, + `TransactionUtils.resolveWriteConflictIfAny`, `SparkRDDWriteClient.preCommit` + - `org\.apache\.hudi\.exception\.HoodieLockException: Unable to acquire lock, lock object (null|LockResponse\(lockid:\d+, state:WAITING\))`; + frames `LockManager.lock`, `TransactionManager.beginTransaction` + - `java\.lang\.IllegalArgumentException: ALREADY_ACQUIRED` at `HiveMetastoreBasedLockProvider.acquireLock` + - `org\.apache\.hadoop\.fs\.FileAlreadyExistsException: File already exists: .*\.hoodie/\d+\.commit\.requested` + at `HoodieActiveTimeline.createImmutableFileInPath` +- **Symptom:** Intermittent failures that scale with writer count. With **no** lock provider there may + be no exception at all — just overwritten timeline state and lost data. +- **Root cause:** Two writers against one table without OCC properly configured, or OCC configured and + genuinely conflicting. The semantics users most often get wrong: **the OCC lock is table-level, not + partition-level**, and it serializes only the commit-validation phase. Writers on disjoint partitions + still serialize on the lock. +- **Evidence required:** stacktrace · **writer props — mandatory** · `hoodie.properties` +- **Disambiguation (the most important row in this catalog):** the conflict exception and the no-lock + corruption produce the **same stacktrace**. Only the writer config splits them. + - `hoodie.write.lock.provider` set, concurrency intended → by-design rejection of the later committer. + Expected. Add orchestrator-level retry of the whole application attempt. Severity: routine. + - No lock provider set, but two writers demonstrably exist → the table is being corrupted and this + exception is the *lucky* case where you found out. Severity: urgent. + - Treat "is a second writer intended?" as a question to ask the user, not something to infer. +- **Mitigation ladder:** + 1. If two writers exist and no lock provider is set, set one. Non-negotiable; absence is silent + corruption, not a performance choice. + 2. **Lower** `hoodie.write.lock.wait_time_ms` to around 5000. Counter-intuitive: a long wait converts + a fast, retryable rejection into a long stall that looks like a hang, and the later writer has to + retry regardless. + 3. For DynamoDB locks, set the endpoint URL explicitly and use on-demand billing; stale lock rows + occasionally need manual removal. + 4. Add retry of the whole application attempt at the orchestrator. The later committer is rejected by + design and must retry. + 5. Where lock serialization is itself the bottleneck on MOR, consider 1.x non-blocking concurrency + control, which pairs best with a bucket index. +- **Governing configs:** `hoodie.write.concurrency.mode`, `hoodie.write.lock.provider`, + `hoodie.write.lock.wait_time_ms`, `hoodie.write.lock.num_retries`, `hoodie.write.lock.dynamodb.*`, + `hoodie.write.lock.zookeeper.*`, `hoodie.cleaner.policy.failed.writes` +- **`fixedInVersion`:** n/a — NBCC added in 1.x as an alternative mode, not a fix +- **`preflightGuardExists`:** no — Hudi does not detect "two writers, no lock" at write time +- **`alsoAppliesWhenHealthy`:** true — lock wait tuning is an optimization on a merely-slow multi-writer setup +- **Provenance:** #7653, #3731, #4456, #6226, #9213 + +--- + +## HUDI-RCA-007 — Hive or Glue catalog sync failure: table written but invisible + +- **Category:** `input/config` · **Discoverability:** `needs-expertise` +- **Signature** (innermost cause): + - `java\.lang\.RuntimeException: Unable to instantiate org\.apache\.hadoop\.hive\.ql\.metadata\.SessionHiveMetaStoreClient`; + frames `HMSDDLExecutor.<init>`, `HiveSyncTool.initClient` + - `NoSuchObjectException: .* table not found` + - Wrapped in `HoodieHiveSyncException: Got runtime exception when hive syncing` or + `Could not sync using the meta sync class` + - Log giveaway: `Trying to connect to metastore with URI thrift://localhost:9083` when the user + believes they configured Glue +- **Symptom:** Data lands on storage correctly but the query engine cannot see it, or the write fails + only in the post-commit sync step. +- **Root cause:** Sync mode and catalog wiring mismatch, Hive metastore version incompatibility, or + table-name case sensitivity. The `localhost:9083` line is the tell that the intended catalog config + never took effect. +- **Evidence required:** stacktrace · writer props · driver log (for the metastore URI line) +- **Mitigation ladder:** + 1. Lowercase the table and database names everywhere. An uppercase table name causes a rename failure. + 2. Set `hoodie.datasource.hive_sync.mode` explicitly — `hms` for Glue and HMS. + 3. For Glue, set the AWS Glue sync tool class explicitly rather than relying on inference. + 4. Verify the Hive metastore version is 2.3 or newer for Spark 3 bundles. + 5. Confirm `hoodie.datasource.hive_sync.partition_extractor_class` matches the actual partitioning — + the non-partitioned extractor for unpartitioned tables, the multi-part extractor for multi-level. +- **Governing configs:** `hoodie.datasource.hive_sync.enable`, `hoodie.datasource.hive_sync.mode`, + `hoodie.datasource.hive_sync.database`, `hoodie.datasource.hive_sync.table`, + `hoodie.datasource.hive_sync.partition_extractor_class`, `hoodie.datasource.hive_sync.partition_fields`, + `hoodie.meta.sync.client.tool.class`, `hoodie.datasource.write.hive_style_partitioning` +- **`fixedInVersion`:** 0.11.1 for the Glue client-factory regression · **`preflightGuardExists`:** no +- **`alsoAppliesWhenHealthy`:** false +- **Provenance:** #5484, #8368, #2409, #954 + +--- + +## HUDI-RCA-008 — Config changed between runs against an existing table + +- **Category:** `input/config` · **Discoverability:** `self-evident` when the guard fires, **`silent`** when it does not +- **Signature:** + - `HoodieException: Config conflict \(key, current value, existing value\): ` — when Hudi's guard + catches it + - When the guard does not catch it: **no exception**. Duplicates, no-op deletes, or an unqueryable + table. +- **Symptom:** A pipeline that ran for months starts failing, or silently producing duplicates, after + someone edited the writer config or upgraded Hudi. +- **Root cause:** Several table-level properties are written into `hoodie.properties` on first write and + are effectively **immutable**: record key field, partition path field, key generator class, table + type, precombine field, hive-style partitioning. Changing them later either trips the conflict guard + or silently reinterprets existing records. +- **Evidence required:** `hoodie.properties` · writer props. **This pattern is definitionally a diff + between the two** — no stacktrace substitutes for it, and this is why the skill requests + `hoodie.properties` by default. +- **Disambiguation:** overlaps HUDI-RCA-013 (silent duplicates). If the properties diff shows a changed + key generator or record-key field, the duplicates are 008's mechanism and 013's symptom. Report 008 — + it names the cause. +- **Mitigation ladder:** + 1. Read `.hoodie/hoodie.properties` and diff it against the current writer config. This is step one of + nearly any Hudi triage, not just this pattern. + 2. Revert the changed property to match the table. + 3. If the change is genuinely required, rewrite the table into a new path with `bulk_insert` rather + than fighting the guard. Editing `hoodie.properties` by hand is not a migration. + 4. For upgrade-induced key-encoding changes, pin the old version or rewrite. A single-field record key + under `ComplexKeyGenerator` was encoded differently from 0.14.1 onward, which silently created + duplicates and silently no-op'd deletes. +- **Governing configs:** `hoodie.table.recordkey.fields`, `hoodie.table.partition.fields`, + `hoodie.table.keygenerator.class`, `hoodie.table.type`, `hoodie.table.precombine.field`, + `hoodie.table.version`, `hoodie.datasource.write.hive_style_partitioning` +- **`fixedInVersion`:** no — the 0.14.1 key-encoding change shipped without a migration path +- **`preflightGuardExists`:** partial — the conflict guard covers some keys, not all +- **`alsoAppliesWhenHealthy`:** false +- **Provenance:** #10508, #10587, #10233 + +--- + +## HUDI-RCA-009 — Wrong query type or engine wiring makes data look missing or stale + +- **Category:** `input/config` · **Discoverability:** **`silent`** +- **Signature:** **No exception.** Recognised by symptom, not by trace: an `INSERT OVERWRITE` wrote + files but `SELECT` returns old data; only a `_ro` table exists where the user expected `_rt`; + timestamps render as implausible far-future dates in the query engine. +- **Symptom:** Data is demonstrably on storage but the query returns stale, empty, or garbled results. +- **Root cause:** Query-side wiring, not write-side. Reading the read-optimized (`_ro`) view of a MOR + table and seeing pre-compaction state; missing `spark.sql.hive.convertMetastoreParquet=false`, so + Spark uses its own Parquet reader and bypasses Hudi's file-slice resolution; expecting two Hive tables + for a Copy-on-Write table, which only ever has one; or a timestamp type the catalog declares + differently from how it was written. +- **Evidence required:** `hoodie.properties` (table type) · the query and the engine · writer props +- **Proactive detection:** compare `COUNT(*)` on the snapshot view against the `_ro` view. A gap that + closes after compaction is this pattern, by design, not data loss. +- **Mitigation ladder:** + 1. Set `spark.sql.hive.convertMetastoreParquet=false`. Note that whitespace around `=` in a SQL `set` + has been enough to make this silently not apply — verify it took effect. + 2. Query the snapshot view, not `_ro`; set `hoodie.datasource.query.type=snapshot`. + 3. For MOR, confirm compaction has actually run if you need read-optimized freshness. + 4. For timestamp rendering, enable `hoodie.datasource.hive_sync.support_timestamp`. +- **Governing configs:** `hoodie.datasource.query.type`, + `hoodie.datasource.hive_sync.support_timestamp`, `spark.sql.hive.convertMetastoreParquet`, + `hoodie.datasource.write.drop.partition.columns` +- **`fixedInVersion`:** not a defect · **`preflightGuardExists`:** no +- **`alsoAppliesWhenHealthy`:** false +- **Provenance:** #4154, #2409, #2123, #2338 + +--- + +## HUDI-RCA-010 — HoodieStreamer source or schema-provider misconfiguration + +- **Category:** `input/config` · **Discoverability:** `needs-expertise` +- **Signature:** + - `org\.apache\.hudi\.exception\.HoodieException: Commit \d+ failed and rolled-back !` at + `DeltaSync.writeToSink` / `StreamSync.writeToSink` — **this message carries no diagnostic content + whatsoever** + - `java\.lang\.NullPointerException` inside `AvroDeserializer` with no Hudi frame + - Structurally wrong data: a whole CDC envelope ingested instead of the unwrapped row +- **Symptom:** The Streamer fails with a generic "commit failed and rolled-back", or ingests + structurally wrong data. +- **Root cause:** Schema provider mismatched to source. A provider returning a null target schema while + a transformer is configured; a generic Avro source used where a CDC-specific source and payload class + are required to unwrap the envelope; Confluent wire-format Avro needing registry-aware deserialization. +- **Evidence required:** stacktrace · **Streamer CLI args** · driver log — the real exception is in the + driver log *below* the "failed and rolled-back" line, not in the thrown one +- **Disambiguation:** `Commit ... failed and rolled-back` on its own is a **content-free wrapper**. Do + not attribute from it. Ask for the driver log and classify from the exception beneath it. If the user + can only supply this line, say so and request the driver log — that is the correct outcome, not a guess. +- **Mitigation ladder:** + 1. Read the driver log below the "failed and rolled-back" line and classify the real exception. + 2. For CDC sources, use the database-specific source class with its matching payload class, not a + generic Avro source. + 3. For Confluent Avro, let the Kafka Avro source fetch from the registry rather than hand-rolling a + schema provider. + 4. If using a transformer, do not return null from the target schema provider. Set a real schema, or + omit the provider so Hudi infers from the dataframe. +- **Governing configs:** `--source-class`, `--schemaprovider-class`, `--payload-class`, + `--transformer-class`, `--source-ordering-field`, `hoodie.streamer.schemaprovider.registry.url`, + `hoodie.streamer.source.kafka.topic` +- **`fixedInVersion`:** no · **`preflightGuardExists`:** no — and the uninformative message is itself + worth an upstream report +- **`alsoAppliesWhenHealthy`:** false +- **Provenance:** #2515, #2149, #2589, #8519, #10233, #5540 + +--- + +## HUDI-RCA-011 — Source initialization fails: credentials or schema registry + +- **Category:** `input/config` · **Discoverability:** `self-evident` for credentials, `needs-expertise` for the registry case +- **Signature:** Failures during writer startup, **before any batch is read**: + - `org\.apache\.kafka\.common\.errors\.\w+Exception`, SSL or keystore load failures, + connection-refused against brokers + - `org\.apache\.hudi\.utilities\.exception\.HoodieSchemaProviderException` +- **Symptom:** The job never ingests anything. No data lands, so there is no partial state to reason + about — which is itself the diagnostic. +- **Root cause:** Source credentials or TLS material are wrong or unreadable, or the source requires a + schema registry that was never configured. A common origin: a stream configuration **cloned** from an + older one created before registry support existed, carrying an empty registry reference that only + fails at writer init. +- **Evidence required:** stacktrace · writer props **as resolved, not as written** · the age of the + failure signal +- **Disambiguation:** distinguish an **init** failure from a **sync** failure. An init-failure counter + that never decrements can reflect a long-past startup failure on a driver that has since recovered. + Confirm the failure is recent before investigating it. +- **Mitigation ladder:** + 1. Fix the credential or registry configuration and restart. + 2. **Do not create a stream by cloning another** when the source type has gained required fields since + the original was created. Recreate from scratch — the clone silently carries empty required fields. + 3. Validate required source fields at creation time rather than discovering them at writer init. +- **Governing configs:** `hoodie.streamer.schemaprovider.registry.url`, + `hoodie.streamer.source.kafka.topic`, Kafka client security properties passed through + `hoodie.streamer.kafka.*` +- **`fixedInVersion`:** not a defect · **`preflightGuardExists`:** no +- **`alsoAppliesWhenHealthy`:** false +- **Provenance:** none public — onboarding-time failures are rarely filed + +--- + +## HUDI-RCA-012 — Checkpoint ahead of the source: offsets aged out or topic recreated + +- **Category:** `input/config` (persistent case) / `environment` (transient case) · **Discoverability:** **`silent`** in its most common form +- **Signature:** + - `org\.apache\.kafka\.clients\.consumer\.OffsetOutOfRangeException: Fetch position \S+ is out of range for partition` + - `Some data may have been lost because they are not available in Kafka any more` + - Or **no exception at all** — a healthy-looking job ingesting nothing +- **Symptom:** The stream looks perfectly healthy and ingests nothing. +- **Root cause — two causes with opposite correct responses:** + - **Persistent:** the topic was deleted and recreated, or retention aged past the committed + checkpoint. The checkpoint is permanently beyond the high watermark and will never recover. + - **Transient:** the broker's latest offset briefly moved backward (unclean leader election, replica + truncation, rebalance). Self-heals on the next produce. +- **Evidence required:** writer checkpoint from commit metadata · source topic offsets and metadata · + stacktrace where present +- **Disambiguation:** compare the committed checkpoint against current end offsets, then **watch the + drift for one interval**. Transient regressions close within minutes; persistent ones do not move at + all. A topic creation timestamp later than your last successful commit means recreation. Retention + settings alone do not delete a topic — if the topic is gone, something deleted it. +- **Mitigation ladder:** + 1. **Transient: do nothing.** Resetting the checkpoint here loses data for no reason. This is the + counter-intuitive step and the most common mistake on this pattern. + 2. **Persistent:** reset the checkpoint to the earliest available offset or a deliberately chosen + position, accepting that the gap is unrecoverable. Write down what was skipped. + 3. Confirm with the source owner before resetting anything. + 4. Prevent: set retention comfortably above your worst-case ingestion outage, and never delete a topic + a live stream is checkpointed against. +- **Governing configs:** `hoodie.streamer.source.kafka.topic`, + `auto.offset.reset` (the native Kafka consumer property, passed through — not a `hoodie.*` key), + `hoodie.streamer.checkpoint.provider.path` +- **`fixedInVersion`:** not a defect · **`preflightGuardExists`:** no +- **`alsoAppliesWhenHealthy`:** false +- **Provenance:** none public with this signature + +--- + +## HUDI-RCA-013 — Wrong operation or index choice produces silent duplicates + +- **Category:** `input/config` · **Discoverability:** **`silent`** — the highest-value row in the catalog +- **Signature:** **No exception.** Found only by running + `SELECT key, COUNT(*) FROM t GROUP BY key HAVING COUNT(*) > 1`, or noticed as unexplained table growth. +- **Symptom:** A primary key appears more than once. Often accompanied by table size growing far faster + than ingested volume. +- **Root cause:** A ladder of distinct mechanisms the user cannot tell apart without help: + `hoodie.datasource.write.operation` set to `insert` or `bulk_insert`, which **do not deduplicate + against existing data by design**; a non-global index with a record that moved partitions, so the key + legitimately exists twice; a global index on MOR where read-optimized queries show duplicates until + compaction runs; querying through Hive without `convertMetastoreParquet=false`; or a key-generator + encoding change across versions. +- **Evidence required:** `hoodie.properties` · writer props · the dedup query result · table type +- **Proactive detection:** `SELECT COUNT(1), COUNT(DISTINCT <record_key>)` on the **snapshot** view. + Run it after any index or key-generator change. +- **Disambiguation:** before calling this a defect, establish which mechanism applies. Cross-partition + duplicates with a non-global index are **working as configured**. Duplicates on a MOR `_ro` view that + disappear after compaction are **transient by design**. Only duplicates on the snapshot view with + `upsert` and a correct index are a real fault — and then check HUDI-RCA-008 for a key-encoding change. +- **Mitigation ladder:** + 1. Confirm `hoodie.datasource.write.operation=upsert`. This alone explains most reports. + 2. Determine whether the duplicates are cross-partition. If so, move to a global index and set + `hoodie.bloom.index.update.partition.path=true`. + 3. Re-run the dedup check against the **snapshot** view, not `_ro`, with + `convertMetastoreParquet=false`. + 4. If MOR, force a compaction and re-check — read-optimized duplicates may be transient by design. + 5. If the table crossed 0.14.1 with a single-field record key under `ComplexKeyGenerator`, compare + `_hoodie_record_key` encoding before and after: this is the known encoding regression, and the + remedy is to pin the old version or rewrite the table. +- **Governing configs:** `hoodie.datasource.write.operation`, `hoodie.index.type`, + `hoodie.bloom.index.update.partition.path`, `hoodie.datasource.write.keygenerator.class`, + `hoodie.combine.before.insert`, `hoodie.datasource.query.type` +- **`fixedInVersion`:** no · **`preflightGuardExists`:** no +- **`alsoAppliesWhenHealthy`:** false +- **Provenance:** #2338, #2255, #5777, #10508 + +--- + +# Group B — sizing/scale + +The config was right at 1,000 file groups and wrong at 500,000. Nobody's bug: the remedy is to re-tune +and add a guardrail. This is the category where a generic LLM most often gives the exactly wrong +advice, because several ladders here run *downward*. + +--- + +## HUDI-RCA-014 — Bloom index OOM from low parallelism or key skew Review Comment: Agreed. The catalog from #20195 confirms it: `hoodie.index.type` has no declared default at all, and the effective one is written per engine in `HoodieIndexConfig.Builder.build()` — `SIMPLE` on Spark, `INMEMORY` on Flink, `SIMPLE` on Java. So on a default Spark table every bloom key in that ladder is read by nothing. Adding "confirm the index type" as step 1, and making the bloom steps conditional on it rather than assumed. The ladder should also say what to do when the answer is `SIMPLE`, otherwise step 1 just dead-ends the reader. ########## hudi-ai-operator/skills/hudi-rca/references/failure-catalog.md: ########## @@ -0,0 +1,1576 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> +# Hudi failure catalog + +38 failure patterns, each with the signature that identifies it, the mechanism behind it, and an +ordered mitigation ladder. Used by the `hudi-rca` skill to classify a pasted failure; readable on its +own as a troubleshooting reference. + +## How to read a row + +| Field | Meaning | +|---|---| +| **ID** | Stable identifier. Never renumber; retire instead. | +| **Category** | `input/config` · `sizing/scale` · `environment` · `defect` · `not-a-failure` | +| **Discoverability** | `self-evident` (the message says what to do) · `needs-expertise` (the message is generic, misleading, or points at the wrong layer) · `silent` (no exception at all) | +| **Signature** | Exception classes, message regexes, and the Hudi frames that identify the pattern. | +| **Symptom** | What the operator sees, in their terms. | +| **Root cause** | The mechanism, in one or two sentences. | +| **Evidence required** | Which rungs of the evidence ladder are needed to reach a confident call. | +| **Disambiguation** | Present only where a signature maps to more than one pattern. The explicit discriminator. | +| **Mitigation ladder** | Ordered, cheapest and most likely first. Counter-intuitive steps are marked and justified. | +| **Governing configs** | Exact `hoodie.*` / `spark.*` keys. | +| **`fixedInVersion`** | The Hudi release that fixes the defect, where one exists. | +| **`preflightGuardExists`** | Whether a current Hudi release fails fast on the condition instead of corrupting silently. | +| **`alsoAppliesWhenHealthy`** | `true` when the same advice is an optimization on a merely-slow job, not only a fix for a broken one. | +| **Provenance** | Public `apache/hudi` issue numbers. | + +## Three rules that govern every row + +**1. Match the innermost `Caused by`, never the wrapper.** In the public tracker the wrapper classes +dominate the counts — bare `HoodieException` 498 occurrences, `HoodieUpsertException` 189 — while the +classes that actually identify the fault barely appear: `SchemaCompatabilityException` 1, +`HoodieAvroSchemaException` 2, `HoodieKeyGeneratorException` 3. Users paste the wrapper. Every regex +below is written against the innermost cause plus the first Hudi frame, because a classifier keyed on +the top-level exception class will mostly see wrappers and learn nothing. + +**2. The stacktrace is the weakest signal, not the backbone.** For a large share of these the terminal +exception is generic and the real cause is table state or config sizing several steps removed. A +`413 Payload Too Large` on timeline init is really a forgotten savepoint blocking archival. A stalled +ingestion is really a rollback that never finished and still holds the lock. Where a row says the +evidence requires table state, the honest move is to run `HoodieTableHealthChecker` rather than parse +the trace harder. + +**3. Ask the Hudi version early.** The heavy issue clusters are 0.9 through 0.15, and several +high-traffic patterns here are fixed outright in 1.0 or 1.2. For this population "upgrade" is a +genuinely good first-line remedy in a way it usually is not. Rows carry `fixedInVersion` where known. + +## Disambiguation index + +Four signatures map to more than one pattern. Resolve them here before attributing. + +| Signature | Candidates | Discriminator | +|---|---|---| +| `ConcurrentModificationException` / `HoodieWriteConflictException` | HUDI-RCA-006 healthy OCC retry **vs** HUDI-RCA-006 no lock provider | **Writer config, not the trace** — the stacktrace is identical. If `hoodie.write.lock.provider` is set and two writers are intended, this is a by-design rejection: retry the application attempt. If no lock provider is set and two writers exist, the table is being corrupted silently and the conflict exception is the lucky case. | +| `HoodieRollbackException` | HUDI-RCA-027 rollback wedge · HUDI-RCA-026 metadata-table divergence · HUDI-RCA-025 dangling files | **The frame below `rollbackFailedWrites`.** `AsyncCleanerService.waitForCompletion` → cleaner/metadata-table path (026). A lock-provider frame (`LockManager.lock`, `*LockProvider.acquireLock`) → lock problem first (006). `HoodieMergeHandle.getLatestBaseFile` with `FileID ... does not exist` → 025. | +| `OutOfMemoryError` | HUDI-RCA-015 merge-side · HUDI-RCA-015 archival-side · HUDI-RCA-014 bloom index · HUDI-RCA-021 driver/timeline-server | **The Hudi frame.** `HoodieMergedLogRecordScanner` / `ExternalSpillableMap` → merge side (015a). `HoodieTimelineArchiveLog.writeToFile` / `HoodieAvroDataBlock.serializeRecords` → **archival side (015b), where adding executor memory is the wrong answer entirely** — the fix is to shrink commit metadata. Kryo / `ExternalSorter` frames under a shuffle read with no Hudi frame → bloom index (014). `Requested array size exceeds VM limit` in a driver Jetty thread → 021, a structural ceiling that more heap cannot move. | +| `FileNotFoundException` | HUDI-RCA-031 markers · HUDI-RCA-033 archiver race · HUDI-RCA-025 stale plan · HUDI-RCA-036 source file | **The path.** `MARKERS*` or `.hoodie/.temp/` → 031. A `.commit` / `.deltacommit` under `.hoodie/` → 033. A data or log file named in a compaction plan → 025. A path under the **source** bucket, not under the table → 036. A path under the table ending `.log.<n>` → treat as a compaction race, see 025's note. | + +## Cross-cutting trap: MOR defers the failure + +On Merge-on-Read an incompatible schema write **succeeds**, and compaction fails hours later in a +*different job* whose stacktrace names `HoodieCompactor` rather than the write. The table then cannot +compact and every later write fails. Cause and symptom are separated by both time and process. + +**Rule: a compaction frame plus a schema exception means the bad write happened earlier, in another +job. Investigate the writes from that window, not the job that just failed.** This applies to +HUDI-RCA-001, HUDI-RCA-003 and HUDI-RCA-004. + +--- + +# Group A — input/config + +The user asked Hudi to do something invalid, or configured it inconsistently. The remedy is to fix +data or config; nothing is wrong with Hudi or the cluster. + +--- + +## HUDI-RCA-001 — Schema evolution the writer rejects + +- **Category:** `input/config` · **Discoverability:** `needs-expertise` +- **Signature** (innermost cause): + - `org\.apache\.avro\.AvroTypeException: Cannot encode decimal with precision (\d+) as max precision (\d+)` + - `org\.apache\.parquet\.io\.InvalidRecordException: Parquet/Avro schema mismatch: Avro field '([^']+)' not found` + - Hudi frames: `HoodieAvroUtils.rewriteRecordWithNewSchema`, `HoodieAvroUtils.rewritePrimaryTypeWithDiffSchemaType`, `HoodieMergeHelper.runMerge`, `AbstractHoodieLogRecordReader.scanInternal`, `HoodieCompactor.compact` +- **Symptom:** A write that worked yesterday fails after an upstream column was retyped or a decimal + widened. On MOR the write succeeds and **compaction** fails hours later, permanently, for every + subsequent write. +- **Root cause:** The incoming Avro schema is not a backward-compatible evolution of the table schema + under Hudi's rules. Full schema-on-read evolution is off by default, and decimal precision widening + was rejected outright until the `SchemaChangeUtils.isTypeUpdateAllow` relaxation. +- **Evidence required:** stacktrace · writer props · `hoodie.properties` (table schema) · for the MOR + variant, table state to show which file slices span the change. +- **Disambiguation:** a compaction frame here means the offending write already committed — see the + MOR deferred-failure rule above. Distinguish from HUDI-RCA-003 by *direction*: a field the table has + and the batch lacks is 003; a field whose **type** changed is 001. +- **Mitigation ladder:** + 1. Upgrade if the signature is decimal precision — fixed in 1.0.x, and no config works around it on + 0.14/0.15. + 2. Set `hoodie.datasource.write.reconcile.schema=true`, with `hoodie.schema.on.read.enable=true`. + 3. Stop dropping or retyping the column in the writer: keep it and write typed nulls. + 4. If MOR and already wedged, the table cannot compact until the pending compaction is unscheduled + or the affected file groups are rewritten. Do not hand-delete the pending instant files. +- **Governing configs:** `hoodie.datasource.write.reconcile.schema`, `hoodie.schema.on.read.enable`, + `hoodie.avro.schema.validate`, `hoodie.datasource.write.schema.allow.auto.evolution.column.drop` +- **`fixedInVersion`:** 1.0.0 for decimal precision widening · **`preflightGuardExists`:** no +- **`alsoAppliesWhenHealthy`:** false +- **Provenance:** #8040, #5452, #11335, #8020, #2919 + +--- + +## HUDI-RCA-002 — Record key or precombine field missing, null, or wrong case + +- **Category:** `input/config` · **Discoverability:** `self-evident` for the key case, `needs-expertise` for precombine +- **Signature** (innermost cause): + - `org\.apache\.hudi\.exception\.HoodieKeyException: recordKey value: "null" for field: "([^"]+)" cannot be null or empty` + - `org\.apache\.hudi\.exception\.HoodieKeyException: Record key has to be non-null!` + - `org\.apache\.hudi\.exception\.HoodieException: \w+\(Part -\w+\) field not found in record\. Acceptable fields were :\[` + - Hudi frames: `HoodieAvroUtils.getNestedFieldVal`, `ComplexKeyGenerator`, often wrapped by + `SparkException: Failed to execute user defined function` so the top frame is Spark, not Hudi. +- **Symptom:** The write fails on the first batch, or only when one composite key field is null. +- **Root cause:** Three sub-cases sharing a remedy family: the configured field does not exist or is + null in some rows; the key config is **case-sensitive** and the user wrote `ID` for a column named + `id`; or HoodieStreamer defaulted `--source-ordering-field` to `ts`, producing a phantom `ts` + requirement that then persisted into `hoodie.properties`. +- **Evidence required:** stacktrace · writer props · `hoodie.properties` +- **Disambiguation:** if the named field is one the user never configured (classically `ts`), this is + the HoodieStreamer default, not a missing column. The message names a field the user has never heard + of, which is the whole confusion — say so explicitly rather than asking them to add a `ts` column. +- **Mitigation ladder:** + 1. Check the exact field name **and case** against the dataframe schema. Hudi does not normalize case. + 2. Filter or default nulls in the key columns before writing. + 3. For HoodieStreamer with no ordering field: set `--source-ordering-field` to a real column, or use + `--op INSERT` / `bulk_insert`, which skip precombine entirely. + 4. Multi-field keys with `bulk_insert` had a genuine bug before 0.13.1 — upgrade, or switch to + `upsert` to confirm. +- **Governing configs:** `hoodie.datasource.write.recordkey.field`, + `hoodie.datasource.write.precombine.field`, `hoodie.datasource.write.keygenerator.class`, + `hoodie.table.precombine.field`, `--source-ordering-field` +- **`fixedInVersion`:** 0.13.1 for the bulk_insert multi-field key bug · **`preflightGuardExists`:** partial — the key check fires at write time, not at config time +- **`alsoAppliesWhenHealthy`:** false +- **Provenance:** #3759, #9799, #11776, #10233 + +--- + +## HUDI-RCA-003 — Source drops a column the table still has + +- **Category:** `input/config` · **Discoverability:** `needs-expertise` +- **Signature** (innermost cause): + - `org\.apache\.parquet\.io\.InvalidRecordException: Parquet/Avro schema mismatch: Avro field '([^']+)' not found` + - Frames, in order: `AvroRecordConverter.getAvroField` → `ParquetReaderIterator.hasNext` → + `HoodieMergeHelper.runMerge` + - Wrapped by `HoodieUpsertException: Error upserting bucketType UPDATE for partition :\d+` +- **Symptom:** Every sync fails after an upstream schema change. The exception arrives on the **merge** + path while *reading existing Parquet*, so operators look at their table instead of their batch. +- **Root cause:** The incoming batch no longer carries a column the table's base files have. Hudi reads + existing records using the latest (incoming) schema, and the Parquet-to-Avro converter cannot find + the field. +- **Evidence required:** stacktrace · writer props · last commit metadata (`extraMetadata.schema`) for + the field's declared type +- **Disambiguation:** same exception class as HUDI-RCA-001's field-not-found variant. The + `HoodieMergeHelper.runMerge` frame and the fact that the field exists in the **table** but not the + **batch** is what makes it 003. Recover the field's declared type including its Avro `logicalType` + before re-adding it, or the re-add fails differently. +- **Mitigation ladder:** + 1. Re-add the column to the incoming batch as a **typed** null via a transformer — `cast(null as + string)`, or `cast(null as date)` where the logical type demands it. Cheapest and reversible. + 2. Set `hoodie.avro.schema.validate=false` only as a stopgap, not as the fix. + 3. **Config-layering trap:** if your config system overrides rather than merges list-valued settings, + adding a transformer silently drops the transformer that was already configured. Re-list all of + them, and verify the resolved config rather than the file you edited. +- **Governing configs:** `hoodie.avro.schema.validate`, + `hoodie.datasource.write.schema.allow.auto.evolution.column.drop`, + `hoodie.datasource.write.reconcile.schema` +- **`fixedInVersion`:** not a defect · **`preflightGuardExists`:** no +- **`alsoAppliesWhenHealthy`:** false +- **Provenance:** #8040 + +--- + +## HUDI-RCA-004 — Unvalidated source schema: records do not parse against the inferred type + +- **Category:** `input/config` · **Discoverability:** `self-evident` for the message, `needs-expertise` for the fix +- **Signature** (innermost cause): + - `org\.apache\.spark\.sql\.catalyst\.util\.BadRecordException` + - `java\.lang\.RuntimeException: Failed to parse a value for data type struct<.*> \(current token: \w+\)` +- **Symptom:** Records fail to parse and the whole job fails, rather than the bad rows being isolated. +- **Root cause:** A semi-structured source (CDC, JSON, document-store feed) emits a field whose actual + shape differs from what was inferred from earlier records — most often an array where a struct was + inferred. +- **Evidence required:** stacktrace · writer props (schema-provider config) · a raw source sample +- **Mitigation ladder:** + 1. Set an explicit **target schema** instead of relying on inference, and enable validation with a + quarantine destination so bad records are isolated rather than failing the commit. + 2. For CDC sources, make sure the target schema declares the envelope fields the transformer adds + (operation type, transaction id, LSN, event timestamps) — otherwise validation rejects every record. + 3. Loosen validation only as a last resort. It converts a loud failure into a silent one. +- **Governing configs:** `hoodie.streamer.schemaprovider.target.schema.file`, + `hoodie.streamer.schemaprovider.source.schema.file`, `hoodie.avro.schema.validate` +- **`fixedInVersion`:** not a defect · **`preflightGuardExists`:** no +- **`alsoAppliesWhenHealthy`:** false +- **Provenance:** none public — the mechanism is config hygiene and rarely filed + +--- + +## HUDI-RCA-005 — Timestamp key generator cannot parse the partition field + +- **Category:** `input/config` · **Discoverability:** `self-evident` +- **Signature** (innermost cause): + - `org\.apache\.hudi\.exception\.HoodieKeyGeneratorException: Unable to parse input partition field: (.+)` + - `java\.lang\.IllegalArgumentException: Invalid format: "(.+)" is malformed at "(.+)"` + - Hudi frame: `TimestampBasedAvroKeyGenerator.getPartitionPath` +- **Symptom:** Every write fails immediately, naming the exact offending value. +- **Root cause:** The configured **input** date format does not match the actual partition-field format. + Typical triggers: microsecond precision, a trailing `Z`, or a switch from epoch to ISO-8601 upstream. +- **Evidence required:** stacktrace alone suffices · writer props to confirm the configured format +- **Mitigation ladder:** + 1. Update the key generator's **input** format config to match the value printed in the exception. + 2. Verify the **output** partition format is unchanged. Changing the output format alters partition + paths and effectively forks the table — a far worse outcome than the parse failure. + 3. If the source format is now inconsistent across records, normalize it in a transformer before the + key generator sees it. +- **Governing configs:** `hoodie.keygen.timebased.input.dateformat`, + `hoodie.keygen.timebased.output.dateformat`, `hoodie.keygen.timebased.timestamp.type`, + `hoodie.keygen.timebased.input.timezone`, `hoodie.keygen.timebased.output.timezone` +- **`fixedInVersion`:** not a defect · **`preflightGuardExists`:** no +- **`alsoAppliesWhenHealthy`:** false +- **Provenance:** none public with this exact signature; included because it is trivially fixable and + almost certainly under-reported + +--- + +## HUDI-RCA-006 — Multi-writer without, or with a misconfigured, lock provider + +- **Category:** `input/config` · **Discoverability:** `needs-expertise`, and **`silent`** in the no-lock-at-all case +- **Signature** (innermost cause): + - `java\.util\.ConcurrentModificationException: Cannot resolve conflicts for overlapping writes`, + wrapped in `HoodieWriteConflictException`; frames + `SimpleConcurrentFileWritesConflictResolutionStrategy.resolveConflict`, + `TransactionUtils.resolveWriteConflictIfAny`, `SparkRDDWriteClient.preCommit` + - `org\.apache\.hudi\.exception\.HoodieLockException: Unable to acquire lock, lock object (null|LockResponse\(lockid:\d+, state:WAITING\))`; + frames `LockManager.lock`, `TransactionManager.beginTransaction` + - `java\.lang\.IllegalArgumentException: ALREADY_ACQUIRED` at `HiveMetastoreBasedLockProvider.acquireLock` + - `org\.apache\.hadoop\.fs\.FileAlreadyExistsException: File already exists: .*\.hoodie/\d+\.commit\.requested` + at `HoodieActiveTimeline.createImmutableFileInPath` +- **Symptom:** Intermittent failures that scale with writer count. With **no** lock provider there may + be no exception at all — just overwritten timeline state and lost data. +- **Root cause:** Two writers against one table without OCC properly configured, or OCC configured and + genuinely conflicting. The semantics users most often get wrong: **the OCC lock is table-level, not + partition-level**, and it serializes only the commit-validation phase. Writers on disjoint partitions + still serialize on the lock. +- **Evidence required:** stacktrace · **writer props — mandatory** · `hoodie.properties` +- **Disambiguation (the most important row in this catalog):** the conflict exception and the no-lock + corruption produce the **same stacktrace**. Only the writer config splits them. + - `hoodie.write.lock.provider` set, concurrency intended → by-design rejection of the later committer. + Expected. Add orchestrator-level retry of the whole application attempt. Severity: routine. + - No lock provider set, but two writers demonstrably exist → the table is being corrupted and this + exception is the *lucky* case where you found out. Severity: urgent. + - Treat "is a second writer intended?" as a question to ask the user, not something to infer. +- **Mitigation ladder:** + 1. If two writers exist and no lock provider is set, set one. Non-negotiable; absence is silent + corruption, not a performance choice. Review Comment: Agreed, and this is the same gap the config consultant found on #20195 from the other direction. `hoodie.write.concurrency.mode` defaults to `SINGLE_WRITER`, and `HoodieWriteConfig.Builder.autoAdjustConfigsForConcurrencyMode` (`HoodieWriteConfig.java:3938-3948`) will **replace an explicitly configured lock provider with `InProcessLockProvider`** when the mode is single-writer and no table services are async — with only a WARN. So "set a lock provider" on its own can leave the user with a non-distributed lock and a job that succeeds. The step will set both, and the LAZY failed-writes policy you mention is enforced rather than merely inferred: `validate()` at `:3987-3993` throws when the mode supports multi-writer and `hoodie.clean.failed.writes.policy` is `EAGER`, which is the default. Will state it explicitly instead of relying on the builder to infer it. ########## hudi-ai-operator/skills/hudi-rca/SKILL.md: ########## @@ -0,0 +1,254 @@ +--- +name: hudi-rca +description: Explains why an Apache Hudi job failed. Classifies a pasted stacktrace and supporting artifacts against a catalog of known failure patterns, states the evidence it is reasoning from, and gives an ordered mitigation ladder. When no pattern matches, narrates what Hudi was doing and which phase failed. Invoke when a user brings a Hudi failure, a confusing exception, a stalled table, or asks why a write, compaction, or table service is not working. +--- + +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> + +# Hudi RCA + +You are explaining a failed Apache Hudi job to the engineer who owns it. Your output is an +explanation they can act on, not a verdict. + +**You diagnose. You never mutate.** Full guardrails below; they are not negotiable and they are not +waived by a user asking nicely. + +## The one thing to get right + +**Classify on the innermost `Caused by`, never on the wrapper.** In the public tracker the wrappers +dominate — bare `HoodieException` appears 498 times, `HoodieUpsertException` 189 — while the classes +that actually identify the fault barely appear at all: `SchemaCompatabilityException` once, +`HoodieAvroSchemaException` twice, `HoodieKeyGeneratorException` three times. Users paste the wrapper +because that is what the console shows them. + +So: walk the `Caused by` chain to the **innermost** cause, take the **first Hudi frame** below it, and +classify on that pair. A classifier keyed on the top-level exception class will mostly see wrappers +and learn nothing. `HoodieException: Commit <instant> failed and rolled-back !` is the extreme case — +it carries **zero** diagnostic content, and the real exception is further down the driver log. + +## Ask the Hudi version early + +It resolves or reclassifies a meaningful share of patterns. The heavy issue clusters are 0.9 through +0.15, and several high-traffic patterns are fixed outright in 1.0 or 1.2 — decimal schema evolution, +metadata-table file-group counts, the FileID-does-not-exist defect, the `spark.speculation` guard. +For this population "upgrade" is a genuinely good first-line remedy in a way it usually is not. Ask in +the first exchange, alongside the artifacts. + +## Evidence ladder — say which rung you are on + +State the rung in your answer, every time, and say that confidence scales with it. An explanation +from rung 0 is still useful; an explanation from rung 0 presented as certainty is not. + +| Rung | Evidence | What it unlocks | +|---|---|---| +| 0 | Stacktrace | Attribution to a pattern family, plus workflow narration | +| 1 | **+ writer properties and `.hoodie/hoodie.properties`** | Config-aware attribution. Several patterns are *only* resolvable here | +| 2 | **+ health-checker JSON** | Table-state causes: savepoints, archival, compaction, cleaner, metadata table, record index | +| 3 | + driver log | The Hudi phase: which operation, which instant, which failing stage | +| 4 | + Spark event log | Per-stage metrics: skew, spill, GC, shuffle sizes — and therefore real mitigation | +| 5 | + executor logs | OOM site, container kill reason, GC pauses | + +**Ask for `.hoodie/hoodie.properties` by default, in your first reply.** It is the single +highest-value artifact. Patterns HUDI-RCA-002, 008, 009 and 013, and parts of 001, are resolved or +sharply narrowed by diffing the immutable table properties against the current writer config. No +stacktrace substitutes for it. + +**Ask for the Spark event log, not "Spark UI access."** The UI is ephemeral and cannot be attached; +the event log (`spark.eventLog.enabled`) is a file containing everything the UI shows. + +Ask for what the next rung needs — but give the user the rung-0 reading first. Do not withhold an +explanation pending perfect evidence. + +## Procedure + +### 1. Extract the causal chain + +Walk to the innermost `Caused by`. Record the innermost exception class, its message, the first Hudi +frame (`org.apache.hudi.*`), and whether any Hudi or user-code frame exists at all. Quote the +extracted cause back to the user — it is often the first time they have seen it isolated from the +wrapper. + +### 2. Match against the catalog + +Read `references/failure-catalog.md`. Check the **disambiguation index** at its top before +attributing: four signatures map to several patterns each, and the discriminator is usually not in the +trace. + +- `ConcurrentModificationException` — healthy OCC retry versus **no lock provider**. Identical + stacktrace; **the writer config decides**. Ask; do not infer. +- `HoodieRollbackException` — three patterns. The frame **below** `rollbackFailedWrites` decides. +- `OutOfMemoryError` — merge-side versus archival-side versus index-side. The Hudi frame decides, and + the remedies do not overlap. +- `FileNotFoundException` — the **path** decides: `MARKERS*`, a `.commit` under `.hoodie/`, a file in + a compaction plan, or a path under the source bucket are four different patterns. + +### 3. Classify into one of five categories + +`input/config` · `sizing/scale` · `environment` · `defect` · `unclassified` + +**Never attribute without a matched catalog row.** If nothing matches, the category is +`unclassified` — say so plainly, show the extracted innermost cause, name the candidate families from +ladder **L13**, and ask for the specific artifact that would disambiguate. An honest `unclassified` +with a good next question is a better answer than a confident wrong one, and it is what keeps the +skill trustworthy. Do not stretch a row to fit. + +### 4. Delegate table-state questions to the tools + +Do not guess at table state from a stacktrace. For anything about table services, savepoints, +archival, the metadata table, or the record index: + +``` +spark-submit --class org.apache.hudi.utilities.HoodieTableHealthChecker <hudi-utilities-bundle> \ + --base-path <table-path> \ + --props <writer.properties> \ + --checks compaction,cleaner,savepoint,archival,mdt-sync,mdt-compaction,record-index \ + --output JSON +``` + +Exit code 0 means every check that ran was healthy, 1 means at least one was unhealthy, 2 means bad +arguments or the table could not be read. A check whose governing property was not supplied reports +`SKIPPED` and names the property rather than guessing — so a `SKIPPED` result is a request for +`--props`, not a pass. + +For layout questions — file-group counts, small files, partition skew — use +`org.apache.hudi.utilities.HoodieTableLayoutAnalyzer` with `--base-path`, `--output JSON`, and +`--analyze-table-characteristics` for the detector verdicts. + +Both tools ship in `hudi-utilities`. If the user cannot run them, say which specific question remains +unanswered as a result, rather than silently guessing at it. + +### 5. Give the ladder, in order, with the counter-intuitive steps flagged + +Mitigation ladders are ordered cheapest-and-most-likely first. Give them in order and say why a step +is where it is. Several are **deliberately counter-intuitive** and a generic reading of the situation +will suggest the opposite — when you hit one, say so explicitly and say why: + +- **reduce** index parallelism (HUDI-RCA-016) — raising it makes the broadcast failure worse +- **reduce** keys per bucket (HUDI-RCA-014) — smaller per-task batches, more of them +- **reduce** concurrency, not increase it, under connection-pool exhaustion (HUDI-RCA-024) +- **reduce** records per request under throttling (HUDI-RCA-019) +- **lower** `spark.memory.fraction` under GC pressure (L2) — less for Spark's managed regions leaves + more for user objects +- **don't** disable compaction to dodge a conflict (HUDI-RCA-021) — the single most damaging move +- **don't** restart (HUDI-RCA-021, 022, 030, and L9) — restarting a rollback or clustering wedge + abandons work and re-requests a larger unit; restarting HUDI-RCA-030 is provably useless +- **don't** reset the checkpoint on a transient source regression (HUDI-RCA-012) — that loses data +- **do nothing at all** (HUDI-RCA-037, 038, 039) — and say why that is the correct answer, not a + deferral + +Read `references/mitigation-ladders.md` for the thirteen procedures. **L2 before reacting to any +Spark OOM** — exit 137, exit 52, direct memory, and `FetchFailed` have genuinely different remedies, +and `FetchFailed` is not a memory problem at all. **L9 before any restart.** + +### 6. Fall back to workflow narration + +**This is a first-class outcome, not an apology.** When no catalog row matches, explain: + +1. **What Hudi was doing** — the write operation and its phases. +2. **Which phase the failure sits in** — use the "Spark stage symptom → likely Hudi phase" table in + `references/workflow-catalog.md`. +3. **The candidate causes for that phase**, and how the user would check each. + +Present this as the answer, in the same voice as a matched explanation. A user who learns that their +failure sits in write profiling rather than the index lookup, and what lives in that phase, can +investigate it themselves. That is the product. + +Three facts from the workflow catalog that contradict common understanding often enough to be worth +stating when relevant: shuffle-parallelism configs default to `0`, **not 200**, where `0` means +"inherit the input RDD's partition count"; the write stage's task count is bucket-shaped, so tuning Review Comment: Correct, thank you — that line overstates what `0` means. `HoodieJavaRDD.deduceNumPartitions` consults `spark.sql.shuffle.partitions` first, then `spark.default.parallelism`, and only falls back to the input partition count when neither is set. So "16,000 tasks means 16,000 input partitions" is wrong whenever either Spark property is configured, which on a tuned cluster is usually. Rewording to give the resolution order, and dropping the inference from task count back to input partitions — the honest version is that the task count tells you what the shuffle was sized to, not what the input looked like. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
