nsivabalan opened a new pull request, #20153: URL: https://github.com/apache/hudi/pull/20153
> **Stacked on #20151.** This branch includes that PR's commit because the check plugs into the > framework it introduces. Review only the last commit > (`feat(utilities): add record-level index sizing check ...`); the diff collapses to ~1,000 lines > once #20151 merges and this is rebased. ### Describe the issue this Pull Request addresses Part of #19262 under the Agentic Lakehouse umbrella #19256; plan in https://github.com/apache/hudi/issues/19262#issuecomment-5750570015. The number of file groups in a record-level index is decided once, when the index is initialized, from the record count at that moment and the writer's sizing configuration. It never changes afterwards. A table that has grown well past what its index was sized for has index file groups far larger than the writer intended: every index lookup reads a bigger file and every index write lands on the same few file groups. Nothing reports this today; operators find out from lookup latency. ### Summary and Changelog Adds `RecordIndexSizingCheck` (`--checks record-index`) to `HoodieTableHealthChecker`. It re-runs the writer's own sizing estimate — `HoodieTableMetadataUtil.estimateFileGroupCount`, the same function `BaseRecordIndexer` calls at initialization — against the index as it is now, and compares the result with the file-group count the index actually has. It also looks at the largest file group directly, since a group past the configured maximum is the symptom regardless of how the count came out. - **Global index:** compares the whole index's file-group count against the ideal. - **Partitioned index** (`hoodie.metadata.record.level.index.enable=true`): the writer sizes per data partition, so the check compares per data partition (via `getBucketizedFileGroupsForPartitionedRLI`) and reports the worst one. - **Layout mismatch:** if the writer configuration says partitioned but the index on storage is global (or the reverse, detected from file-group ids), the count comparison is skipped with the reason echoed; the size rule still runs. - **Verdict:** `UNHEALTHY` when observed file groups are below half the ideal, or when the largest file group exceeds 1.5x `hoodie.metadata.record.index.max.filegroup.size`. Both constants are documented with their rationale (the ideal already carries the growth factor; log files push a group past the maximum transiently between metadata compactions). - **Remedy:** the finding states that the count is fixed at initialization and names the actual commands to re-bootstrap — `metadata delete-record-index` in `hudi-cli` or `HoodieIndexer --mode dropindex`, then `HoodieIndexer --mode scheduleAndExecute --index-types RECORD_INDEX` — and that this is disruptive and not undoable in place. - **Record count** is estimated from the index's own bytes on storage divided by the 48-byte average record the writer sizes with; no data file is scanned. The estimate runs low (index files are compressed), so the count rule under-alerts rather than over-alerts, and the report labels it an estimate. Engine-agnostic: `HoodieBackedTableMetadata` with `HoodieLocalEngineContext`, closed per run. The check follows the fail-fast contract from #20151 — the RLI sizing properties must be supplied (old alternative key spellings are accepted) or `--apply-all-defaults` chosen — and echoes every value it reasoned with under the `observed.*` / `effective.*` convention. Registration is three one-line edits (`allChecks()`, `--checks` help, test count) plus a runbook section. No code copied from elsewhere. **Tests:** `TestRecordIndexSizingCheck` (no Spark: no index, missing keys, alternative keys, degenerate max size) and `TestRecordIndexSizingCheckWithSpark` (real global and partitioned RLI fixtures: matching sizing, undersized, boundary at exactly half, oversized file group, layout mismatch both directions, `--apply-all-defaults`). 24 tests. ### Impact Additive. No public API or storage changes; read-only. No new `hoodie.*` configs. The runbook's "no Spark session required" statement remains true. ### Risk Level none ### Documentation Update `### record-index` section added to `HoodieTableHealthChecker.md`, including the partitioned-index and layout-mismatch behavior. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
