nsivabalan opened a new pull request, #20152: URL: https://github.com/apache/hudi/pull/20152
### Describe the issue this Pull Request addresses Part of #19262 (Hudi-aware agent skills and tools), under the Agentic Lakehouse umbrella #19256; the overall plan is in https://github.com/apache/hudi/issues/19262#issuecomment-5750570015. This is the "is the data laid out well?" tool; #20151 covers "are table services keeping up?". Operators have no single way to tell whether a table's layout has gone bad — too many tiny files, partitions too fine to be useful, or a handful of partitions absorbing every write. The existing `TableSizeStats` prints size histograms to the log, which is useful for a human eyeballing one table but gives nothing an operator can threshold on or an agent can read. ### Summary and Changelog Adds `HoodieTableLayoutAnalyzer`, a Spark utility that walks a table's partitions and reports per-partition size and file-count distributions, skew metrics, optional row counts, and — under `--analyze-table-characteristics` — three detectors with explicit verdicts: - **micro-partition** (table-level, `CLEAN | FLAGGED | SKIPPED`): partition count above a threshold, or partitions holding many files that average well under the target file size. The size rule only applies once the table is old enough for it to mean something. - **small-files** (`CLEAN | MODERATE | SEVERE | SKIPPED`): the fraction of qualifying partitions whose files average under the small-file threshold. Skipped when the table has too few ingest commits for the signal to be reliable. - **hot-partitions** (per-partition): partitions written by at least half of the last N ingest commits, with compaction and clustering excluded from the window. Every threshold is a CLI flag with a documented default. Output is a human-readable table or `--output JSON`; the JSON carries, per detector, a `status`, a one-line `summary`, `findings[]` written as actionable sentences, and `effectiveConfigs{}` where `effective.*` keys are the thresholds actually applied and `observed.*` keys are measurements — the same envelope convention as `HoodieTableHealthChecker` in #20151, so one skill or agent can read both. Row counts (`--include-row-counts`) use metadata-table column stats when available and fall back to Parquet footers only for files the stats did not cover. **Why a new class rather than extending `TableSizeStats`.** `TableSizeStats` is a long-standing `spark-submit` entry point whose log output someone may be scraping; changing it to structured output would be a silent behavior change. And "size stats" no longer describes a tool with detectors, verdicts, and skew analysis. So the analyzer is additive: `TableSizeStats` and its tests are untouched, and whether to deprecate it later is a separate conversation. The analyzer builds on the same partition-walking and histogram approach but is self-contained; no shared helpers were extracted. The per-table `FileSystemView` is built once and honors the table's metadata-table setting, so on large object-store tables partitions are listed from the metadata table instead of one storage listing per partition. Both the metadata reader and the view are closed per table, which matters under `--props-path` batch runs over many tables. **Docs** — `HoodieTableLayoutAnalyzer.md`: invocation, every flag, each detector's rule, the JSON schema, and caveats (table age and ingest-commit counts are lower bounds; log files are not counted toward small-file sizing on Merge-on-Read). **Skill** — `hudi-agent-gateway/skills/hudi-table-layout/`, following the layout of `hudi-architect` (#19380) and `hudi-table-health` (#20151): `SKILL.md` drives collecting the base path and thresholds, running with `--output JSON`, and turning each detector's verdict into a recommendation (clustering, small-file config, partitioning revisit) without acting on the table. **Tests** — `TestHoodieTableLayoutAnalyzer`, 31 tests on `HoodieSparkClientTestBase`: each detector's verdict at and around its thresholds, the skipped cases, hot-partition record accounting (per-operation counters, not `numWrites`), JSON escaping, and the JSON envelope shape. ### Impact Purely additive. New class, new test, new runbook, new skill directory. - No public API changes; no `@PublicAPIClass` / `@PublicAPIMethod` touched. - No storage format changes. Read-only; never modifies a table. - **No new `hoodie.*` configs.** Thresholds are CLI flags on this tool only. - `TableSizeStats` is not modified. ### Risk Level none Read-only tool on a new code path, reachable only by explicit invocation. On size: this is 3.5k lines, of which ~1.1k are tests and ~0.6k are docs and the skill. It is one self-contained tool; the only clean split would be moving the skill directory to a follow-up. Happy to do that if reviewers prefer. ### Documentation Update `HoodieTableLayoutAnalyzer.md` is added alongside the tool. No new configs. Happy to add a website page if reviewers would like one. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
