nsivabalan opened a new pull request, #19230: URL: https://github.com/apache/hudi/pull/19230
### Change Logs Revamps the `TableSizeStats` Spark utility from a logs-only file-size dumper into a structured forensic tool. Extracts the small-file classification logic into a new public `SmallFileDetector` utility so external callers can score a table without invoking the full Spark utility. #### Bug fix (motivating change) The previous per-partition `FileSystemView` was always built with `metadataConfig.enable(false)` regardless of whether the table had MDT enabled — forcing a full filesystem listing per partition. On large object-store tables this is a 10-100× slowdown. The view is now built once, reused across partitions, and inherits the table's actual MDT status. #### New output | Capability | Flag | |---|---| | Output format | `--output TABLE\|JSON` (default `TABLE`) | | Per-partition rows sorted by `totalBytes` desc | (always, when partition stats are on) | | Cap partition rows to top N largest | `--top-n N` | | Per-partition file-count distribution | (always) | | Skew metrics (CV, Gini, top-N share, outliers) | (with `--enable-table-stats`) | | Row counts (MDT col-stats fast path → Parquet footer fallback) | `--include-row-counts` | | Table characteristics audit | `--analyze-table-characteristics` | #### `--analyze-table-characteristics` detectors - **Micro-partitioned (table-level verdict)** — `numPartitions > 10000` OR partitions with `fileCount >= 25 AND avgFileSize < 50 MB`. The size rule only applies to tables ≥ 30 days old. - **Small-file pile-up (table-level verdict via prevalence)** — verdict tier `CLEAN` / `MODERATE` / `SEVERE` / `SKIPPED` based on the fraction of qualifying partitions (`fileCount >= 5`) whose `avgFileSize < 50 MB`. `MODERATE` at ≥10%, `SEVERE` at ≥30%. `SKIPPED` when the table has fewer than 10 ingest commits (active + archived). - **Hot partitions (per-partition)** — scans the last N completed ingest commits (`commit` + `deltacommit` + INSERT_OVERWRITE `replacecommit`, **excluding** `COMPACT` and `CLUSTER`), aggregates per partition by commit count + bytes + records, and flags partitions appearing in ≥50% of the window. All thresholds are individually overridable via CLI flags. See `TableSizeStats.md` for the full runbook. #### New public API — `org.apache.hudi.utilities.smallfile.SmallFileDetector` - **Stateless.** Everything is `public static`; the detector reads all state through parameters, so callers can invoke individual pieces (`classifyVerdict` alone, `countTotalIngestCommits` alone) without going through `run()`. - Immutable `Config` value object with `equals`/`hashCode`. `Config.Builder.build()` throws `IllegalArgumentException` when `moderatePct > severePct`. - `Verdict` enum, `PartitionStats` input, `Result` output. ### Impact - **New**: `TableSizeStats` gains 4 new detectors + structured output. New public utility class `SmallFileDetector` for programmatic reuse. - **Perf**: 10-100× speedup on large object-store tables with MDT (per-partition FS listing fix). - **No breaking changes**: no `@PublicAPIClass` / `@PublicAPIMethod` touched, no storage format changes, no new `hoodie.*` configs. Default `LOG.info`-style output is replaced with a structured printf table. ### Risk level **low** — no changes to existing configs, storage format, or public APIs. ### Documentation Update - New file: `hudi-utilities/src/main/java/org/apache/hudi/utilities/TableSizeStats.md` — full runbook (classpath, flag reference, JSON schema, caveats, examples). - Class-level Javadoc updated on `TableSizeStats` and new `SmallFileDetector`. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Change Logs and Impact were stated clearly - [x] Adequate tests were added if applicable - [x] CI passed locally: `mvn -pl hudi-utilities -Dspark3.5 -o test -Dtest=TestTableSizeStats,TestSmallFileDetector` → **53/53 passing** (30 + 23) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
