nsivabalan opened a new pull request, #19128:
URL: https://github.com/apache/hudi/pull/19128
### Describe the issue this Pull Request addresses
Today the small-file threshold used by upsert/insert bin-packing is a
static byte value (`hoodie.parquet.small.file.limit`, default 100 MB)
with no relationship to `hoodie.parquet.max.file.size` (default 120 MB).
When operators tune `max.file.size` up or down for a workload, they also
have to remember to re-tune `small.file.limit` or the two settings drift
out of proportion — e.g., a 1 GB `max.file.size` still treats anything
under 100 MB as "small", which is a fraction most operators would not
choose deliberately.
This PR makes the small-file threshold travel with the max file size by
expressing it as a fraction.
### Summary and Changelog
Adds a new config `hoodie.parquet.small.file.fraction.max.file.size`
(`ConfigProperty<Double>`, `HoodieCompactionConfig`) that becomes the
single source of truth for the small-file threshold at runtime. An
`InferFunction` derives the fraction from
`hoodie.parquet.small.file.limit / hoodie.parquet.max.file.size` when
the new key is not explicitly set, so existing tables see identical
behavior without any migration.
Interaction matrix:
| User sets | Effective threshold |
| --- | --- |
| Neither config | `100 MB / 120 MB ≈ 0.833` × max = 100 MB (bit-identical
to today) |
| Only `small.file.limit` | inferred `user_bytes / max_size` × max =
user_bytes |
| Only `max.file.size` (e.g. 1 GB) | inferred `100 MB / 1 GB ≈ 0.098` × 1 GB
≈ 100 MB (ratio auto-scales) |
| Only new fraction | user fraction × max |
| Both | fraction wins; byte-limit ignored |
| `small.file.limit=0` (disable idiom) | inferred 0.0 → effective threshold
0 → every `<= 0` disable check trips as before |
**Files:**
- `HoodieCompactionConfig.java` — new
`PARQUET_SMALL_FILE_FRACTION_MAX_FILE_SIZE` config with the infer
function. Existing `PARQUET_SMALL_FILE_LIMIT` doc updated to note its
new role as infer input.
- `HoodieWriteConfig.java` — `getParquetSmallFileLimit()` now returns
`long` and computes `(long) (fraction * getParquetMaxFileSize())`.
This is the single behavior-change site. All 10 existing call sites
(Spark/Java `UpsertPartitioner`, `SparkUpsertDeltaCommitPartitioner`,
Flink `WriteProfile`, `AverageRecordSizeEstimator`) already accept
`long`, so no source edits at any call site.
- `TestHoodieWriteConfig.java` — new
`testParquetSmallFileFractionDerivation` covering: default
back-compat, byte-limit-only, fraction-only, both-set precedence, the
disable sentinel, and a custom `max.file.size`.
### Impact
- **User-facing configs**: 1 new config
(`hoodie.parquet.small.file.fraction.max.file.size`, default `0.0`
with infer). The existing `hoodie.parquet.small.file.limit` is not
deprecated — it becomes the infer input, so setting it alone still
works. No user has to change anything to keep today's behavior.
- **Public API**: `HoodieWriteConfig.getParquetSmallFileLimit()` return
type widens from `int` to `long`. Source-compatible at every known
call site (all compare against `long` file sizes or multiply into
`long`). Binary-compat is a re-compile away for external callers.
- **Performance**: none. The accessor does one extra double-to-long
multiplication per call — negligible vs. the file-listing work at
every call site.
### Risk Level
low.
Verification: full `hudi-client-common` module test suite green
(1088/1088 tests, 0 failures, 0 errors), including 41/41
`TestHoodieWriteConfig` (of which 6 are the new fraction-derivation
cases). `hudi-client-common`, `hudi-spark-client`, `hudi-java-client`,
`hudi-flink-datasource/hudi-flink` all compile clean with the return-type
widening.
### Documentation Update
- Inline config documentation on `PARQUET_SMALL_FILE_FRACTION_MAX_FILE_SIZE`
and updated docstring on `PARQUET_SMALL_FILE_LIMIT` — covered in the
patch itself via `@ConfigProperty` `withDocumentation(...)`.
- No user-facing website change needed; both configs are marked
`markAdvanced()` and appear in the generated config reference.
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]