nsivabalan opened a new pull request, #19128:
URL: https://github.com/apache/hudi/pull/19128

   ### Describe the issue this Pull Request addresses
   
   Today the small-file threshold used by upsert/insert bin-packing is a
   static byte value (`hoodie.parquet.small.file.limit`, default 100 MB)
   with no relationship to `hoodie.parquet.max.file.size` (default 120 MB).
   When operators tune `max.file.size` up or down for a workload, they also
   have to remember to re-tune `small.file.limit` or the two settings drift
   out of proportion — e.g., a 1 GB `max.file.size` still treats anything
   under 100 MB as "small", which is a fraction most operators would not
   choose deliberately.
   
   This PR makes the small-file threshold travel with the max file size by
   expressing it as a fraction.
   
   ### Summary and Changelog
   
   Adds a new config `hoodie.parquet.small.file.fraction.max.file.size`
   (`ConfigProperty<Double>`, `HoodieCompactionConfig`) that becomes the
   single source of truth for the small-file threshold at runtime. An
   `InferFunction` derives the fraction from
   `hoodie.parquet.small.file.limit / hoodie.parquet.max.file.size` when
   the new key is not explicitly set, so existing tables see identical
   behavior without any migration.
   
   Interaction matrix:
   
   | User sets | Effective threshold |
   | --- | --- |
   | Neither config | `100 MB / 120 MB ≈ 0.833` × max = 100 MB (bit-identical 
to today) |
   | Only `small.file.limit` | inferred `user_bytes / max_size` × max = 
user_bytes |
   | Only `max.file.size` (e.g. 1 GB) | inferred `100 MB / 1 GB ≈ 0.098` × 1 GB 
≈ 100 MB (ratio auto-scales) |
   | Only new fraction | user fraction × max |
   | Both | fraction wins; byte-limit ignored |
   | `small.file.limit=0` (disable idiom) | inferred 0.0 → effective threshold 
0 → every `<= 0` disable check trips as before |
   
   **Files:**
   
   - `HoodieCompactionConfig.java` — new
     `PARQUET_SMALL_FILE_FRACTION_MAX_FILE_SIZE` config with the infer
     function. Existing `PARQUET_SMALL_FILE_LIMIT` doc updated to note its
     new role as infer input.
   - `HoodieWriteConfig.java` — `getParquetSmallFileLimit()` now returns
     `long` and computes `(long) (fraction * getParquetMaxFileSize())`.
     This is the single behavior-change site. All 10 existing call sites
     (Spark/Java `UpsertPartitioner`, `SparkUpsertDeltaCommitPartitioner`,
     Flink `WriteProfile`, `AverageRecordSizeEstimator`) already accept
     `long`, so no source edits at any call site.
   - `TestHoodieWriteConfig.java` — new
     `testParquetSmallFileFractionDerivation` covering: default
     back-compat, byte-limit-only, fraction-only, both-set precedence, the
     disable sentinel, and a custom `max.file.size`.
   
   ### Impact
   
   - **User-facing configs**: 1 new config
     (`hoodie.parquet.small.file.fraction.max.file.size`, default `0.0`
     with infer). The existing `hoodie.parquet.small.file.limit` is not
     deprecated — it becomes the infer input, so setting it alone still
     works. No user has to change anything to keep today's behavior.
   - **Public API**: `HoodieWriteConfig.getParquetSmallFileLimit()` return
     type widens from `int` to `long`. Source-compatible at every known
     call site (all compare against `long` file sizes or multiply into
     `long`). Binary-compat is a re-compile away for external callers.
   - **Performance**: none. The accessor does one extra double-to-long
     multiplication per call — negligible vs. the file-listing work at
     every call site.
   
   ### Risk Level
   
   low.
   
   Verification: full `hudi-client-common` module test suite green
   (1088/1088 tests, 0 failures, 0 errors), including 41/41
   `TestHoodieWriteConfig` (of which 6 are the new fraction-derivation
   cases). `hudi-client-common`, `hudi-spark-client`, `hudi-java-client`,
   `hudi-flink-datasource/hudi-flink` all compile clean with the return-type
   widening.
   
   ### Documentation Update
   
   - Inline config documentation on `PARQUET_SMALL_FILE_FRACTION_MAX_FILE_SIZE`
     and updated docstring on `PARQUET_SMALL_FILE_LIMIT` — covered in the
     patch itself via `@ConfigProperty` `withDocumentation(...)`.
   - No user-facing website change needed; both configs are marked
     `markAdvanced()` and appear in the generated config reference.
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to