nsivabalan opened a new pull request, #18354:
URL: https://github.com/apache/hudi/pull/18354
### Describe the issue this Pull Request addresses
RLI (Record Level Index) initialization for large tables can be slow
because it counts all records to estimate the optimal file group count. For
tables
with thousands of file slices, this full record count operation adds
significant latency during bootstrap.
This PR optimizes RLI initialization by decoupling file group estimation
from actual record generation and using size-weighted sampling to estimate
total record count, reducing initialization time by ~10x for large tables.
### Summary and Changelog
**User Benefit:** Significantly faster RLI initialization for large tables
(10x improvement for tables with 1000+ file slices) while maintaining
accurate file group sizing.
**Detailed Changes:**
1. **Decoupled file group estimation from record generation**
- File group count is now estimated separately before reading all
records
- Estimation no longer requires constructing RLI records - just counts
2. **Implemented size-weighted sampling for record count estimation**
- Samples 10% of file slices to estimate total record count
- Uses base file sizes as weights for accurate extrapolation
- Calculates records-per-byte ratio from sample and applies to total
size
- Returns `Pair<recordCount, actualSampledSize>` to handle filtered
file slices
3. **Smart threshold-based sampling**
- Skips sampling for small file slice counts (≤10 for partitioned RLI,
≤50 for global RLI)
- Only applies sampling when min != max file group count (dynamic
sizing expected)
- Uses exact count for small datasets, sampling only for large ones
4. **Optimized for MOR tables**
- Only reads base files for estimation (skips log files)
- Lightweight counting using
`HoodieFileGroupReader.getClosableKeyIterator()`
- No RLI record construction during estimation phase
5. **Added `sum()` API to HoodieData**
- New terminal operation to sum Long elements in collections
- Implemented in both `HoodieListData` and `HoodieJavaRDD`
- Used for efficient aggregation in sampling logic
6. **Code cleanup**
- Extracted `getRLIFileGroupCountBounds()` helper to eliminate
duplication
- Updated `estimateFileGroupCountBySampling()` to accept min/max as
parameters
- Improved testability and maintainability
7. **Comprehensive test coverage**
- `testRecordIndexSamplingBasedEstimation`: Validates sampling with min
!= max
- `testRecordIndexWithFixedFileGroupCount`: Validates bypass when min
== max
- `testRecordIndexSamplingWithLargerDataset`: Validates scaling with
data size
- `testGlobalRecordIndexSamplingBasedEstimation`: Validates global RLI
with higher thresholds
- All tests parameterized for both CoW and MoR table types
### Impact
**Performance:**
- ~10x faster RLI initialization for large tables (1000+ file slices)
- Small datasets see minimal difference (direct counting is already fast)
- Memory usage reduced during estimation phase (no RLI record construction)
**Behavior:**
- No user-facing behavior changes
- File group sizing remains accurate with size-weighted sampling
- Respects existing min/max file group count configurations
- Only applies optimization when min != max (user can disable by setting
min == max)
**API:**
- Added `sum()` method to `HoodieData` interface (new public API)
### Risk Level
**Low**
**Rationale:**
- Optimization only applies when min != max file group count (dynamic
sizing)
- Users with min == max bypass sampling entirely (existing behavior
preserved)
- Sampling uses base file sizes as weights, ensuring accurate estimation
- Small datasets skip sampling (threshold-based), using exact counts
- Comprehensive test coverage for both sampling and non-sampling paths
- No changes to storage format or existing public APIs (except additive
`sum()`)
**Verification:**
- Unit tests validate sampling logic, threshold behavior, and file group
sizing
- Integration tests verify RLI functionality after sampling-based
initialization
- Tests cover both partitioned and global RLI with CoW and MoR tables
- Manual testing can be done with large tables to verify performance gains
### Documentation Update
<!-- Describe any necessary documentation update if there is any new
feature, config, or user-facing change. If not, put "none".
- The config description must be updated if new configs are added or the
default value of the configs are changed.
- Any new feature or user-facing change requires updating the Hudi website.
Please follow the
[instruction](https://hudi.apache.org/contribute/developer-setup#website)
to make changes to the website. -->
### Contributor's checklist
- [ ] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Enough context is provided in the sections above
- [ ] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
