lokeshj1703 opened a new pull request, #19550: URL: https://github.com/apache/hudi/pull/19550
### Describe the issue this Pull Request addresses Closes #19549. `hoodie.streamer.sample.writes.enabled` (default true) estimates average record size on the first commit by sampling up to `hoodie.streamer.sample.writes.size` records (default 25000) and shipping them to a single Spark task via `jsc.parallelize(samples, 1)`. When the average record size is large, the serialized task exceeds `spark.rpc.message.maxSize` (default 128 MiB) and estimation fails, falling back to the default record-size estimate. ### Summary and Changelog Bound the sample by total serialized size in addition to record count: - Sampling now runs on the executor via `mapPartitions` on the single coalesced partition and stops once the accumulated serialized size would exceed `0.5 × spark.rpc.message.maxSize`. This keeps both the driver collect and the downstream single-partition write under the RPC frame limit. - At least one record is always retained, so a single oversized record still yields an estimate. - Serialized size is measured with `SerializationUtils.serialize(...).length`, which reflects the bytes actually shipped in the task (for the Avro record type the payload holds `recordBytes`, so the schema is not counted per record). - The record-count cap (`hoodie.streamer.sample.writes.size`) still applies; the byte cap only engages where the previous code would have failed, so behavior is unchanged for tables whose sample already fits under the budget. - No copied code. Added `TestSparkSampleWritesBounding` covering the count bound, the byte-budget boundary, monotonicity of the sample size with the budget, and the always-retain-one guarantee. ### Impact No config or public API change. For large-record tables the first-commit record-size estimate now succeeds instead of silently falling back to the default estimate; subsequent commits already derive record size from real commit metadata. ### Risk Level low Sampling-only change. Existing sample-writes tests are unchanged and new unit tests were added for the bounding logic. ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
