lokeshj1703 opened a new pull request, #19550: URL: https://github.com/apache/hudi/pull/19550
### Describe the issue this Pull Request addresses Closes #19549. Record-size estimation on the first commit samples incoming records into a shadow table and reads back the average size. The sampler collected the sample to the driver and re-parallelized it into one task (`jsc.parallelize(samples, 1)`), so every sampled record crossed the driver: the collect is bounded by `spark.driver.maxResultSize` and the single-task re-parallelize by `spark.rpc.message.maxSize`. For large-record tables the sample exceeds those limits and the estimation round fails. ### Summary and Changelog Hand the sample to `bulkInsert` as a persisted single-partition RDD instead of collecting it to the driver and re-parallelizing. The records move through the block manager rather than a driver round trip, so the sample is no longer bounded by `spark.driver.maxResultSize` or `spark.rpc.message.maxSize` at any record size. `persist` pins the sample because it is read more than once (isEmpty, bulkInsert, commit). The record-count cap (`hoodie.streamer.sample.writes.size`) still applies. No copied code. `TestSampleWritesDriverBudget` is added as a regression guard: it pins `spark.driver.maxResultSize` below the sample size and asserts the estimate is still produced, so it fails if the sampler ever routes the sample through the driver again. ### Impact No config or public API change. For large-record tables the first-commit record-size estimate no longer risks failing the estimation round; subsequent commits already derive record size from real commit metadata. ### Risk Level low Sampling-only change on the first-commit estimation path. The existing `TestSparkSampleWritesUtils` suite is unchanged and passes; the new regression test guards the driver-transfer behavior. ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
