Vamsi-klu opened a new pull request, #19082: URL: https://github.com/apache/pinot/pull/19082
## Why Batch Spark/Hadoop segment generation jobs were spending a large share of wall-clock time on a **single-threaded** `PinotFS` copy/move of segment tars from the staging directory to deep store (S3/HDFS/…). For jobs that produce hundreds of segments, that serial hop dominated the SLA even after segment *build* and controller *push* were already parallelized. This is the remaining bottleneck called out in #7510 (staging→deep-store copy), not tar push (which already supports `pushParallelism`). ## Impact - **Faster batch ingestion jobs** when staging is used (default path for Hadoop; optional for Spark-3). - **Lower time-to-deep-store** for large segment counts on object stores where each `move` is a remote copy+delete. - **No change** to segment contents, naming, overwrite semantics, or controller upload protocol. - Default parallelism is bounded (not “one thread per segment”) to avoid thrashing remote FS connection pools. ## How - Extended `SegmentGenerationJobUtils.moveFiles` with a bounded thread-pool parallel implementation and an optional parallelism argument. - Wired Spark-3 and Hadoop generation runners to pass configurable parallelism (via job extra configs / sensible default). - Preserved existing overwrite / relative-path layout behavior; aggregate failures so a single failed move fails the job cleanly. ## Test plan - [x] Unit tests in `SegmentGenerationJobUtilsTest` covering parallel vs serial layout equivalence, overwrite true/false, and failure aggregation (`LocalPinotFS`). - [ ] `./mvnw -pl pinot-plugins/pinot-batch-ingestion/pinot-batch-ingestion-common -am -Dtest=SegmentGenerationJobUtilsTest -Dsurefire.failIfNoSpecifiedTests=false test` - [ ] Optional: Spark/Hadoop generation job tests if staging path is covered in module suites. ## Related closes: #7510 ## Reviewers Suggested: xiangfu0, kkrugler (issue discussion / batch ingestion) --- ##### Was generative AI tooling used to co-author this PR? - [x] Yes — Grok Build (xAI) Generated-by: Grok Build (xAI) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
