xiangfu0 opened a new issue #7510:
URL: https://github.com/apache/pinot/issues/7510


   SparkIngestionJob uses single threaded copy for segments copy from staging 
dir to final deep store dir(e.g. S3):
   
https://github.com/apache/pinot/blob/7ce8b756dff2b153151517afd552759e44d46c75/pinot-plugins/pinot-batch-ingestion/pinot-batch-ingestion-spark/src/main/java/org/apache/pinot/plugin/ingestion/batch/spark/SparkSegmentGenerationJobRunner.java#L345
   
   For a few hundred segments being uploaded, it can take forever. In this 
case, we should consider generating a list of source and destination paths for 
segment copy then use RDD to parallel this job.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to