gianm commented on PR #14677: URL: https://github.com/apache/druid/pull/14677#issuecomment-1656047741
> Does this mean that for cloud input sources, if we don't specify a split hint spec explicitly, each file would correspond to a single split? Couldn't that potentially cause a large skew between the amount of work each batch ingestion task does if the files differ a lot in size? Ah, no, because the code here runs on worker tasks after the splits have already been made by the main task. (Or on the Overlord to do sampling, where there is no parallelism anyway.) It is using `createSplits` simply as a way of getting the list of objects out of `prefixes`, `objects`, or `uris`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
