viirya commented on PR #189: URL: https://github.com/apache/arrow-datafusion-comet/pull/189#issuecomment-1988998580
> In that particular place, the batch size is configurable through `spark.comet.columnar.shuffle.batch.size`, which could be different from the batch size in other places. I'm not sure if this works in those cases. Although it is a separate config, I think it should not be too far from the batch size. Actually I'm thinking if we can just use the batch size as the shuffle batch size to simplify them. At least, I don't think we will set shuffle batch size larger than batch size (it doesn't make sense to me for the original purpose of the config). For a shuffle batch size smaller than batch size, there is even no reason to do slicing. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
