westonpace commented on pull request #11084: URL: https://github.com/apache/arrow/pull/11084#issuecomment-915681730
I've moved this back to ready for review. AsyncTaskGroup is similar but not the same as the TaskGroups. SerialTaskGroup and ThreadedTaskGroup take care of scheduling the task (submitting it to the executor) as well as tracking the task. The AsyncTaskGroup is only responsible for tracking a set of tasks that were scheduled elsewhere. ThreadedTaskGroup could maybe use AsyncTaskGroup through composition but that could be done elsewhere. The scheduling in the dataset writer case happens in the file queue which doesn't quite fit the SerialTaskGroup / ThreadedTaskGroup model because each task should run serially (like SerialTaskGroup) but asynchronously (like ThreadedTaskGroup). I could potentially extract the logic in DatasetWriterFileQueue into a AsyncSerialTaskGroup but for the moment I think what this PR and the dataset writer PR proposes is fine. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
