westonpace commented on a change in pull request #9678:
URL: https://github.com/apache/arrow/pull/9678#discussion_r594965503
##########
File path: cpp/src/arrow/filesystem/s3fs.cc
##########
@@ -1537,16 +1568,14 @@ class S3FileSystem::Impl {
}
req.SetBucket(ToAwsString(bucket));
req.SetDelete(std::move(del));
- delete_handlers.emplace_back();
- futures.push_back(&delete_handlers.back().future);
- client_->DeleteObjectsAsync(req, delete_handlers.back());
+ ARROW_ASSIGN_OR_RAISE(auto fut, io_context_.executor()->Submit(
+ io_context_.stop_token(), [client,
req]() {
+ return client->DeleteObjects(req);
+ }));
+ futures.push_back(std::move(fut).Then(delete_cb));
}
- WaitForAll(futures);
- for (const auto* fut : futures) {
- RETURN_NOT_OK(fut->status());
- }
- return Status::OK();
+ return AllComplete(futures).status();
Review comment:
Blocking at the end sort of defeats the purpose but it can be a fine
first step. If I do something like this I'll rename the method
DeleteObjectsAsync that returns `Future<>` and then add back in `DeleteObjects`
which calls `DeleteObjectsAsync` and waits on the status. This ensures
backwards compatibility but makes it even easier to consume the async version
in the future.
##########
File path: cpp/src/arrow/filesystem/s3fs.cc
##########
@@ -1035,6 +1025,26 @@ class ObjectOutputStream final : public io::OutputStream
{
return Status::OK();
}
+ static void HandleUploadOutcome(const std::shared_ptr<UploadState>& state,
+ int part_number, const
S3Model::UploadPartRequest& req,
+ const Result<S3Model::UploadPartOutcome>&
result) {
+ std::unique_lock<std::mutex> lock(state->mutex);
+ if (!result.ok()) {
+ state->status &= result.status();
+ } else {
+ const auto& outcome = *result;
+ if (!outcome.IsSuccess()) {
+ state->status &= UploadPartError(req, outcome);
+ } else {
+ AddCompletedPart(state, part_number, outcome.GetResult());
+ }
+ }
+ // Notify completion, regardless of success / error status
+ if (--state->parts_in_progress == 0) {
Review comment:
Similar to the other comment, if you use `Flush()` it blocks. You could
create a `FlushAsync` and update the code here so that instead of just
`notify_all()` it marked a future finished. Or you could use a threaded task
group which already has this logic.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
