seittema commented on a change in pull request #11594:
URL: https://github.com/apache/arrow/pull/11594#discussion_r741961642
##########
File path: cpp/src/arrow/filesystem/s3fs.cc
##########
@@ -563,6 +566,97 @@ class S3Client : public Aws::S3::S3Client {
req.SetBucket(ToAwsString(bucket));
return GetBucketRegion(req);
}
+
+ S3Model::CompleteMultipartUploadOutcome
CompleteMultipartUploadWithErrorFixup(
+ S3Model::CompleteMultipartUploadRequest&& request) const {
+ // CompletedMultipartUpload can return a 200 OK response with an error
+ // encoded in the response body, in which case we should either retry
+ // or propagate the error to the user (see
+ //
https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html).
+ //
+ // Unfortunately the AWS SDK doesn't detect such situations but lets them
+ // return successfully (see https://github.com/aws/aws-sdk-cpp/issues/658).
+ //
+ // We work around the issue by registering a DataReceivedEventHandler
+ // which parses the XML response for embedded errors.
+
+ util::optional<AWSError<Aws::Client::CoreErrors>> aws_error;
+
+ auto handler = [&](const Aws::Http::HttpRequest* http_req,
+ Aws::Http::HttpResponse* http_resp,
+ long long) { // NOLINT runtime/int
+ auto& stream = http_resp->GetResponseBody();
+ const auto pos = stream.tellg();
+ const auto doc =
Aws::Utils::Xml::XmlDocument::CreateFromXmlStream(stream);
Review comment:
I know it's possible to get a partial XML response from S3 due to
network failure, does this handle that and throw a 5xx/retry or will an XML
error be thrown back up?
https://github.com/boto/botocore/blob/04d1fae43b657952e49b21d16daa86378ddb4253/botocore/handlers.py#L141
Per AWS:
```
# In cases of network disruptions, we may end up with a partial
# streamed response from S3. We need to treat these cases as
# 500 Service Errors and try again.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
