JonSnow1807 opened a new pull request, #55437: URL: https://github.com/apache/arrow/pull/55437
Closes #55428. ### Rationale for this change A ranged GetObject response currently writes into a buffer sized to the requested read. An S3 error document can be larger than that buffer, even when the requested object is small. When this happens, the response stream fails before the AWS SDK can classify the error. A SlowDown response becomes INTERNAL_FAILURE with “Failed to flush response stream”, preventing the expected retry behavior. ### What changes are included in this PR? Introduce a GET-specific response stream that writes successful reads directly into the caller’s buffer. If a response exceeds that buffer, the stream retains a prefix of up to 64 KiB for SDK error parsing and consumes subsequent bytes without growing the retained buffer. This lets the SDK classify and retry throttling responses. After a successful SDK result, check whether the response overflowed. Oversized successful responses return an IOError instead of being accepted as a valid ranged read. The upload stream is unchanged. ### Are these changes tested? Six new C++ tests cover normal and short writes, bulk and split overflow, character writes, direct put-area writes, rewinding, spill-cap boundaries, and overflow detection. Eleven Python integration cases use a local HTTP server to cover: - HTTP 429 and 503 SlowDown responses larger than the requested range. - A control case where the error body fits. - Recovery after throttling, verifying the returned bytes. - Oversized HTTP 200 and 206 responses, with Content-Length and chunked transfer encoding. - Retry behavior for error bodies larger than the 64 KiB retained prefix. The two small-range regression cases fail against the original source library, while the fitting control passes. All eleven integration cases pass with the patch. The complete C++ S3 suite passes with 81 tests passed and two skipped. Validation used Linux, the curl HTTP backend, and AWS SDK C++ 1.11.778. Actual Windows and CRT HTTP backends were not run; direct put-area behavior is covered by the C++ tests. ### Are there any user-facing changes? Small ranged reads can now retain the SDK’s throttling classification and retry behavior when the error body exceeds the requested range. Oversized successful responses produce an explicit IOError. There are no public API changes. Error documents larger than 64 KiB may lose XML details and rely on HTTP-status classification. The retained buffer is bounded, but consuming the response still takes network time. ### Was AI used for this PR? AI tools assisted with implementation, regression tests, independent review, and validation. I reviewed the code, tests, and description before submission. **PR code and description written by:** - [x] Human - [x] AI **Reviewed before submission by:** - [x] Human - [x] AI - [ ] Not reviewed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
