ava-windsorb opened a new issue, #55428: URL: https://github.com/apache/arrow/issues/55428
### Describe the bug, including details regarding any error messages, version, and platform. When Arrow’s S3 client does a ranged `GetObject`, the response stream buffer is sized from the declared read length. If the server returns an error XML body (e.g. throttling SlowDown on HTTP 429/503) that is longer than that buffer, the AWS C++ SDK ends up reporting something like ` INTERNAL_FAILURE / Failed to flush response stream` instead of surfacing the XML error and retry policy—so AwsStandardS3RetryStrategy does not see SLOW_DOWN. Repro: minimal fake S3 over HTTP: HEAD returns a small Content-Length; GET returns a 429 + throttling XML longer than the small range read. Observed with PyArrow S3FileSystem + AwsStandardS3RetryStrategy. Related context: [Doris #66557](https://github.com/apache/doris/pull/66557) discusses the same class of buffer/sizing vs error-body behavior. reproduced: ```python """PyArrow S3 GetObject uses a fixed buffer; S3 error XML can overflow it. Arrow writes the full HTTP body into a buffer sized to the ranged read. If the body is an error document larger than that buffer, the AWS SDK reports ``INTERNAL_FAILURE`` / ``Failed to flush response stream`` and does not retry. Real S3 throttling often uses HTTP 503 + SlowDown XML; this fake server uses HTTP 429 with the same XML shape. See https://github.com/apache/doris/pull/66557. """ import threading from contextlib import contextmanager from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer import pytest from pyarrow.fs import AwsStandardS3RetryStrategy, S3FileSystem _OBJECT = "repro-bucket/tiny.bin" _ERR_XML = ( b"<?xml version='1.0'?><Error><Code>SlowDown</Code>" b"<Message>Please reduce your request rate.</Message>" b"<RequestId>rid</RequestId></Error>" ) _FLUSH = "Failed to flush response stream (eof: 0, bad: 1)" @contextmanager def _fake_s3(declared_size: int): ranges: list[str | None] = [] class Handler(BaseHTTPRequestHandler): protocol_version = "HTTP/1.1" def do_HEAD(self): self.send_response(200) self.send_header("Content-Length", str(declared_size)) self.end_headers() def do_GET(self): if "tiny.bin" not in self.path: self.send_error(404) return ranges.append(self.headers.get("Range")) self.send_response(429) self.send_header("Content-Type", "application/xml") self.send_header("Content-Length", str(len(_ERR_XML))) self.end_headers() self.wfile.write(_ERR_XML) def log_message(self, *_args): pass # Threading: client may issue concurrent HEAD + GET (single-thread hangs). server = ThreadingHTTPServer(("127.0.0.1", 0), Handler) threading.Thread(target=server.serve_forever, daemon=True).start() try: host, port = server.server_address yield f"{host}:{port}", ranges finally: server.shutdown() server.server_close() def _read(endpoint: str) -> None: fs = S3FileSystem( access_key="test", secret_key="test", region="us-west-2", scheme="http", endpoint_override=endpoint, retry_strategy=AwsStandardS3RetryStrategy(max_attempts=3), connect_timeout=2, ) with fs.open_input_file(_OBJECT) as src: src.read() def test_when_error_body_fits_buffer_sdk_retries_slowdown(monkeypatch): """Control: large range fits the SlowDown XML; retries work today.""" assert 12 < len(_ERR_XML) < 8192 declared_size = 8192 monkeypatch.setenv("AWS_EC2_METADATA_DISABLED", "true") with _fake_s3(declared_size) as (endpoint, ranges): with pytest.raises(OSError) as excinfo: _read(endpoint) message = str(excinfo.value) assert "SLOW_DOWN" in message assert _FLUSH not in message assert ranges == [f"bytes=0-{declared_size - 1}"] * 3 def test_slowdown_preserved_when_error_xml_exceeds_small_read_buffer(monkeypatch): """Desired: same as control—throttle stays visible and SDK retries 3 times.""" declared_size = 121 monkeypatch.setenv("AWS_EC2_METADATA_DISABLED", "true") with _fake_s3(declared_size) as (endpoint, ranges): with pytest.raises(OSError) as excinfo: _read(endpoint) message = str(excinfo.value) assert "SLOW_DOWN" in message ``` ### Component(s) C++ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
