Janek Bevendorff created BEAM-13585:
---------------------------------------
Summary: Python SDK S3 reader vastly inefficient
Key: BEAM-13585
URL: https://issues.apache.org/jira/browse/BEAM-13585
Project: Beam
Issue Type: Improvement
Components: sdk-py-core
Affects Versions: 2.34.0
Reporter: Janek Bevendorff
This is an "after-the-fact" Jira issue for my [GitHub
PR|https://github.com/apache/beam/pull/15931#issuecomment-999945083] to make S3
streaming in the Python SDK vastly more efficient.
The issue with the old implementation is that a new connection is opened for
each range request, which is very inefficient for both the client and the
server, adding a lot of unnecessary latency. The new implementation tries to
reused an existing connection and continues reading from the same HTTP stream
if possible.
Speed gain: 1.7-12x in benchmarks, more like 10x in real-word applications.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
