[
https://issues.apache.org/jira/browse/BEAM-13585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Janek Bevendorff updated BEAM-13585:
------------------------------------
Description:
This is an "after-the-fact" Jira issue for my [GitHub
PR|https://github.com/apache/beam/pull/15931#issuecomment-999945083] to make S3
streaming in the Python SDK vastly more efficient.
The issue with the old implementation was that a new connection is opened for
each range request, which is very inefficient for both the client and the
server, adding a lot of unnecessary latency. The new implementation tries to
reused an existing connection and continues reading from the same HTTP stream
if possible.
Speed gain: 1.7-12x in benchmarks, more like 10x in real-word applications.
was:
This is an "after-the-fact" Jira issue for my [GitHub
PR|https://github.com/apache/beam/pull/15931#issuecomment-999945083] to make S3
streaming in the Python SDK vastly more efficient.
The issue with the old implementation is that a new connection is opened for
each range request, which is very inefficient for both the client and the
server, adding a lot of unnecessary latency. The new implementation tries to
reused an existing connection and continues reading from the same HTTP stream
if possible.
Speed gain: 1.7-12x in benchmarks, more like 10x in real-word applications.
> Python SDK S3 reader vastly inefficient
> ---------------------------------------
>
> Key: BEAM-13585
> URL: https://issues.apache.org/jira/browse/BEAM-13585
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
> Affects Versions: 2.34.0
> Reporter: Janek Bevendorff
> Priority: P2
> Time Spent: 10m
> Remaining Estimate: 0h
>
> This is an "after-the-fact" Jira issue for my [GitHub
> PR|https://github.com/apache/beam/pull/15931#issuecomment-999945083] to make
> S3 streaming in the Python SDK vastly more efficient.
> The issue with the old implementation was that a new connection is opened for
> each range request, which is very inefficient for both the client and the
> server, adding a lot of unnecessary latency. The new implementation tries to
> reused an existing connection and continues reading from the same HTTP stream
> if possible.
> Speed gain: 1.7-12x in benchmarks, more like 10x in real-word applications.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
