[
https://issues.apache.org/jira/browse/BEAM-13585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anonymous updated BEAM-13585:
-----------------------------
Status: Triage Needed (was: Resolved)
> Python SDK S3 reader 2x - 10x performance opportunity
> -----------------------------------------------------
>
> Key: BEAM-13585
> URL: https://issues.apache.org/jira/browse/BEAM-13585
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
> Affects Versions: 2.34.0, 2.35.0, 2.36.0
> Reporter: Janek Bevendorff
> Assignee: Janek Bevendorff
> Priority: P2
> Fix For: 2.37.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> This is an "after-the-fact" Jira issue for my [GitHub
> PR|https://github.com/apache/beam/pull/15931#issuecomment-999945083] to make
> S3 streaming in the Python SDK vastly more efficient.
> The issue with the old implementation was that a new connection was opened
> for each range request, which is very inefficient for both the client and the
> server, adding a lot of unnecessary latency. The new implementation tries to
> reused an existing connection and continues reading from the same HTTP stream
> if possible.
> Speed gain: 1.7-12x in benchmarks, more like 10x in real-word applications.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
