[issue21790] Change blocksize in http.client to the value of resource.getpagesize

[Berker Peksag]

Change by Berker Peksag :


--
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21790] Change blocksize in http.client to the value of resource.getpagesize

[Martin Panter]

Martin Panter  added the comment:

Issue 31945 proposes adding a “blocksize” parameter to HTTPConnection objects, 
so I suggest to closing in favour of that one.

--
resolution:  -> rejected
superseder:  -> Configurable blocksize in HTTP(S)Connection

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21790] Change blocksize in http.client to the value of resource.getpagesize

[Martin Panter]

Martin Panter added the comment:

For plain-text (non-SSL) HTTP uploads, perhaps using “socket.sendfile” would 
help: Issue 13559.

Another idea would be to expose and document the low-level socket (or SSL 
wrapper) and let the user copy data with whatever chunk size they desire.

--
nosy: +martin.panter

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21790] Change blocksize in http.client to the value of resource.getpagesize

[Clay Gerrard]

Clay Gerrard added the comment:

I don't think the default is bad/wrong so much as it's an inflexible magic 
number.

I think blocksize should be parameterized as an attribute on the connection - 
or at least a class attribute or module level constant so we can monkey patch 
it without having to create a subclass and redefine the *entire* send method 
(!?).

Not having a good string to pull at here is causing a non-trivial amount of 
pain:

https://github.com/sigmavirus24/requests-toolbelt/pull/84/files
https://github.com/sigmavirus24/requests-toolbelt/issues/75
https://webcache.googleusercontent.com/search?q=cache:WPSngjyUzg4J:https://technology.jana.com/2015/12/30/dont-use-python-httplib-or-any-library-that-uses-it-to-transfer-large-files-especially-to-china/+&cd=5&hl=en&ct=clnk&gl=us
https://bugs.launchpad.net/python-swiftclient/+bug/1671621

--
nosy: +Clay Gerrard

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21790] Change blocksize in http.client to the value of resource.getpagesize

[Demian Brecht]

Demian Brecht added the comment:

I very well could be missing something here (and, admittedly, my OS knowledge 
is rusty at best), but for the most part, page sizes are 4096, except for 
SPARC, which is 8192.

> So your page double the number of calls to read(), right?

No. read() is called until EOF. I'm assuming that 8192 may have been used to 
accommodate worst case architecture?

I'd have to dig through the C side of things (which I haven't done yet) to see 
what's going on down at that level, but my assumption is that it's allocating 
8192 bytes for each read. Should EOF be reached with <= 1 page filled, it'd 
result in a wasted page.

Far from the end of the world, but just something I noticed in passing.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21790] Change blocksize in http.client to the value of resource.getpagesize

[STINNER Victor]

STINNER Victor added the comment:

> When sending data, the blocksize is currently hardcoded to 8192.

Yeah, same value for io.DEFAULT_BUFFER_SIZE.

> It should likely be set to the value of resource.getpagesize().

Could you please elaborate? A page size is 4096 bytes on my Linux. So your page 
double the number of calls to read(), right?

shutil.copyfileobj() uses a buffer of 16 KB by default.

See also the issue #21679 which adds a private copy of the stat().st_blksize 
attribute in a FileIO object.

--
nosy: +haypo

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21790] Change blocksize in http.client to the value of resource.getpagesize

[Demian Brecht]

Demian Brecht added the comment:

Updated to mmap.PAGESIZE, which seems to be available on all systems.

--
Added file: http://bugs.python.org/file35707/issue21790.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21790] Change blocksize in http.client to the value of resource.getpagesize

[Claudiu Popa]

Claudiu Popa added the comment:

resource module is available only on Unix.

--
nosy: +Claudiu.Popa

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21790] Change blocksize in http.client to the value of resource.getpagesize

[Demian Brecht]

New submission from Demian Brecht:

When sending data, the blocksize is currently hardcoded to 8192. It should 
likely be set to the value of resource.getpagesize().

--
components: Library (Lib)
files: set_blocksize_to_getpagesize.diff
keywords: patch
messages: 220839
nosy: dbrecht
priority: normal
severity: normal
status: open
title: Change blocksize in http.client to the value of resource.getpagesize
versions: Python 3.5
Added file: http://bugs.python.org/file35671/set_blocksize_to_getpagesize.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com