[issue26049] Poor performance when reading large xmlrpc data

[Oscar Andres Alvarez Montero]

Changes by Oscar Andres Alvarez Montero :


--
nosy: +Oscar Andres Alvarez Montero

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26049] Poor performance when reading large xmlrpc data

[Cédric Krier]

Cédric Krier added the comment:

One advantage, I see, is when xmlrpclib is overridden to use an other 
marshaller which is can not be feed chunk by chunk. So reducing the number of 
call to feed will have a bigger impact.
But I don't know if this is enough for Python.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26049] Poor performance when reading large xmlrpc data

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Thanks Cédric.

There is no improvement for the first request. But if repeat requests, there is 
small (about 5%) improvement. This is too small.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26049] Poor performance when reading large xmlrpc data

[Cédric Krier]

Cédric Krier added the comment:

Here is the client/server scripts.
I don't measure a big performance improvement with it.
I think the improvement measured in msg257756 are linked to the way xmlrpclib 
is overriden in Tryton.

--
Added file: http://bugs.python.org/file42044/server.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26049] Poor performance when reading large xmlrpc data

[Cédric Krier]

Changes by Cédric Krier :


Added file: http://bugs.python.org/file42043/client.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26049] Poor performance when reading large xmlrpc data

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

There is no special infrastructure. You can just provide two files. One script 
should implement simple server that produces large output. Other script should 
implement the client that makes a number of requests and measure the time. Of 
course you can implement this in one file using multiprocessing if you prefer.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26049] Poor performance when reading large xmlrpc data

[Cédric Krier]

Cédric Krier added the comment:

Is there an infrastructure already in place for such microbenchmark?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26049] Poor performance when reading large xmlrpc data

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Could you please provide a microbenchmark that exposes the benefit of this 
optimization?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26049] Poor performance when reading large xmlrpc data

[Cédric Krier]

Cédric Krier added the comment:

ping

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26049] Poor performance when reading large xmlrpc data

[Raimon Esteve]

Changes by Raimon Esteve :


--
nosy: +resteve

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26049] Poor performance when reading large xmlrpc data

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26049] Poor performance when reading large xmlrpc data

[Terry J. Reedy]

Terry J. Reedy added the comment:

I believe performance enhancements are, by default, limited to 'default', but 
this one might be a good candidate for backport.

--
nosy: +loewis, terry.reedy
versions: +Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26049] Poor performance when reading large xmlrpc data

[Sergi Almacellas Abellana]

New submission from Sergi Almacellas Abellana:

By default, python xmlrpclib parser reads data by chunks of 1024 bytes [1], 
which leads to a lot of data concatenations when reading large data, which is 
very slow in python.

Increasing the chuck size from 1024 bytes to a higher value makes improve in 
performance. 

On the same machine, we have tested with 20MB of data and we've got the 
following results: 

Chucks of 1024: 1min 48.933491sec
Chucks of 10 * 1024 * 1024: 0.245282sec

We have chosen 10 * 1024 * 1024, as it is the same value used in issue792570

We have done our tests on python2.7, but same code exists for default branch 
[2] (and 3.x branches also [3][4][5][6]), so I belive all the versions are 
affected. 

I can work on a patch if you think this change is acceptable.

IMHO it's logical to allow the developer to customize the chunk size instead of 
using a hard coded one. 

[1] https://hg.python.org/cpython/file/2.7/Lib/xmlrpclib.py#l1479
[2] https://hg.python.org/cpython/file/default/Lib/xmlrpc/client.py#l1310
[3] https://hg.python.org/cpython/file/3.5/Lib/xmlrpc/client.py#l1310
[4] https://hg.python.org/cpython/file/3.4/Lib/xmlrpc/client.py#l1324
[5] https://hg.python.org/cpython/file/3.3/Lib/xmlrpc/client.py#l1316
[6] https://hg.python.org/cpython/file/3.2/Lib/xmlrpc/client.py#l1301

--
components: XML
messages: 257756
nosy: pokoli
priority: normal
severity: normal
status: open
title: Poor performance when reading large xmlrpc data

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26049] Poor performance when reading large xmlrpc data

[Cédric Krier]

Cédric Krier added the comment:

I don't think it is necessary to allow to customize the chunk size. Indeed 
Python should provide a good value by default that works for all platforms.

--
nosy: +ced
type:  -> performance

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26049] Poor performance when reading large xmlrpc data

[Sergi Almacellas Abellana]

Sergi Almacellas Abellana added the comment:

And Also another patch for 2.7 branches

--
Added file: http://bugs.python.org/file41535/python27.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26049] Poor performance when reading large xmlrpc data

[Sergi Almacellas Abellana]

Sergi Almacellas Abellana added the comment:

I attach a patch to fix on default series

--
keywords: +patch
Added file: http://bugs.python.org/file41534/default.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26049] Poor performance when reading large xmlrpc data

[Sergi Almacellas Abellana]

Sergi Almacellas Abellana added the comment:

I added a new patch which fixed comments

--
Added file: http://bugs.python.org/file41536/default.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26049] Poor performance when reading large xmlrpc data

[Sergi Almacellas Abellana]

Changes by Sergi Almacellas Abellana :


Removed file: http://bugs.python.org/file41534/default.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com