Re: [Python-Dev] Investigating Python memory footprint of one real Web application

2017-01-20 Thread INADA Naoki
I've filed an issue about merging tuples: http://bugs.python.org/issue29336 I'll try the patch with my company's codebase again in next week. But could someone try the patch with realworld large application too? Or if you know OSS large application easy to install, could you share requirements.tx

[Python-Dev] Summary of Python tracker Issues

2017-01-20 Thread Python tracker
ACTIVITY SUMMARY (2017-01-13 - 2017-01-20) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open5755 (+36) closed 35335 (+35) total 41090 (+71) Open issues wit

Re: [Python-Dev] Investigating Python memory footprint of one real Web application

2017-01-20 Thread Antoine Pitrou
On Fri, 20 Jan 2017 22:30:16 +0900 INADA Naoki wrote: > > > > Moving the refcount out of the PyObject will probably make increfs / > > decrefs more costly, and there are a lot of them. We'd have to see > > actual measurements if a patch is written, but my intuition is that the > > net result won'

Re: [Python-Dev] Investigating Python memory footprint of one real Web application

2017-01-20 Thread INADA Naoki
> > Moving the refcount out of the PyObject will probably make increfs / > decrefs more costly, and there are a lot of them. We'd have to see > actual measurements if a patch is written, but my intuition is that the > net result won't be positive. > > Regards > > Antoine. I agree with you. But I

Re: [Python-Dev] Investigating Python memory footprint of one real Web application

2017-01-20 Thread Victor Stinner
Larry Hastings' Gilectomy also moved the reference counter into a separated memory block, no? (grouping all refcounts into large memory blocks if I understood correctly.) https://github.com/larryhastings/gilectomy Victor 2017-01-20 13:40 GMT+01:00 Christian Heimes : > On 2017-01-20 13:15, INADA

Re: [Python-Dev] Investigating Python memory footprint of one real Web application

2017-01-20 Thread Antoine Pitrou
On Fri, 20 Jan 2017 13:40:14 +0100 Christian Heimes wrote: > > IIRC Thomas Wouters (?) has been working on a patch to move the ref > counter out of the PyObject struct and into a dedicated memory area. He > proposed the idea to improve cache affinity, reduce cache evictions and > to make CoW more

Re: [Python-Dev] Investigating Python memory footprint of one real Web application

2017-01-20 Thread INADA Naoki
On Fri, Jan 20, 2017 at 8:17 PM, Victor Stinner wrote: > 2017-01-20 11:49 GMT+01:00 INADA Naoki : >> Report is here >> https://gist.github.com/methane/ce723adb9a4d32d32dc7525b738d3c31 > > Very interesting report, thanks! > >> My thoughts are: >> >> * Interning (None,) seems worth enough. > > I gue

Re: [Python-Dev] Investigating Python memory footprint of one real Web application

2017-01-20 Thread Christian Heimes
On 2017-01-20 13:15, INADA Naoki wrote: >> >> "this script counts static memory usage. It doesn’t care about dynamic >> memory usage of processing real request" >> >> You may be trying to optimize something which is only a very small >> fraction of your actual memory footprint. That said, the mars

Re: [Python-Dev] Investigating Python memory footprint of one real Web application

2017-01-20 Thread Christian Heimes
On 2017-01-20 13:15, INADA Naoki wrote: >> >> "this script counts static memory usage. It doesn’t care about dynamic >> memory usage of processing real request" >> >> You may be trying to optimize something which is only a very small >> fraction of your actual memory footprint. That said, the mars

Re: [Python-Dev] Investigating Python memory footprint of one real Web application

2017-01-20 Thread INADA Naoki
On Fri, Jan 20, 2017 at 8:52 PM, Ivan Levkivskyi wrote: > On 20 January 2017 at 11:49, INADA Naoki wrote: >> >> * typing may increase memory footprint, through functions >> __attributes__ and abc. >>* Can we add option to remove or lazy evaluate __attributes__ ? > > > This idea already appear

Re: [Python-Dev] Investigating Python memory footprint of one real Web application

2017-01-20 Thread INADA Naoki
> > "this script counts static memory usage. It doesn’t care about dynamic > memory usage of processing real request" > > You may be trying to optimize something which is only a very small > fraction of your actual memory footprint. That said, the marshal > module could certainly try to intern som

Re: [Python-Dev] Investigating Python memory footprint of one real Web application

2017-01-20 Thread Ivan Levkivskyi
On 20 January 2017 at 11:49, INADA Naoki wrote: > * typing may increase memory footprint, through functions > __attributes__ and abc. >* Can we add option to remove or lazy evaluate __attributes__ ? > This idea already appeared few times. I proposed to introduce a flag (e.g. -OOO) to ignore

Re: [Python-Dev] Investigating Python memory footprint of one real Web application

2017-01-20 Thread Antoine Pitrou
On Fri, 20 Jan 2017 19:49:01 +0900 INADA Naoki wrote: > > Report is here > https://gist.github.com/methane/ce723adb9a4d32d32dc7525b738d3c31 "this script counts static memory usage. It doesn’t care about dynamic memory usage of processing real request" You may be trying to optimize something whi

Re: [Python-Dev] Investigating Python memory footprint of one real Web application

2017-01-20 Thread Victor Stinner
2017-01-20 11:49 GMT+01:00 INADA Naoki : > Report is here > https://gist.github.com/methane/ce723adb9a4d32d32dc7525b738d3c31 Very interesting report, thanks! > My thoughts are: > > * Interning (None,) seems worth enough. I guess that (None,) comes from constants of code objects: >>> def f(): pa

[Python-Dev] Investigating Python memory footprint of one real Web application

2017-01-20 Thread INADA Naoki
Hi, all. After reading Instagram's blog article [1], I’m thinking about how Python can reduce memory usage of Web applications. My company creating API server with Flask, SQLAlchemy and typing. (sorry, it's closed source). So I can get some data from it's codebase. [1]: https://engineering.insta