Re: [Python-Dev] Investigating Python memory footprint of one real Web application

Thanks, Ivan. I confirmed 3000 negative cache entries were removed by your patch! https://gist.github.com/methane/3c34f11fb677365a7e92afe73aca24e7 On Thu, Feb 2, 2017 at 1:16 AM, Ivan Levkivskyi wrote: > Inada-san, > > I have made a PR for typing module upstream > https://github.com/python/ty

Re: [Python-Dev] Investigating Python memory footprint of one real Web application

Inada-san, I have made a PR for typing module upstream https://github.com/python/typing/pull/383 It should reduce the memory consumption significantly (and also increase isinstance() speed). Could you please try it with your real code base and test memory consumption (and maybe speed) as compared

Re: [Python-Dev] Investigating Python memory footprint of one real Web application

On Thu, Jan 26, 2017 at 2:33 AM, Antoine Pitrou wrote: > On Wed, 25 Jan 2017 20:54:02 +0900 > INADA Naoki wrote: >> >> ## Stripped annotation + without pydebug > > Does this mean the other measurements were done with pydebug enabled? > pydebug is not meant to be used on production systems so, wit

Re: [Python-Dev] Investigating Python memory footprint of one real Web application

On Wed, 25 Jan 2017 20:54:02 +0900 INADA Naoki wrote: > > ## Stripped annotation + without pydebug Does this mean the other measurements were done with pydebug enabled? pydebug is not meant to be used on production systems so, without wanting to disparage the effort this went into these measurem

Re: [Python-Dev] Investigating Python memory footprint of one real Web application

More detailed information: ## With annotations === tracemalloc stat === traced: (46969277, 46983753) 18,048,888 / 181112 File "", line 488 File "", line 780 File "", line 675 === size by types === dict 9,083,816 (8,870.91KB) / 21846 = 415.811bytes (21.38%) tuple

Re: [Python-Dev] Investigating Python memory footprint of one real Web application

On Tue, 24 Jan 2017 10:21:45 -0800 Nathaniel Smith wrote: > > The thing I found most surprising about that blog post was that contrary to > common wisdom, refcnt updates per se had essentially no effect on the > amount of memory shared between CoW processes, and the problems were all > due to the

Re: [Python-Dev] Investigating Python memory footprint of one real Web application

On Jan 24, 2017 3:35 AM, "Thomas Wouters" wrote: On Fri, Jan 20, 2017 at 1:40 PM, Christian Heimes wrote: > On 2017-01-20 13:15, INADA Naoki wrote: > >> > >> "this script counts static memory usage. It doesn’t care about dynamic > >> memory usage of processing real request" > >> > >> You may

Re: [Python-Dev] Investigating Python memory footprint of one real Web application

On Tue, Jan 24, 2017 at 11:08 PM, Victor Stinner wrote: > 2017-01-24 15:00 GMT+01:00 INADA Naoki : >> And here are top 3 tracebacks from tracemalloc: >> >> 15109615 (/180598) >> File "", line 488 >> File "", line 780 >> File "", line 675 >> File "", line 655 > > FYI at Python startup, usua

Re: [Python-Dev] Investigating Python memory footprint of one real Web application

2017-01-24 15:00 GMT+01:00 INADA Naoki : > And here are top 3 tracebacks from tracemalloc: > > 15109615 (/180598) > File "", line 488 > File "", line 780 > File "", line 675 > File "", line 655 FYI at Python startup, usually the largest memory block comes from the dictionary used to intern

Re: [Python-Dev] Investigating Python memory footprint of one real Web application

FWIW, I tried to skip compiler_visit_annotations() in Python/compile.c a) default: 41278060 b) remove annotations: 37140094 c) (b) + const merge: 35933436 (a-b)/a = 10% (a-c)/a = 13% And here are top 3 tracebacks from tracemalloc: 15109615 (/180598) File "", line 488 File

Re: [Python-Dev] Investigating Python memory footprint of one real Web application

> 3. I am -1 on ignoring annotations altogether. Sometimes they could be > helpful at runtime: typing.NamedTuple and mypy_extensions.TypedDict are two > examples. ignoring annotations doesn't mean ignoring typing at all. You can use typing.NamedTuple even when functions doesn't have __annotations_

Re: [Python-Dev] Investigating Python memory footprint of one real Web application

1. It looks like there is still a room for performance improvement of typing w.r.t. how ABCs and issubclass() works. I will try to play with this soon. (the basic idea is that some steps could be avoided for parameterized generics). 2. I am +1 on having three separate options to independently igno

Re: [Python-Dev] Investigating Python memory footprint of one real Web application

On Fri, Jan 20, 2017 at 1:40 PM, Christian Heimes wrote: > On 2017-01-20 13:15, INADA Naoki wrote: > >> > >> "this script counts static memory usage. It doesn’t care about dynamic > >> memory usage of processing real request" > >> > >> You may be trying to optimize something which is only a very

Re: [Python-Dev] Investigating Python memory footprint of one real Web application

>> So basically the equivalent of -OO for docstrings? Maybe this can be the >> final motivator for some of us to come up with a design to generalize -O or >> something as it keeps coming up. > Yes, please. We've talked about generalizing this for years now. FWIW, I know > of projects that run wi

Re: [Python-Dev] Investigating Python memory footprint of one real Web application

> On Jan 23, 2017, at 12:10 PM, Brett Cannon wrote: > > > > On Mon, 23 Jan 2017 at 04:27 INADA Naoki > wrote: > On Mon, Jan 23, 2017 at 8:33 PM, Victor Stinner > mailto:victor.stin...@gmail.com>> wrote: > > 2017-01-23 12:25 GMT+01:00 INADA Naoki >

Re: [Python-Dev] Investigating Python memory footprint of one real Web application

On Mon, 23 Jan 2017 at 04:27 INADA Naoki wrote: > On Mon, Jan 23, 2017 at 8:33 PM, Victor Stinner > wrote: > > 2017-01-23 12:25 GMT+01:00 INADA Naoki : > >> I gave advice to use 'List[User]' instead of List[User] to the team of > >> the project, > >> if the team think RAM usage or boot speed is

Re: [Python-Dev] Investigating Python memory footprint of one real Web application

On Mon, Jan 23, 2017 at 8:33 PM, Victor Stinner wrote: > 2017-01-23 12:25 GMT+01:00 INADA Naoki : >> I gave advice to use 'List[User]' instead of List[User] to the team of >> the project, >> if the team think RAM usage or boot speed is important. > > I would prefer a Python option (ex: "-o noannot

Re: [Python-Dev] Investigating Python memory footprint of one real Web application

2017-01-23 12:25 GMT+01:00 INADA Naoki : > I gave advice to use 'List[User]' instead of List[User] to the team of > the project, > if the team think RAM usage or boot speed is important. I would prefer a Python option (ex: "-o noannotation" command line option) to opt-out annotations rather than h

Re: [Python-Dev] Investigating Python memory footprint of one real Web application

On Fri, Jan 20, 2017 at 8:52 PM, Ivan Levkivskyi wrote: > On 20 January 2017 at 11:49, INADA Naoki wrote: >> >> * typing may increase memory footprint, through functions >> __attributes__ and abc. >>* Can we add option to remove or lazy evaluate __attributes__ ? > > > This idea already appear

Re: [Python-Dev] Investigating Python memory footprint of one real Web application

I've filed an issue about merging tuples: http://bugs.python.org/issue29336 I'll try the patch with my company's codebase again in next week. But could someone try the patch with realworld large application too? Or if you know OSS large application easy to install, could you share requirements.tx

Re: [Python-Dev] Investigating Python memory footprint of one real Web application

On Fri, 20 Jan 2017 22:30:16 +0900 INADA Naoki wrote: > > > > Moving the refcount out of the PyObject will probably make increfs / > > decrefs more costly, and there are a lot of them. We'd have to see > > actual measurements if a patch is written, but my intuition is that the > > net result won'

Re: [Python-Dev] Investigating Python memory footprint of one real Web application

> > Moving the refcount out of the PyObject will probably make increfs / > decrefs more costly, and there are a lot of them. We'd have to see > actual measurements if a patch is written, but my intuition is that the > net result won't be positive. > > Regards > > Antoine. I agree with you. But I

Re: [Python-Dev] Investigating Python memory footprint of one real Web application

Larry Hastings' Gilectomy also moved the reference counter into a separated memory block, no? (grouping all refcounts into large memory blocks if I understood correctly.) https://github.com/larryhastings/gilectomy Victor 2017-01-20 13:40 GMT+01:00 Christian Heimes : > On 2017-01-20 13:15, INADA

Re: [Python-Dev] Investigating Python memory footprint of one real Web application

On Fri, 20 Jan 2017 13:40:14 +0100 Christian Heimes wrote: > > IIRC Thomas Wouters (?) has been working on a patch to move the ref > counter out of the PyObject struct and into a dedicated memory area. He > proposed the idea to improve cache affinity, reduce cache evictions and > to make CoW more

Re: [Python-Dev] Investigating Python memory footprint of one real Web application

On Fri, Jan 20, 2017 at 8:17 PM, Victor Stinner wrote: > 2017-01-20 11:49 GMT+01:00 INADA Naoki : >> Report is here >> https://gist.github.com/methane/ce723adb9a4d32d32dc7525b738d3c31 > > Very interesting report, thanks! > >> My thoughts are: >> >> * Interning (None,) seems worth enough. > > I gue

Re: [Python-Dev] Investigating Python memory footprint of one real Web application

On 2017-01-20 13:15, INADA Naoki wrote: >> >> "this script counts static memory usage. It doesn’t care about dynamic >> memory usage of processing real request" >> >> You may be trying to optimize something which is only a very small >> fraction of your actual memory footprint. That said, the mars

Re: [Python-Dev] Investigating Python memory footprint of one real Web application

On 2017-01-20 13:15, INADA Naoki wrote: >> >> "this script counts static memory usage. It doesn’t care about dynamic >> memory usage of processing real request" >> >> You may be trying to optimize something which is only a very small >> fraction of your actual memory footprint. That said, the mars

Re: [Python-Dev] Investigating Python memory footprint of one real Web application

On Fri, Jan 20, 2017 at 8:52 PM, Ivan Levkivskyi wrote: > On 20 January 2017 at 11:49, INADA Naoki wrote: >> >> * typing may increase memory footprint, through functions >> __attributes__ and abc. >>* Can we add option to remove or lazy evaluate __attributes__ ? > > > This idea already appear

Re: [Python-Dev] Investigating Python memory footprint of one real Web application

> > "this script counts static memory usage. It doesn’t care about dynamic > memory usage of processing real request" > > You may be trying to optimize something which is only a very small > fraction of your actual memory footprint. That said, the marshal > module could certainly try to intern som

Re: [Python-Dev] Investigating Python memory footprint of one real Web application

On 20 January 2017 at 11:49, INADA Naoki wrote: > * typing may increase memory footprint, through functions > __attributes__ and abc. >* Can we add option to remove or lazy evaluate __attributes__ ? > This idea already appeared few times. I proposed to introduce a flag (e.g. -OOO) to ignore

Re: [Python-Dev] Investigating Python memory footprint of one real Web application

On Fri, 20 Jan 2017 19:49:01 +0900 INADA Naoki wrote: > > Report is here > https://gist.github.com/methane/ce723adb9a4d32d32dc7525b738d3c31 "this script counts static memory usage. It doesn’t care about dynamic memory usage of processing real request" You may be trying to optimize something whi

Re: [Python-Dev] Investigating Python memory footprint of one real Web application

2017-01-20 11:49 GMT+01:00 INADA Naoki : > Report is here > https://gist.github.com/methane/ce723adb9a4d32d32dc7525b738d3c31 Very interesting report, thanks! > My thoughts are: > > * Interning (None,) seems worth enough. I guess that (None,) comes from constants of code objects: >>> def f(): pa

[Python-Dev] Investigating Python memory footprint of one real Web application

Hi, all. After reading Instagram's blog article [1], I’m thinking about how Python can reduce memory usage of Web applications. My company creating API server with Flask, SQLAlchemy and typing. (sorry, it's closed source). So I can get some data from it's codebase. [1]: https://engineering.insta