Re: [Python-Dev] Investigating Python memory footprint of one real Web application

2017-01-23 Thread INADA Naoki 
On Fri, Jan 20, 2017 at 8:52 PM, Ivan Levkivskyi  wrote:
> On 20 January 2017 at 11:49, INADA Naoki  wrote:
>>
>> * typing may increase memory footprint, through functions
>> __attributes__ and abc.
>>* Can we add option to remove or lazy evaluate __attributes__ ?
>
>
> This idea already appeared few times. I proposed to introduce a flag (e.g.
> -OOO) to ignore function and variable annotations in compile.c
> It was decide to postpone this, but maybe we can get back to this idea.
>
> In 3.6, typing is already (quite heavily) optimized for both speed and
> space.
> I remember doing an experiment comparing a memory footprint with and without
> annotations, the difference was few percent.
> Do you have such comparison (with and without annotations) for your app?
> It would be nice to have a realistic number to estimate what would the
> additional optimization flag save.
>
> --
> Ivan
>
>

Hi, Ivan.

I investigated why our app has so many WeakSet today.

We have dozen or hundreds of annotations like Iterable[User] or List[User].
(User is one example of application's domain object.  There are
hundreds of classes).

On the other hand, SQLAlchemy calls isinstance(obj,
collections.Iterable) many times,
in 
[sqlalchemy.util._collections.to_list](https://github.com/zzzeek/sqlalchemy/blob/master/lib/sqlalchemy/util/_collections.py#L795-L804)
method.

So there are (# of iterable subclasses) weaksets for negative cache,
and each weakset
contains (# of column types) entries.  That's why WeakSet ate much RAM.

It may be slowdown application startup too, because thousands of
__subclasscheck_ is called.

I gave advice to use 'List[User]' instead of List[User] to the team of
the project,
if the team think RAM usage or boot speed is important.

FWIW, stacktrace is like this:

  File "/Users/inada-n/local/py37dbg/lib/python3.7/_weakrefset.py", line 84
self.data.add(ref(item, self._remove))
  File "/Users/inada-n/local/py37dbg/lib/python3.7/abc.py", line 233
cls._abc_negative_cache.add(subclass)
  File "/Users/inada-n/local/py37dbg/lib/python3.7/abc.py", line 226
if issubclass(subclass, scls):
  File "/Users/inada-n/local/py37dbg/lib/python3.7/abc.py", line 226
if issubclass(subclass, scls):
  File "/Users/inada-n/local/py37dbg/lib/python3.7/abc.py", line 191
return cls.__subclasscheck__(subclass)
  File "venv/lib/python3.7/site-packages/sqlalchemy/util/_collections.py",
line 803
or not isinstance(x, collections.Iterable):
  File "venv/lib/python3.7/site-packages/sqlalchemy/orm/mapper.py", line 1680
columns = util.to_list(prop)
  File "venv/lib/python3.7/site-packages/sqlalchemy/orm/mapper.py", line 1575
prop = self._property_from_column(key, prop)
  File "venv/lib/python3.7/site-packages/sqlalchemy/orm/mapper.py", line 1371
setparent=True)
  File "venv/lib/python3.7/site-packages/sqlalchemy/orm/mapper.py", line 675
self._configure_properties()

Regards,
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Investigating Python memory footprint of one real Web application

2017-01-23 Thread Victor Stinner 
2017-01-23 12:25 GMT+01:00 INADA Naoki :
> I gave advice to use 'List[User]' instead of List[User] to the team of
> the project,
> if the team think RAM usage or boot speed is important.

I would prefer a Python option (ex: "-o noannotation" command line
option) to opt-out annotations rather than having to write annotations
in strings, which is IMHO more "ugly".

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Investigating Python memory footprint of one real Web application

2017-01-23 Thread INADA Naoki 
On Mon, Jan 23, 2017 at 8:33 PM, Victor Stinner
 wrote:
> 2017-01-23 12:25 GMT+01:00 INADA Naoki :
>> I gave advice to use 'List[User]' instead of List[User] to the team of
>> the project,
>> if the team think RAM usage or boot speed is important.
>
> I would prefer a Python option (ex: "-o noannotation" command line
> option) to opt-out annotations rather than having to write annotations
> in strings, which is IMHO more "ugly".
>
> Victor

Personally speaking, I hope annotations are just static hint, and
makes zero overhead at runtime.
(startup time, memory consumption, and execution speed).

Anyway, many users are starting to use typing, for code completion or
static checking.
And very few user noticed it affects to performance of `isinstance(x,
collections.Sequence)`.
Python 3.7 may be too slow to help them.
Can't we skip abc registration of typing.List[MyClass] completely?

I'm sorry if it's silly idea.  I don't know about background of
current typing.py design. And I
don't use abc much too.

Naoki
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Hello World

2017-01-23 Thread Brett Cannon 
In case you haven't already done so, Ramsey, please consider reading
https://cpython-devguide.readthedocs.io if you want to contribute to the
project. You can also subscribe to the core-mentorship mailing list which
is specifically for people who want to help out but haven't done to
previously.

On Sun, 22 Jan 2017 at 18:53 Ramsey D'silva  wrote:

> Hello World,
>
> I am a python developer that has lurked beyond my box.
>
> Nice to meet all of you'll. I'm excited to learn and hopefully contribute
> someday.
>
> Best Regards,
> Ramsey
>
> https://linkedin.com/in/ramseydsilva
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Investigating Python memory footprint of one real Web application

2017-01-23 Thread Brett Cannon 
On Mon, 23 Jan 2017 at 04:27 INADA Naoki  wrote:

> On Mon, Jan 23, 2017 at 8:33 PM, Victor Stinner
>  wrote:
> > 2017-01-23 12:25 GMT+01:00 INADA Naoki :
> >> I gave advice to use 'List[User]' instead of List[User] to the team of
> >> the project,
> >> if the team think RAM usage or boot speed is important.
> >
> > I would prefer a Python option (ex: "-o noannotation" command line
> > option) to opt-out annotations rather than having to write annotations
> > in strings, which is IMHO more "ugly".
>

So basically the equivalent of -OO for docstrings? Maybe this can be the
final motivator for some of us to come up with a design to generalize -O or
something as it keeps coming up.


> >
> > Victor
>
> Personally speaking, I hope annotations are just static hint, and
> makes zero overhead at runtime.
> (startup time, memory consumption, and execution speed).
>

Local variable annotations are nothing but info in the AST. Parameter
annotations and class annotations are stored on their respective objects so
there's memory usage from that and the construction of them at object
creation time, but that's it (e.g. the cost of creating
func.__annotations__ when the function object is created is all you pay for
performance-wise). And using strings will make those introspective
attributes difficult to use, hence why I don't think people have said to do
that everywhere.


>
> Anyway, many users are starting to use typing, for code completion or
> static checking.
> And very few user noticed it affects to performance of `isinstance(x,
> collections.Sequence)`.
> Python 3.7 may be too slow to help them.
> Can't we skip abc registration of typing.List[MyClass] completely?
>
> I'm sorry if it's silly idea.  I don't know about background of
> current typing.py design. And I
> don't use abc much too.


Since isinstance() checks are expected to be rare I don't think anyone has
worried too much about the performance beyond the initial work to introduce
ABCs and __instancecheck__.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Investigating Python memory footprint of one real Web application

2017-01-23 Thread Lukasz Langa 

> On Jan 23, 2017, at 12:10 PM, Brett Cannon  wrote:
> 
> 
> 
> On Mon, 23 Jan 2017 at 04:27 INADA Naoki  > wrote:
> On Mon, Jan 23, 2017 at 8:33 PM, Victor Stinner
> mailto:victor.stin...@gmail.com>> wrote:
> > 2017-01-23 12:25 GMT+01:00 INADA Naoki  > >:
> >> I gave advice to use 'List[User]' instead of List[User] to the team of
> >> the project,
> >> if the team think RAM usage or boot speed is important.
> >
> > I would prefer a Python option (ex: "-o noannotation" command line
> > option) to opt-out annotations rather than having to write annotations
> > in strings, which is IMHO more "ugly".
> 
> So basically the equivalent of -OO for docstrings? Maybe this can be the 
> final motivator for some of us to come up with a design to generalize -O or 
> something as it keeps coming up.

Yes, please. We've talked about generalizing this for years now. FWIW, I know 
of projects that run with -OO for the memory wins stemming from docstrings and 
had to codemod assert statements into a "prod_assert" function call to achieve 
this. If you think docstrings aren't that much, multiply this by a few hundred 
processes on a box and it ends up being a substantial win to strip them out.

> >
> > Victor
> 
> Personally speaking, I hope annotations are just static hint, and
> makes zero overhead at runtime.
> (startup time, memory consumption, and execution speed).
> 
> Local variable annotations are nothing but info in the AST. Parameter 
> annotations and class annotations are stored on their respective objects so 
> there's memory usage from that and the construction of them at object 
> creation time, but that's it (e.g. the cost of creating func.__annotations__ 
> when the function object is created is all you pay for performance-wise). And 
> using strings will make those introspective attributes difficult to use, 
> hence why I don't think people have said to do that everywhere.

I suggested making all annotations just strings at runtime and PEP 484 still 
lists this as a possible course for the future. So far Guido blocked this on a 
legitimate question: how much do type hints actually cost? Nobody knows yet, 
the biggest annotated codebase is at Dropbox and this is using comments (so no 
runtime cost).

>  
> Anyway, many users are starting to use typing, for code completion or
> static checking.
> And very few user noticed it affects to performance of `isinstance(x,
> collections.Sequence)`.
> Python 3.7 may be too slow to help them.
> Can't we skip abc registration of typing.List[MyClass] completely?
> 
> I'm sorry if it's silly idea.  I don't know about background of
> current typing.py design. And I
> don't use abc much too.
> 
> Since isinstance() checks are expected to be rare I don't think anyone has 
> worried too much about the performance beyond the initial work to introduce 
> ABCs and __instancecheck__.

Similar to the above, I would advise against crippling functionality unless we 
prove this is affecting performance in a significant way.

- Ł___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Hello World

2017-01-23 Thread Raymond Hettinger 

> On Jan 22, 2017, at 6:31 PM, Ramsey D'silva  wrote:
> 
> Nice to meet all of you'll. I'm excited to learn and hopefully contribute 
> someday.

Welcome aboard.


Raymond
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Hello World

2017-01-23 Thread Ramsey D'silva 
Thank you Brett. I will be doing as much reading as I can before seeking a
mentor. I have also subscribed to that mailing list.

On Mon, Jan 23, 2017 at 2:59 PM, Brett Cannon  wrote:

> In case you haven't already done so, Ramsey, please consider reading
> https://cpython-devguide.readthedocs.io if you want to contribute to the
> project. You can also subscribe to the core-mentorship mailing list which
> is specifically for people who want to help out but haven't done to
> previously.
>
> On Sun, 22 Jan 2017 at 18:53 Ramsey D'silva 
> wrote:
>
>> Hello World,
>>
>> I am a python developer that has lurked beyond my box.
>>
>> Nice to meet all of you'll. I'm excited to learn and hopefully contribute
>> someday.
>>
>> Best Regards,
>> Ramsey
>>
>> https://linkedin.com/in/ramseydsilva
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
>> brett%40python.org
>>
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Hello World

2017-01-23 Thread Ramsey D'silva 
Thank you!

On Mon, Jan 23, 2017 at 4:12 PM, Raymond Hettinger <
raymond.hettin...@gmail.com> wrote:

>
> > On Jan 22, 2017, at 6:31 PM, Ramsey D'silva 
> wrote:
> >
> > Nice to meet all of you'll. I'm excited to learn and hopefully
> contribute someday.
>
> Welcome aboard.
>
>
> Raymond
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Investigating Python memory footprint of one real Web application

2017-01-23 Thread INADA Naoki 
>> So basically the equivalent of -OO for docstrings? Maybe this can be the 
>> final motivator for some of us to come up with a design to generalize -O or 
>> something as it keeps coming up.
> Yes, please. We've talked about generalizing this for years now. FWIW, I know 
> of projects that run with -OO for the memory wins stemming from docstrings 
> and had to codemod assert statements into a "prod_assert" function call to 
> achieve this. If you think docstrings aren't that much, multiply this by a 
> few hundred processes on a box and it ends up being a substantial win to 
> strip them out.

Strong +1.


> So far Guido blocked this on a legitimate question: how much do type hints 
> actually cost? Nobody knows yet,

"Nobody knows yet" is difficult problem.

We may think "let's keep runtime cost, because nobody knows how large it is".
Users may think "let's use string/comment form annotation to avoid runtime cost,
because nobody knows how large it is."

And problem may happen in closed source application.
When building closed source application, the project can drop Python 2
support easily,
and buy PyCharm for all members.

(BTW, PyCharm's survey result [1] is very encouraging.
PyCharm users adopts Python 3 (relative) early.  I think they will
adopt typing early too.)

Early and large adopters of typing may be such teams (like my company).
And they may feel "Python is slow and fat!" if there are no easy way to check
runtime overhead of typing.

Optimize option to drop annotation will provide (1) easy way to check
runtime overhead of typing,
and (2) straightforward solution to remove the overhead, if it isn't negligible.


[1]: https://www.jetbrains.com/pycharm/python-developers-survey-2016/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com