Thanks, Ivan. I confirmed 3000 negative cache entries were removed by your patch!
https://gist.github.com/methane/3c34f11fb677365a7e92afe73aca24e7 On Thu, Feb 2, 2017 at 1:16 AM, Ivan Levkivskyi <levkivs...@gmail.com> wrote: > Inada-san, > > I have made a PR for typing module upstream > https://github.com/python/typing/pull/383 > It should reduce the memory consumption significantly (and also increase > isinstance() speed). > Could you please try it with your real code base and test memory consumption > (and maybe speed) as compared to master? > > -- > Ivan > > > On 23 January 2017 at 12:25, INADA Naoki <songofaca...@gmail.com> wrote: >> >> On Fri, Jan 20, 2017 at 8:52 PM, Ivan Levkivskyi <levkivs...@gmail.com> >> wrote: >> > On 20 January 2017 at 11:49, INADA Naoki <songofaca...@gmail.com> wrote: >> >> >> >> * typing may increase memory footprint, through functions >> >> __attributes__ and abc. >> >> * Can we add option to remove or lazy evaluate __attributes__ ? >> > >> > >> > This idea already appeared few times. I proposed to introduce a flag >> > (e.g. >> > -OOO) to ignore function and variable annotations in compile.c >> > It was decide to postpone this, but maybe we can get back to this idea. >> > >> > In 3.6, typing is already (quite heavily) optimized for both speed and >> > space. >> > I remember doing an experiment comparing a memory footprint with and >> > without >> > annotations, the difference was few percent. >> > Do you have such comparison (with and without annotations) for your app? >> > It would be nice to have a realistic number to estimate what would the >> > additional optimization flag save. >> > >> > -- >> > Ivan >> > >> > >> >> Hi, Ivan. >> >> I investigated why our app has so many WeakSet today. >> >> We have dozen or hundreds of annotations like Iterable[User] or >> List[User]. >> (User is one example of application's domain object. There are >> hundreds of classes). >> >> On the other hand, SQLAlchemy calls isinstance(obj, >> collections.Iterable) many times, >> in >> [sqlalchemy.util._collections.to_list](https://github.com/zzzeek/sqlalchemy/blob/master/lib/sqlalchemy/util/_collections.py#L795-L804) >> method. >> >> So there are (# of iterable subclasses) weaksets for negative cache, >> and each weakset >> contains (# of column types) entries. That's why WeakSet ate much RAM. >> >> It may be slowdown application startup too, because thousands of >> __subclasscheck_ is called. >> >> I gave advice to use 'List[User]' instead of List[User] to the team of >> the project, >> if the team think RAM usage or boot speed is important. >> >> FWIW, stacktrace is like this: >> >> File "/Users/inada-n/local/py37dbg/lib/python3.7/_weakrefset.py", line >> 84 >> self.data.add(ref(item, self._remove)) >> File "/Users/inada-n/local/py37dbg/lib/python3.7/abc.py", line 233 >> cls._abc_negative_cache.add(subclass) >> File "/Users/inada-n/local/py37dbg/lib/python3.7/abc.py", line 226 >> if issubclass(subclass, scls): >> File "/Users/inada-n/local/py37dbg/lib/python3.7/abc.py", line 226 >> if issubclass(subclass, scls): >> File "/Users/inada-n/local/py37dbg/lib/python3.7/abc.py", line 191 >> return cls.__subclasscheck__(subclass) >> File "venv/lib/python3.7/site-packages/sqlalchemy/util/_collections.py", >> line 803 >> or not isinstance(x, collections.Iterable): >> File "venv/lib/python3.7/site-packages/sqlalchemy/orm/mapper.py", line >> 1680 >> columns = util.to_list(prop) >> File "venv/lib/python3.7/site-packages/sqlalchemy/orm/mapper.py", line >> 1575 >> prop = self._property_from_column(key, prop) >> File "venv/lib/python3.7/site-packages/sqlalchemy/orm/mapper.py", line >> 1371 >> setparent=True) >> File "venv/lib/python3.7/site-packages/sqlalchemy/orm/mapper.py", line >> 675 >> self._configure_properties()<PasteEnd> >> >> Regards, > > _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
