Re: [Python-Dev] Investigating Python memory footprint of one real Web application
Thanks, Ivan. I confirmed 3000 negative cache entries were removed by your patch! https://gist.github.com/methane/3c34f11fb677365a7e92afe73aca24e7 On Thu, Feb 2, 2017 at 1:16 AM, Ivan Levkivskyi wrote: > Inada-san, > > I have made a PR for typing module upstream > https://github.com/python/typing/pull/383 > It should reduce the memory consumption significantly (and also increase > isinstance() speed). > Could you please try it with your real code base and test memory consumption > (and maybe speed) as compared to master? > > -- > Ivan > > > On 23 January 2017 at 12:25, INADA Naoki wrote: >> >> On Fri, Jan 20, 2017 at 8:52 PM, Ivan Levkivskyi >> wrote: >> > On 20 January 2017 at 11:49, INADA Naoki wrote: >> >> >> >> * typing may increase memory footprint, through functions >> >> __attributes__ and abc. >> >>* Can we add option to remove or lazy evaluate __attributes__ ? >> > >> > >> > This idea already appeared few times. I proposed to introduce a flag >> > (e.g. >> > -OOO) to ignore function and variable annotations in compile.c >> > It was decide to postpone this, but maybe we can get back to this idea. >> > >> > In 3.6, typing is already (quite heavily) optimized for both speed and >> > space. >> > I remember doing an experiment comparing a memory footprint with and >> > without >> > annotations, the difference was few percent. >> > Do you have such comparison (with and without annotations) for your app? >> > It would be nice to have a realistic number to estimate what would the >> > additional optimization flag save. >> > >> > -- >> > Ivan >> > >> > >> >> Hi, Ivan. >> >> I investigated why our app has so many WeakSet today. >> >> We have dozen or hundreds of annotations like Iterable[User] or >> List[User]. >> (User is one example of application's domain object. There are >> hundreds of classes). >> >> On the other hand, SQLAlchemy calls isinstance(obj, >> collections.Iterable) many times, >> in >> [sqlalchemy.util._collections.to_list](https://github.com/zzzeek/sqlalchemy/blob/master/lib/sqlalchemy/util/_collections.py#L795-L804) >> method. >> >> So there are (# of iterable subclasses) weaksets for negative cache, >> and each weakset >> contains (# of column types) entries. That's why WeakSet ate much RAM. >> >> It may be slowdown application startup too, because thousands of >> __subclasscheck_ is called. >> >> I gave advice to use 'List[User]' instead of List[User] to the team of >> the project, >> if the team think RAM usage or boot speed is important. >> >> FWIW, stacktrace is like this: >> >> File "/Users/inada-n/local/py37dbg/lib/python3.7/_weakrefset.py", line >> 84 >> self.data.add(ref(item, self._remove)) >> File "/Users/inada-n/local/py37dbg/lib/python3.7/abc.py", line 233 >> cls._abc_negative_cache.add(subclass) >> File "/Users/inada-n/local/py37dbg/lib/python3.7/abc.py", line 226 >> if issubclass(subclass, scls): >> File "/Users/inada-n/local/py37dbg/lib/python3.7/abc.py", line 226 >> if issubclass(subclass, scls): >> File "/Users/inada-n/local/py37dbg/lib/python3.7/abc.py", line 191 >> return cls.__subclasscheck__(subclass) >> File "venv/lib/python3.7/site-packages/sqlalchemy/util/_collections.py", >> line 803 >> or not isinstance(x, collections.Iterable): >> File "venv/lib/python3.7/site-packages/sqlalchemy/orm/mapper.py", line >> 1680 >> columns = util.to_list(prop) >> File "venv/lib/python3.7/site-packages/sqlalchemy/orm/mapper.py", line >> 1575 >> prop = self._property_from_column(key, prop) >> File "venv/lib/python3.7/site-packages/sqlalchemy/orm/mapper.py", line >> 1371 >> setparent=True) >> File "venv/lib/python3.7/site-packages/sqlalchemy/orm/mapper.py", line >> 675 >> self._configure_properties() >> >> Regards, > > ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Heads up: possible double-comments on bpo for commits
Oh, I noticed another strange thing: The Roundup Robot closed the issue http://bugs.python.org/issue29368 but I don't see any explicit "Close issue #xxx" or "Close #xxx" in the commit message of the two commits. Is it deliberate to close an issue after any commit? Victor 2017-01-31 19:18 GMT+01:00 Brett Cannon : > I've activated the webhook for receiving comments on issues when a commit > lands mentioning an issue, so if you see a commit from our hg integration > and another from GitHub, understand that's why (mention issues as "bpo " > in commit messages if you want to see it in action). If it becomes too much > of a hassle to see the duplicates before we migrate I can turn off the > notifications, but obviously more testing the better. :) > > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com > ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] SSL certificates recommendations for downstreampython packagers
> On 2 Feb 2017, at 03:38, Stephen J. Turnbull > wrote: > > Cory Benfield writes: > >> The TL;DR is: I understand Christian’s concern, but I don’t think >> it’s important if you’re very, very careful. > > But AIUI, the "you" above is the end-user or admin of end-user's > system, no? We know that they aren't very careful (or perhaps more > accurate, this is too fsckin' complicated for anybody but an infosec > expert to do very well). I think "you" is the coder of the interface. From a security perspective I think we have to discount the possibility of administrator error from our threat model. A threat model that includes “defend the system against intrusions that the administrator incorrectly allows” is an insanely difficult one to respond to, given that it basically requires psychic powers to determine what the administrator *meant* instead of what they configured. Now, where we allow configuration we have a duty to ensure that it’s as easy as possible to configure correctly, but when using the system trust store most of the configuration is actually provided by the OS tools, rather than by the above-mentioned “you”, so that’s not in our control. The risk, and the need to be very very careful, comes from ensuring that the semantics of the OS configuration are preserved through to the behaviour of the program. This is definitely a place with razor-blades all around, which is why I have tended to defer to the Chrome security team on this issue. In particular, the BoringSSL developers are razor-sharp people who have their heads screwed on when it comes to practical security decisions, and I’ve found that emulating them is usually a safe bet in the face of ambiguity. However, it’s unquestionable that the *safest* route to go down in terms of preserving the expectations of users is to use the platform-native TLS implementation wholesale, rather than do a hybrid model like Chrome does where OpenSSL does the protocol bits and the system does the X509 bits. That way Python ends up behaving basically like Edge or Safari on the relevant platforms, or perhaps more importantly behaving like .NET on Windows and like CoreFoundation on macOS, which is a much better place to be in terms of user and administrator expectations. As a side benefit, that approach helps take Python a bit closer to feeling “platform-native” on many platforms, which can only be a good thing for those of us who want to see more Python on the desktop (or indeed on the mobile device). > I[1] still agree with you that it's *unlikely* that end-users/admins > will need to worry about it. But we need to be really careful about > what we say here, or at least where the responsible parties will be > looking. I agree. In an ideal world I’d say to Steve that he should shelve his current work and wait for the TLS ABC PEP that is incoming (hopefully I’ll send a first-draft to python-dev today). However, I’m nothing if not pragmatic, and having Steve continue his current work in parallel to the TLS ABC PEP is probably a good idea so that we can avoid having all our eggs in one basket. Perhaps we can get the TLS ABC stuff in place in time for Steve to just swap over to using SChannel altogether, but if that doesn’t work out and Steve can get a halfway-house out the door earlier then that’s fine by me. Cory ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Heads up: possible double-comments on bpo for commits
2017-02-01 18:32 GMT+01:00 Victor Stinner : > To finish with an empty change: Information on IRC: Known issue: http://psf.upfronthosting.co.za/roundup/meta/issue611 "Maciej is working on it" Good & thanks! Victor ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] re performance
On 2017-02-02 04:37, Franklin? Lee wrote: On Thu, Jan 26, 2017 at 4:13 PM, Sven R. Kunze wrote: Hi folks, I recently refreshed regular expressions theoretical basics *indulging in reminiscences* So, I read https://swtch.com/~rsc/regexp/regexp1.html However, reaching the chart in the lower third of the article, I saw Python 2.4 measured against a naive Thompson matching implementation. And I was surprised about how bad it performed compared to an unoptimized version of an older than dirt algorithm. From my perspective, I can say, that regular expressions might worth optimizing especially for web applications (url matching usually uses regexes) but also for other applications where I've seen many tight loops using regexes as well. So, I am probing interest on this topic here. What I (think I) know: - Both re and regex use the same C backend, which is not based on NFA. - The re2 library, which the writer of that article made, allows capture groups (but only up to a limit) and bounded repetitions (up to a limit). - Perl has started to optimize some regex patterns. [snip] re and regex use different C backends. Both are based on NFA. re2 is based on DFA, with a fallback to re. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Heads up: possible double-comments on bpo for commits
On Thu, Feb 2, 2017 at 6:01 PM, Victor Stinner wrote: > 2017-02-01 18:32 GMT+01:00 Victor Stinner : > > To finish with an empty change: > > Information on IRC: > > Known issue: http://psf.upfronthosting.co.za/roundup/meta/issue611 > > "Maciej is working on it" > > Good & thanks! > > I just fixed all the problems mentioned by Victor in this thread, this includes: 1. accidentally closing issues 2. multiple entries in history 3. empty date for gh commit messages (path by Berker - thank you) Ezio confirmed pushing all the changes to bugs.python.org already. I'm very sorry for all the inconvenience caused by previous mistakes. If your issues was closed make sure to verify it was closed properly and re-open if needed. Maciej ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Heads up: possible double-comments on bpo for commits
2017-02-02 22:24 GMT+01:00 Maciej Szulik : > Ezio confirmed pushing all the changes to bugs.python.org already. > I'm very sorry for all the inconvenience caused by previous mistakes. If > your > issues was closed make sure to verify it was closed properly and re-open if > needed. Don't worry. Brett warned us and it was quickly fixed. Thanks! Victor ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Generator objects and list comprehensions?
Craig Rodrigues : > Make this return a list on Python 3, like in Python 2: [(yield 1) for x in > range(10)] Give Python 2 a little more credit. Python 2.7.10 (default, May 23 2015, 09:40:32) [MSC v.1500 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> [(yield 1) for x in range(10)] File "", line 1 SyntaxError: 'yield' outside function regards, Anders ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Generator objects and list comprehensions?
> On Feb 2, 2017, at 2:17 AM, Anders Munch wrote: > > Give Python 2 a little more credit. We are, it told you what your issue was: yield outside a function. Consider: >>> def f(): ... l = [(yield 1) for x in range(10)] ... print(l) >>> gen = f() >>> for i in range(11): ... gen.send(i or None) ... 1 1 1 1 1 1 1 1 1 1 [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] But this is a very convoluted "feature" and likely people don't expect *this* to be what's happening. - Ł ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
