[Python-Dev] Idea: reduce GC threshold in development mode (-X dev)
Hi, Yury Selivanov pushed his implementation of the PEP 567 -- Context Variables at January 23, 2018. Yesterday, 4 months after the commit and only 3 weeks before 3.7.0 final release, a crash has been found in the implementation: https://bugs.python.org/issue33803 (it's now fixed, don't worry Ned!) The bug is a "common" mistake in an object constructor implemented in C: the object was tracked by the garbage collector before it was fully initialized, and a GC collection caused a crash somewhere in "object traversing". By "common", I mean that I saw this exact bug between 5 and 10 times over the last 5 years. In the bpo issue, I asked why we only spotted the bug yesterday? It seems like changing the threshold of the GC generation 0 from 700 to 5 triggers the bug immediately in test_context (tests of the PEP 567). I wrote a proof-of-concept patch to change the threshold when using -X dev. Question: Do you think that bugs spotted by a GC collection are common enough to change the GC thresholds in development mode (new -X dev flag of Python 3.7)? GC collections detect various kinds of bugs. Another "common" bug is when an object remains somehow alive in the GC whereas its memory has been freed: using PYTHONMALLOC=debug (debug feature already enabled by -X dev), a GC collection will always crash in such case. I'm not sure about the exact thresholds that would be used in development mode. The general question is more if it would be useful. Then the side question is if reducing the threshold would kill performances or not. About performances, -X dev allows to enable debug features which have an "acceptable" cost in term of performance and memory, but enabled features are chosen on a case by case basis. For example, I chose to *not* enable tracemalloc using -X dev because the cost in term of CPU *and* memory is too high (usually 2x slower and memory x2). Victor ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Keeping an eye on Travis CI, AppVeyor and buildbots: revert on regression
2018-06-07 4:45 GMT+02:00 Mariatta Wijaya : > Are there APIs we can use to check the status of builbots? Buildbots offer different ways to send notifications: emails and IRC bot for example. If you want to *poll* for recent builds, I don't know. I would suggest to use notifications (push) rather than polling. > Maybe we can have the our bots check for the buildbot status in backport > PRs. Right now, I'm not confortable with this idea because there is a risk to scare newcomers with false alarms and real bugs which are not coming from their changes, but may be known or are new but still unrelated to their changes. Moreover, even when a buildbot fails because of a real regression, a build may include multiple changes (I saw builds with more than 25 changes on some buildbots). Buildbot is different than a CI: slow buildbots are able to pack test multiple new changes at once. So again, there is a high risk of false alarms. Maybe I'm too conservative and we can try something with good documentation and proper warnings to explain properly such hypotetical notifications on pull requests. See also my other email which explains this differently: https://mail.python.org/pipermail/python-dev/2018-May/153759.html Victor ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Idea: reduce GC threshold in development mode (-X dev)
08.06.18 10:48, Victor Stinner пише: Yury Selivanov pushed his implementation of the PEP 567 -- Context Variables at January 23, 2018. Yesterday, 4 months after the commit and only 3 weeks before 3.7.0 final release, a crash has been found in the implementation: https://bugs.python.org/issue33803 (it's now fixed, don't worry Ned!) The bug is a "common" mistake in an object constructor implemented in C: the object was tracked by the garbage collector before it was fully initialized, and a GC collection caused a crash somewhere in "object traversing". By "common", I mean that I saw this exact bug between 5 and 10 times over the last 5 years. In the bpo issue, I asked why we only spotted the bug yesterday? It seems like changing the threshold of the GC generation 0 from 700 to 5 triggers the bug immediately in test_context (tests of the PEP 567). I wrote a proof-of-concept patch to change the threshold when using -X dev. Question: Do you think that bugs spotted by a GC collection are common enough to change the GC thresholds in development mode (new -X dev flag of Python 3.7)? GC collections detect various kinds of bugs. Another "common" bug is when an object remains somehow alive in the GC whereas its memory has been freed: using PYTHONMALLOC=debug (debug feature already enabled by -X dev), a GC collection will always crash in such case. I'm not sure about the exact thresholds that would be used in development mode. The general question is more if it would be useful. Then the side question is if reducing the threshold would kill performances or not. About performances, -X dev allows to enable debug features which have an "acceptable" cost in term of performance and memory, but enabled features are chosen on a case by case basis. For example, I chose to *not* enable tracemalloc using -X dev because the cost in term of CPU *and* memory is too high (usually 2x slower and memory x2). Reducing GC threshold can hide other bugs that will be reproducible only in the release mode (because of earlier releasing of resources or changed order of destroying objects). What is the cost of traversing all objects? Would it be too high if just traverse all objects every time when the garbage collecting potentially can happen, but without modifying any data, just check for consistency of GC headers? It may be worth to write also suggestions for testing extensions (including setting low GC threshold) and include them in the Devguide (for core developers) and the "Extending and Embedding" section of the documentation (for authors of third-party extensions). ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Keeping an eye on Travis CI, AppVeyor and buildbots: revert on regression
2018-06-04 18:31 GMT+02:00 Victor Stinner : > Quick update a few days later. > (...) > Except Windows 7 which has issues with test_asyncio and > multiprocessing tests because this buildbot is slow, it seems like > most CIs are now stable. The bug wasn't specific to this buildbot, it was a very old race condition in the Windows ProactorEventLoop which only started recently to trigger test_asyncio failures. See https://bugs.python.org/issue33694 for the details. > Known issues: > > * PPC64 Fedora 3.x, PPC64LE Fedora 3.x, s390x RHEL 3.x: > https://bugs.python.org/issue33630 > * AIX: always red > * USBan: experimental buildbot > * Alpine: platform not supported yet (musl issues) Except of these few CIs, the main issue was test__xxsubinterpreters which is known to crash: https://bugs.python.org/issue33615 To fix CIs and give more time to Eric Snow to look at this crash, I decided to skip test__xxsubinterpreters. You may want to offer help to Eric to look into these tricky issues. So again, except of the few known issues listed above, all other CIs (Travis CI, AppVeyor, all other buildbots) should now pass. Victor ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Idea: reduce GC threshold in development mode (-X dev)
2018-06-08 10:17 GMT+02:00 Serhiy Storchaka : > Reducing GC threshold can hide other bugs that will be reproducible only in > the release mode (because of earlier releasing of resources or changed order > of destroying objects). > > What is the cost of traversing all objects? Would it be too high if just > traverse all objects every time when the garbage collecting potentially can > happen, but without modifying any data, just check for consistency of GC > headers? Do you suggest to trigger a fake "GC collection" which would just visit all objects with a no-op visit callback? I like the idea! Yeah, that would help to detect objects in an inconsistent state and reuse the existing implemented visit methods of all types. Would you be interested to try to implement this new debug feature? Victor ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Idea: reduce GC threshold in development mode (-X dev)
08.06.18 11:31, Victor Stinner пише:
Do you suggest to trigger a fake "GC collection" which would just
visit all objects with a no-op visit callback? I like the idea!
Yeah, that would help to detect objects in an inconsistent state and
reuse the existing implemented visit methods of all types.
Would you be interested to try to implement this new debug feature?
It is simple:
#ifdef Py_DEBUG
void
_PyGC_CheckConsistency(void)
{
int i;
if (_PyRuntime.gc.collecting) {
return;
}
_PyRuntime.gc.collecting = 1;
for (i = 0; i < NUM_GENERATIONS; ++i) {
update_refs(GEN_HEAD(i));
}
for (i = 0; i < NUM_GENERATIONS; ++i) {
subtract_refs(GEN_HEAD(i));
}
for (i = 0; i < NUM_GENERATIONS; ++i) {
revive_garbage(GEN_HEAD(i));
}
_PyRuntime.gc.collecting = 0;
}
#endif
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Unicode 11.0.0 released
On Thu, Jun 07, 2018 at 06:02:40PM +0100, MRAB wrote: > The Python community _is_ meant to be inclusive, and we should support > the addition of ginger emoijs. :-) https://www.youtube.com/watch?v=KVN_0qvuhhw -- Steve ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Idea: reduce GC threshold in development mode (-X dev)
> On 8 Jun 2018, at 12:36, Serhiy Storchaka wrote:
>
> 08.06.18 11:31, Victor Stinner пише:
>> Do you suggest to trigger a fake "GC collection" which would just
>> visit all objects with a no-op visit callback? I like the idea!
>>
>> Yeah, that would help to detect objects in an inconsistent state and
>> reuse the existing implemented visit methods of all types.
>>
>> Would you be interested to try to implement this new debug feature?
>
> It is simple:
>
> #ifdef Py_DEBUG
> void
> _PyGC_CheckConsistency(void)
> {
> int i;
> if (_PyRuntime.gc.collecting) {
> return;
> }
> _PyRuntime.gc.collecting = 1;
> for (i = 0; i < NUM_GENERATIONS; ++i) {
> update_refs(GEN_HEAD(i));
> }
> for (i = 0; i < NUM_GENERATIONS; ++i) {
> subtract_refs(GEN_HEAD(i));
> }
> for (i = 0; i < NUM_GENERATIONS; ++i) {
> revive_garbage(GEN_HEAD(i));
> }
> _PyRuntime.gc.collecting = 0;
> }
> #endif
Wouldn’t it be enough to visit just the the newly tracked object in
PyObject_GC_Track with a visitor function that does something minimal to verify
that the object value is sane, for example by checking
PyType_Ready(Py_TYPE(op)).
That would find issues where objects are tracked before they are initialised
far enough to be save to visit, without changing GC behavior. I have no idea
what the performance impact of this is though.
Ronald
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Summary of Python tracker Issues
ACTIVITY SUMMARY (2018-06-01 - 2018-06-08) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open6691 ( +8) closed 38869 (+66) total 45560 (+74) Open issues with patches: 2640 Issues opened (39) == #23835: configparser does not convert defaults to strings https://bugs.python.org/issue23835 reopened by barry #24622: tokenize.py: missing EXACT_TOKEN_TYPES https://bugs.python.org/issue24622 reopened by skrah #33736: Improve the documentation of asyncio stream API https://bugs.python.org/issue33736 opened by Elvis.Pranskevichus #33738: PyIndex_Check conflicts with PEP 384 https://bugs.python.org/issue33738 opened by Christian.Tismer #33740: PyByteArray_AsString C-API description lacks the assurance, th https://bugs.python.org/issue33740 opened by realead #33741: UnicodeEncodeError onsmtplib.login(MAIL_USER, MAIL_PASSWORD) https://bugs.python.org/issue33741 opened by JustAnother1 #33742: Unsafe memory access in PyStructSequence_InitType https://bugs.python.org/issue33742 opened by Pasha Stetsenko #33745: 3.7.0b5 changes the line number of empty functions with docstr https://bugs.python.org/issue33745 opened by nedbat #33746: testRegisterResult in test_unittest fails in verbose mode https://bugs.python.org/issue33746 opened by serhiy.storchaka #33747: Failed separate test_patch_propogrates_exc_on_exit in test_uni https://bugs.python.org/issue33747 opened by serhiy.storchaka #33748: test_discovery_failed_discovery in test_unittest modifies sys. https://bugs.python.org/issue33748 opened by serhiy.storchaka #33751: Failed separate testTruncateOnWindows in test_file https://bugs.python.org/issue33751 opened by serhiy.storchaka #33754: f-strings should be part of the Grammar https://bugs.python.org/issue33754 opened by davidhalter #33757: Failed separate test_pdb_next_command_in_generator_for_loop in https://bugs.python.org/issue33757 opened by serhiy.storchaka #33758: Unexpected success of test_get_type_hints_modules_forwardref https://bugs.python.org/issue33758 opened by serhiy.storchaka #33762: temp file isn't IOBase https://bugs.python.org/issue33762 opened by Dutcho #33766: Grammar Incongruence https://bugs.python.org/issue33766 opened by Isaac Elliott #33770: base64 throws 'incorrect padding' exception when the issue is https://bugs.python.org/issue33770 opened by dniq #33771: Module: timeit. According to documentation default_repeat shou https://bugs.python.org/issue33771 opened by svyatoslav #33772: Fix few dead code paths https://bugs.python.org/issue33772 opened by David Carlier #33774: Document that @lru_cache caches based on exactly how the funct https://bugs.python.org/issue33774 opened by solstag #33775: argparse: the word 'default' (in help) is not marked as transl https://bugs.python.org/issue33775 opened by woutgg #33777: dummy_threading: .is_alive method returns True after execution https://bugs.python.org/issue33777 opened by njatkinson #33779: Error while installing python 3.6.5 on windows 10 https://bugs.python.org/issue33779 opened by sid1987 #33780: [subprocess] Better Unicode support for shell=True on Windows https://bugs.python.org/issue33780 opened by Yoni Rozenshein #33782: VSTS Windows-PR: internal error https://bugs.python.org/issue33782 opened by vstinner #33783: Use proper class markup for random.Random docs https://bugs.python.org/issue33783 opened by ncoghlan #33787: Argument clinic and Windows line endings https://bugs.python.org/issue33787 opened by giampaolo.rodola #33788: Argument clinic: use path_t in _winapi.c https://bugs.python.org/issue33788 opened by giampaolo.rodola #33793: asyncio: _ProactorReadPipeTransport reads by chunk of 32 KiB: https://bugs.python.org/issue33793 opened by vstinner #33797: json int encoding incorrect for dbus.Byte https://bugs.python.org/issue33797 opened by radsquirrel #33799: Remove non-ordered dicts comments from FAQ https://bugs.python.org/issue33799 opened by adelfino #33800: Fix default argument for parameter dict_type of ConfigParser/R https://bugs.python.org/issue33800 opened by adelfino #33801: Remove non-ordered dict comment from plistlib https://bugs.python.org/issue33801 opened by adelfino #33802: Regression in logging configuration https://bugs.python.org/issue33802 opened by barry #33804: Document the default value of the size parameter of io.TextIOB https://bugs.python.org/issue33804 opened by adelfino #33805: dataclasses: replace() give poor error message if using InitVa https://bugs.python.org/issue33805 opened by eric.smith #33808: ssl.get_server_certificate fails with openssl 1.1.0 but works https://bugs.python.org/issue33808 opened by dsanghan #33809: Expose `capture_locals` parameter in `traceback` convenience f https://bugs.python.org/issue33809 opened by ulope Most recent 15 issues with n
Re: [Python-Dev] Idea: reduce GC threshold in development mode (-X dev)
On Fri, Jun 8, 2018 at 9:24 AM Ronald Oussoren wrote: [..] > Wouldn’t it be enough to visit just the the newly tracked object in > PyObject_GC_Track with a visitor function that does something minimal to > verify that the object value is sane, for example by checking > PyType_Ready(Py_TYPE(op)). +1. Yury ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python3 compiled listcomp can't see local var - bug or feature?
On 06/06/2018 03:51 PM, Nick Coghlan wrote: > On 6 June 2018 at 15:31, Rob Cliffe via Python-Dev mailto:[email protected]>> wrote: > ... > *In other words, it looks as if in Python 3.6.5, the compiled list comprehension** > **can "see" a pre-existing global variable but not a local one.* > > Yes, this is expected behaviour - the two-namespace form of exec (which is what you get implicitly when you use it inside a function body) is similar to a class body, and hence nested functions > (including the implicit ones created for comprehensions) can't see the top level local variables. In issue 13557 [1] Amaury gives the following explanation by quoting the documentation [2] "Free variables are not resolved in the nearest enclosing namespace, but in the global namespace" and hints at the same solution that is proposed by Nick. FWIW in issue 21161 [3] folks have been bitten by this when trying to run a list comprehension in pdb. [1] https://bugs.python.org/issue13557 [2] http://docs.python.org/py3k/reference/executionmodel.html#interaction-with-dynamic-features [3] https://bugs.python.org/issue21161 Xavier ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Add __reversed__ methods for dict
Am I correct in saying that the consensus is +1 for inclusion in v3.8? The last point in the thread was INADA Naoki researching various implementations and deciding that it's OK to include this feature in 3.8. As I understand it, Guido was in agreement with INADA's advice to wait for MicroPython's implementation of v3.7. Since INADA has changed minds, I'm guessing it's all in favor? ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python3 compiled listcomp can't see local var - bug or feature?
> Is this a bug or a feature? The bug was me being so excited about the new construct (I pushed in someone else's work, can't recall who now, maybe Fredrik Lundh?) that I didn't consider that leaking the loop variable out of the list comprehension was a bad idea. Think of the Py3 behavior as one of those "corrections" to things which were "got wrong" in Python 1 or 2. :-) Skip ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Add __reversed__ methods for dict
That sounds right to me. We will then have had two versions where this was the case: - 3.6 where order preserving was implemented in CPython but in the language spec - 3.7 where it was also added to the language spec On Fri, Jun 8, 2018 at 11:05 AM, Michael Selik wrote: > Am I correct in saying that the consensus is +1 for inclusion in v3.8? > > The last point in the thread was INADA Naoki researching various > implementations and deciding that it's OK to include this feature in 3.8. > As I understand it, Guido was in agreement with INADA's advice to wait for > MicroPython's implementation of v3.7. Since INADA has changed minds, I'm > guessing it's all in favor? > > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > guido%40python.org > > -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 2018 Python Language Summit coverage
+1 -- thanks so much for acting as our scribe again! On Wed, Jun 6, 2018 at 5:47 PM, Eric Snow wrote: > Thanks for doing this Jake. > > -eric > > On Wed, Jun 6, 2018 at 3:56 PM, Jake Edge wrote: > > > > Hola python-dev, > > > > I have been remiss in posting about my coverage from this year's Python > > Language Summit -- not to mention remiss in getting it all written up. > > But I am about half-way done with the sessions from this year. > > > > I am posting SubscriberLinks for articles that are still behind the > > paywall. LWN subscribers can always see our content right away; one > > week after they are published in a weekly edition, they become freely > > available for everyone. SubscriberLinks are a way around the paywall. > > Please feel free to share the SubscriberLinks I am posting here. > > > > The starting point is here: https://lwn.net/Articles/754152/ That is > > an overview article with links to the articles. It will be updated as > > I add more articles. Here is what we have so far: > > > > - Subinterpreter support for Python https://lwn.net/Articles/754162/ > > > > - Modifying the Python object model https://lwn.net/Articles/754163/ > > > > - A Gilectomy update https://lwn.net/Articles/754577/ > > > > - Using GitHub Issues for Python https://lwn.net/Articles/754779/ > > > > - Shortening the Python release schedule > > https://lwn.net/Articles/755224/ > > > > - Unplugging old batteries > > https://lwn.net/SubscriberLink/755229/df78cf17181dbdca/ > > > > Hopefully I captured things reasonably well -- if you have corrections > > or clarifications (or just comments :) , I would recommend posting them > > as comments on the article. > > > > I will post an update soon with the next round (with luck, all of the > > rest of them). > > > > enjoy! > > > > jake > > > > -- > > Jake Edge - LWN - [email protected] - http://lwn.net > > ___ > > Python-Dev mailing list > > [email protected] > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > ericsnowcurrently%40gmail.com > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > guido%40python.org > -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
