[Python-Dev] Re: Multiple reference leaks in master

2019-12-04 Thread Victor Stinner 
Le mer. 4 déc. 2019 à 01:45, Charalampos Stratakis
 a écrit :
> Also noting here that the reference leaks builds are triggered once a day so 
> many times it's not possible to find the commit that triggered a leak through 
> the web ui of buildbot. In this case it's good to also check the previous 
> builds of the same worker to figure out what changed.

Once a day is a tradeoff to not stress buildbot workers too often. In
my experience, it's quite easy to look at changes made the day before
and use bisect tools to identify the leak.

I'm using different tools:

* Refleak buildbots tell you which tests leak
* ./python -m test.bisect_cmd -R 3:3 test_xxx: identify one test
method which is enough to trigger the leak in test_xxx. This speedup a
lot following steps to check if a commit has the leak or not (git
bisect), since -R 3:3 makes tests 6x slower.
* git bisect helps a lot to identify which commit introduced the leak

Overall, it's rare than it takes me longer than 30 minutes to identify
the commit and the leaking tests and test methods. Sometimes, it's
done in under 10 minutes. It's just a few simple command and wait
until the computer works for me :-) (test.bisect_cmd that I wrote
helps me a lot!)

Pablo and me are opened issues to reference leak regressions. If
possible, we comment the issue introducing the regression, or we open
new issues if it's too hard to identify the leak.

The worst is when the leak is not a recent regression, but it was
hidden for whatever reasons and can now be reproduced in Python 6
months ago, if not older. But that's the least common case.

Note: test.bisect_cmd can also be used to identify the test method
responsible for a "altered environment" (ENV_CHANGED) if you add
--fail-env-changed option.

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/RTXIHFKXANABGUPMGFYDXDKLDWBAC6WM/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Multiple reference leaks in master

2019-12-03 Thread Charalampos Stratakis 



- Original Message -
> From: "Pablo Galindo Salgado" 
> To: "Python Dev" 
> Sent: Tuesday, December 3, 2019 10:14:01 PM
> Subject: [Python-Dev] Multiple reference leaks in master
> 
> Hi,
> 
> Me (and Victor) have not been able to attend the buildbots for a while
> unfortunately and today
> I was checking them out after some fixes to the SSL tests and sadly the
> refleaks buildbots have many
> independent issues. At least these tests are failing due to reference leaks:
> 
> test__xxsubinterpreters, test_atexit, test_capi, test_httpservers,
> test_threading
> 
> I am trying to slowly fix them but finding them and debugging often takes
> several hours per leak. I was tracking them
> in https://bugs.python.org/issue38962 until I discover that there are
> multiple sources of leaks so I may open more issues.
> 
> Please, when you merge a PR, take a look at the buildbots (especially the
> refleak buildbots that do not report still to the PR
> in case of failure) so the issues don't keep piling up, as multiple refleaks
> together are much difficult to deal with.
> 
> Thanks, everyone!
> 
> Regards from cloudy London,
> Pablo Galindo Salgado
> 
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/6DWAFYZGH3X34VJHSOUVCEAHLZ4KZNPQ/
> Code of Conduct: http://python.org/psf/codeofconduct/
> 

Also noting here that the reference leaks builds are triggered once a day so 
many times it's not possible to find the commit that triggered a leak through 
the web ui of buildbot. In this case it's good to also check the previous 
builds of the same worker to figure out what changed.

-- 
Regards,

Charalampos Stratakis
Software Engineer
Python Maintenance Team, Red Hat
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZP25JNHP5R5VIAQFYXTWANXSETW4V6MC/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Multiple reference leaks in master

2019-12-03 Thread Pablo Galindo Salgado 
> Maybe you or Victor can update the devguide with this information? This
is not the first time that we've been asked to pay attention to the
refleaks buildbots, and it's frustrating that they are so hard to find.

Good point! I will make a PR to the dev guide improving the existing
documentation and I will think of other ways to make discovery easier.

On Wed, 4 Dec 2019 at 00:00, Guido van Rossum  wrote:

> Maybe you or Victor can update the devguide with this information? This is
> not the first time that we've been asked to pay attention to the refleaks
> buildbots, and it's frustrating that they are so hard to find.
>
> On Tue, Dec 3, 2019 at 2:17 PM Pablo Galindo Salgado 
> wrote:
>
>> > How do I find the refleak buildbots?
>>
>> In this page:
>>
>> https://buildbot.python.org/all/#/builders
>>
>> all the buildbots that have "Refleaks" in the name. You can click on the
>> tags to filter only the "stable" and the ones in master "3.x".
>>
>> On Tue, 3 Dec 2019 at 22:11, Guido van Rossum  wrote:
>>
>>> How do I find the refleak buildbots? I went to the devguide and searched
>>> for "buildbot", which pointed to https://www.python.org/dev/buildbot/
>>> -- but searching there for "refleak" finds nothing.
>>>
>>> On Tue, Dec 3, 2019 at 1:16 PM Pablo Galindo Salgado <
>>> pablog...@gmail.com> wrote:
>>>
 Hi,

 Me (and Victor) have not been able to attend the buildbots for a while
 unfortunately and today
 I was checking them out after some fixes to the SSL tests and sadly the
 refleaks buildbots have many
 independent issues. At least these tests are failing due to reference
 leaks:

 test__xxsubinterpreters, test_atexit, test_capi, test_httpservers,
 test_threading

 I am trying to slowly fix them but finding them and debugging often
 takes several hours per leak. I was tracking them
 in https://bugs.python.org/issue38962 until I discover that there are
 multiple sources of leaks so I may open more issues.

 Please, when you merge a PR, take a look at the buildbots (especially
 the refleak buildbots that do not report still to the PR
 in case of failure) so the issues don't keep piling up, as multiple
 refleaks together are much difficult to deal with.

 Thanks, everyone!

 Regards from cloudy London,
 Pablo Galindo Salgado

 ___
 Python-Dev mailing list -- python-dev@python.org
 To unsubscribe send an email to python-dev-le...@python.org
 https://mail.python.org/mailman3/lists/python-dev.python.org/
 Message archived at
 https://mail.python.org/archives/list/python-dev@python.org/message/6DWAFYZGH3X34VJHSOUVCEAHLZ4KZNPQ/
 Code of Conduct: http://python.org/psf/codeofconduct/

>>>
>>>
>>> --
>>> --Guido van Rossum (python.org/~guido)
>>> *Pronouns: he/him **(why is my pronoun here?)*
>>> 
>>>
>>
>
> --
> --Guido van Rossum (python.org/~guido)
> *Pronouns: he/him **(why is my pronoun here?)*
> 
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6AK5ZLHHZL5M2OTJUYQGEX3DSXBTH3N6/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Multiple reference leaks in master

2019-12-03 Thread Guido van Rossum 
Maybe you or Victor can update the devguide with this information? This is
not the first time that we've been asked to pay attention to the refleaks
buildbots, and it's frustrating that they are so hard to find.

On Tue, Dec 3, 2019 at 2:17 PM Pablo Galindo Salgado 
wrote:

> > How do I find the refleak buildbots?
>
> In this page:
>
> https://buildbot.python.org/all/#/builders
>
> all the buildbots that have "Refleaks" in the name. You can click on the
> tags to filter only the "stable" and the ones in master "3.x".
>
> On Tue, 3 Dec 2019 at 22:11, Guido van Rossum  wrote:
>
>> How do I find the refleak buildbots? I went to the devguide and searched
>> for "buildbot", which pointed to https://www.python.org/dev/buildbot/ --
>> but searching there for "refleak" finds nothing.
>>
>> On Tue, Dec 3, 2019 at 1:16 PM Pablo Galindo Salgado 
>> wrote:
>>
>>> Hi,
>>>
>>> Me (and Victor) have not been able to attend the buildbots for a while
>>> unfortunately and today
>>> I was checking them out after some fixes to the SSL tests and sadly the
>>> refleaks buildbots have many
>>> independent issues. At least these tests are failing due to reference
>>> leaks:
>>>
>>> test__xxsubinterpreters, test_atexit, test_capi, test_httpservers,
>>> test_threading
>>>
>>> I am trying to slowly fix them but finding them and debugging often
>>> takes several hours per leak. I was tracking them
>>> in https://bugs.python.org/issue38962 until I discover that there are
>>> multiple sources of leaks so I may open more issues.
>>>
>>> Please, when you merge a PR, take a look at the buildbots (especially
>>> the refleak buildbots that do not report still to the PR
>>> in case of failure) so the issues don't keep piling up, as multiple
>>> refleaks together are much difficult to deal with.
>>>
>>> Thanks, everyone!
>>>
>>> Regards from cloudy London,
>>> Pablo Galindo Salgado
>>>
>>> ___
>>> Python-Dev mailing list -- python-dev@python.org
>>> To unsubscribe send an email to python-dev-le...@python.org
>>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>>> Message archived at
>>> https://mail.python.org/archives/list/python-dev@python.org/message/6DWAFYZGH3X34VJHSOUVCEAHLZ4KZNPQ/
>>> Code of Conduct: http://python.org/psf/codeofconduct/
>>>
>>
>>
>> --
>> --Guido van Rossum (python.org/~guido)
>> *Pronouns: he/him **(why is my pronoun here?)*
>> 
>>
>

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UHSOYEXKJXLEZGGU7QZ5DS5Z7OM4S6FV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Multiple reference leaks in master

2019-12-03 Thread Pablo Galindo Salgado 
> How do I find the refleak buildbots?

In this page:

https://buildbot.python.org/all/#/builders

all the buildbots that have "Refleaks" in the name. You can click on the
tags to filter only the "stable" and the ones in master "3.x".

On Tue, 3 Dec 2019 at 22:11, Guido van Rossum  wrote:

> How do I find the refleak buildbots? I went to the devguide and searched
> for "buildbot", which pointed to https://www.python.org/dev/buildbot/ --
> but searching there for "refleak" finds nothing.
>
> On Tue, Dec 3, 2019 at 1:16 PM Pablo Galindo Salgado 
> wrote:
>
>> Hi,
>>
>> Me (and Victor) have not been able to attend the buildbots for a while
>> unfortunately and today
>> I was checking them out after some fixes to the SSL tests and sadly the
>> refleaks buildbots have many
>> independent issues. At least these tests are failing due to reference
>> leaks:
>>
>> test__xxsubinterpreters, test_atexit, test_capi, test_httpservers,
>> test_threading
>>
>> I am trying to slowly fix them but finding them and debugging often takes
>> several hours per leak. I was tracking them
>> in https://bugs.python.org/issue38962 until I discover that there are
>> multiple sources of leaks so I may open more issues.
>>
>> Please, when you merge a PR, take a look at the buildbots (especially the
>> refleak buildbots that do not report still to the PR
>> in case of failure) so the issues don't keep piling up, as multiple
>> refleaks together are much difficult to deal with.
>>
>> Thanks, everyone!
>>
>> Regards from cloudy London,
>> Pablo Galindo Salgado
>>
>> ___
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-le...@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-dev@python.org/message/6DWAFYZGH3X34VJHSOUVCEAHLZ4KZNPQ/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
>
> --
> --Guido van Rossum (python.org/~guido)
> *Pronouns: he/him **(why is my pronoun here?)*
> 
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/4EB7QF7VV4LRJ4XRHYERJI5327OHGYHW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Multiple reference leaks in master

2019-12-03 Thread Guido van Rossum 
How do I find the refleak buildbots? I went to the devguide and searched
for "buildbot", which pointed to https://www.python.org/dev/buildbot/ --
but searching there for "refleak" finds nothing.

On Tue, Dec 3, 2019 at 1:16 PM Pablo Galindo Salgado 
wrote:

> Hi,
>
> Me (and Victor) have not been able to attend the buildbots for a while
> unfortunately and today
> I was checking them out after some fixes to the SSL tests and sadly the
> refleaks buildbots have many
> independent issues. At least these tests are failing due to reference
> leaks:
>
> test__xxsubinterpreters, test_atexit, test_capi, test_httpservers,
> test_threading
>
> I am trying to slowly fix them but finding them and debugging often takes
> several hours per leak. I was tracking them
> in https://bugs.python.org/issue38962 until I discover that there are
> multiple sources of leaks so I may open more issues.
>
> Please, when you merge a PR, take a look at the buildbots (especially the
> refleak buildbots that do not report still to the PR
> in case of failure) so the issues don't keep piling up, as multiple
> refleaks together are much difficult to deal with.
>
> Thanks, everyone!
>
> Regards from cloudy London,
> Pablo Galindo Salgado
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/6DWAFYZGH3X34VJHSOUVCEAHLZ4KZNPQ/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/23VHKPOXDRI2CD52VCE2YU3CTTIHGIDO/
Code of Conduct: http://python.org/psf/codeofconduct/