[Python-Dev] Re: Tests now start with only 131 imported modules, instead of 233

2021-03-23 Thread Ivan Pozdeev via Python-Dev 

I didn't quite get what the big effect is. Saving 30 milliseconds?

Also, how is the now-split-off funcionality to be invoked? Does it require two or more imports now, or it's imported on demand when one 
invokes an appropriate test.support entry?


On 23.03.2021 4:29, Victor Stinner wrote:

Hi,

tl;dr Tests of the Python Test Suite now start with 131 imported
modules, instead of 233. For example, asyncio, logging,
multiprocessing and warnings are no longer imported by default.

--

The Python test suite is run by a custom test runner called
"libregrtest" (test.libregrtest). It can detect reference leaks, write
the output as JUnit XML, run tests in parallel with multiple
processes, etc. Tests are written with test.support which is a
collection of helper functions.

Over the years, test.support got more and more functions, and
libregrtest got more and more features. The problem is that they
import more and more modules. For example, "import test.support"
imports 173 modules in Python 3.8!

$ python3.8

import sys, sys
before=set(sys.modules); import test.support; after=set(sys.modules)
len(after - before)

173

In these imports, you can find some "heavy" modules like asyncio and
multiprocessing. Moreover, some modules have "side effects" on import.
For example, "import logging" registers an "at fork" callback.

In April 2020, I worked with Hai Shi on Python 3.10 in bpo-40275 to
reduce the number of test.support imports from 171 to 25! test.support
was splitted into multiple sub-modules:

- bytecode_helper
- hashlib_helper
- import_helper
- logging_helper
- os_helper
- script_helper
- socket_helper
- threading_helper
- warnings_helper

I continued the work in bpo-41718 to reduce the number of libregrtest
imports, since libregrtest also imported many modules.

A dummy test which does nothing in the master branch now only has 131
modules in sys.modules, whereas in Python 3.9 it has 233 (+102)
modules! For example, asyncio, logging, multiprocessing and warnings
are no longer imported by default.

"import test.support" is now faster. master branch compared to Python
3.8 using ./python -c 'import test.support' command and Python built
in release mode:

[py38] 85.7 ms +- 6.8 ms -> [master] 32.4 ms +- 1.3 ms: 2.64x faster

More realistic command running a single test method: ./python -m test
test_os -m test_access

[py38] 136 ms +- 5 ms -> [master] 97.8 ms +- 2.4 ms: 1.39x faster

The only annoying point is that splitting test.support into
sub-modules was not backported to Python 3.8 and 3.9, and so it will
make backports a little bit more annoying. Sorry about that!

--

If you know that two modules should be tested together, please write a
dedicated test for that ;-)

Victor


--
Regards,
Ivan

___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/3Q2VZVU3BEON36Z4JW6EUCMN4AZJYTF6/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Tests now start with only 131 imported modules, instead of 233

2021-03-23 Thread Victor Stinner 
Hi Ivan,

On Tue, Mar 23, 2021 at 9:04 AM Ivan Pozdeev via Python-Dev
 wrote:
> I didn't quite get what the big effect is. Saving 30 milliseconds?

I started to dig into this issue while debugging a random crash on AIX
(bpo-40091). A test_threading test using fork randomly crashed on AIX.
I discovered that the crash was triggered by the logging module. It
surprised me since threading and test_threading don't use the logging
module.

"import test.support" not only loads code but also has side effects
because it runs code. Some examples:

* "import logging" calls atexit.register() and os.register_at_fork().
* "import random" calls os.register_at_fork().
* "import readline" sets SIGWINCH signal handler.

In general, it's perfectly fine in tests to import modules which have
side effects. But I'm talking about the very specific case of the
Python test suite which tests low-level Python internals, any side
effect causes multiple annoying issues and makes debugging way harder.

I'm working for years on making the Python test suite more reliable.
These side effects indirectly make tests less reliable and less
reproducible.

IMO test_threading must not test the logging module. If someone wants
to test the relationship between logging, threading and fork, a new
dedicated test should be written in test_logging. In this case, there
are already fork tests in test_logging ;-)

By the way, I'm also working on fixing these random crashes ;-) For
example, I partially fixed the AIX crash at fork in a generic way by
adding new _at_fork_reinit() methods to locks: see also the underlying
_PyThread_at_fork_reinit() function.


> Also, how is the now-split-off funcionality to be invoked? Does it require 
> two or more imports now, or it's imported on demand when one
> invokes an appropriate test.support entry?

For example, TESTFN should now be get from test.support.os_helper.

Some tests use "from test.support.os_helper import TESTFN", other
tests prefer "from test.support import os_helper" and then use
"os_helper.TESTFN".

All tests in Lib/test/ have been updated last year, you don't have to
do anything.

By the way, the test module should not be used outside Python, it's
clearly specified in its documentatioin ;-) On Fedora, the "test"
module is not installed by default, but is in a separated python3-test
package. (Another reason is to reduce the disk space.)


I hope that it gives you a better idea why these changes were made ;-)

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/OVGPFLZ762MKCGXFCXI7Y6QNR5PKJF7L/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Tests now start with only 131 imported modules, instead of 233

2021-03-23 Thread Victor Stinner 
On Tue, Mar 23, 2021 at 9:04 AM Ivan Pozdeev via Python-Dev
 wrote:
> Also, how is the now-split-off funcionality to be invoked? Does it require 
> two or more imports now, or it's imported on demand when one
> invokes an appropriate test.support entry?

By the way, splitting test.support into sub-modules is also a way to
work around the lack of lazy import in Python. It would be nice to
have this feature, but so far, no consensus on the lazy import
specification was reached.

One issue is that some use cases need the import side effects to be
applied as soon as the import is done ... It goes against the
principle of lazy import :-( I don't recall the details.

Victor
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/KZERLO3VIUG37AS4DIUITFSJHFU4DDSF/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] typo in PEP 636

2021-03-23 Thread codeofdusk 
Hello,

The GUI section of PEP 636 reads: "It will also requires that the event has
a position attribute that matches the (x, y) pattern". It should read "It
will also require that the event has a position attribute that matches the
(x, y) pattern".

 

Thanks,

Bill

___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/GD27GMRKSAXTIIHKFCQMICEAAUILKNAS/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: typo in PEP 636

2021-03-23 Thread Jelle Zijlstra 
Submitted a PR to fix it: https://github.com/python/peps/pull/1891. In the
future, feel free to submit a PR directly to the PEP repo for this sort of
minor fix.

El mar, 23 mar 2021 a las 10:22,  escribió:

> Hello,
>
> The GUI section of PEP 636 reads: “It will also requires that the event
> has a position attribute that matches the (x, y) pattern”. It should read
> “It will also require that the event has a position attribute that matches
> the (x, y) pattern”.
>
>
>
> Thanks,
>
> Bill
> ___
> Python-Dev mailing list -- [email protected]
> To unsubscribe send an email to [email protected]
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/[email protected]/message/GD27GMRKSAXTIIHKFCQMICEAAUILKNAS/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/LTTTLH64IFRI7G2V6JXXD565C4WBG7FW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: typo in PEP 636

2021-03-23 Thread Guido van Rossum 
Thanks for your keen eye, Bill!

If you want to, you can submit a PR to the peps repo (
https://github.com/python/peps) to fix the typo directly.

On Tue, Mar 23, 2021 at 10:25 AM  wrote:

> Hello,
>
> The GUI section of PEP 636 reads: “It will also requires that the event
> has a position attribute that matches the (x, y) pattern”. It should read
> “It will also require that the event has a position attribute that matches
> the (x, y) pattern”.
>
>
>
> Thanks,
>
> Bill
> ___
> Python-Dev mailing list -- [email protected]
> To unsubscribe send an email to [email protected]
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/[email protected]/message/GD27GMRKSAXTIIHKFCQMICEAAUILKNAS/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/OUIXIKKM2EWDM372FLGZ7GVBFDSMWZEU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: typo in PEP 636

2021-03-23 Thread Guido van Rossum 
Sorry, I see Jelle already did this.
https://github.com/python/peps/pull/1891/files

On Tue, Mar 23, 2021 at 11:13 AM Guido van Rossum  wrote:

> Thanks for your keen eye, Bill!
>
> If you want to, you can submit a PR to the peps repo (
> https://github.com/python/peps) to fix the typo directly.
>
> On Tue, Mar 23, 2021 at 10:25 AM  wrote:
>
>> Hello,
>>
>> The GUI section of PEP 636 reads: “It will also requires that the event
>> has a position attribute that matches the (x, y) pattern”. It should read
>> “It will also require that the event has a position attribute that matches
>> the (x, y) pattern”.
>>
>>
>>
>> Thanks,
>>
>> Bill
>> ___
>> Python-Dev mailing list -- [email protected]
>> To unsubscribe send an email to [email protected]
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/[email protected]/message/GD27GMRKSAXTIIHKFCQMICEAAUILKNAS/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
>
> --
> --Guido van Rossum (python.org/~guido)
> *Pronouns: he/him **(why is my pronoun here?)*
> 
>


-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/CZQX6EQEUE2JNW4WX5Z6ZWWNUUB2JNZM/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: typo in PEP 636

2021-03-23 Thread codeofdusk 
Thanks all! Approved the PR.

 

Bill

 

From: Guido van Rossum  
Sent: Tuesday, 23 March 2021 14:14
To: [email protected]
Cc: Python-Dev 
Subject: Re: [Python-Dev] typo in PEP 636

 

Sorry, I see Jelle already did this. 
https://github.com/python/peps/pull/1891/files

 

On Tue, Mar 23, 2021 at 11:13 AM Guido van Rossum mailto:[email protected]> > wrote:

Thanks for your keen eye, Bill!

 

If you want to, you can submit a PR to the peps repo 
(https://github.com/python/peps) to fix the typo directly.

 

On Tue, Mar 23, 2021 at 10:25 AM mailto:[email protected]> > wrote:

Hello,

The GUI section of PEP 636 reads: “It will also requires that the event has a 
position attribute that matches the (x, y) pattern”. It should read “It will 
also require that the event has a position attribute that matches the (x, y) 
pattern”.

 

Thanks,

Bill

___
Python-Dev mailing list -- [email protected]  
To unsubscribe send an email to [email protected] 
 
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/GD27GMRKSAXTIIHKFCQMICEAAUILKNAS/
Code of Conduct: http://python.org/psf/codeofconduct/



-- 

--Guido van Rossum (python.org/~guido  )

Pronouns: he/him  

 (why is my pronoun here?)



-- 

--Guido van Rossum (python.org/~guido  )

Pronouns: he/him  

 (why is my pronoun here?)

___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/EAGA37FB5R2MEBPDHFFFUFTUGSYSVTL7/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Steering Council reply regarding conduct (was Re: Steering Council update for February)

2021-03-23 Thread Python Steering Council 
From Thomas Wouters, on behalf of and with full support of the Python Steering 
Council:

I’ll post a separate reply about the merits of the decision, but we have to 
talk about this post from Steven D’Aprano in particular.

Steven, this reply -- its tone and its message -- are completely unacceptable. 
It is offensive, dismissive and insulting. It’s a bad faith argument ridiculing 
a good faith effort. It is particularly insulting that you take the time to 
draw false equivalences to arguments that weren’t made on this list, and that 
you’re assuming motivation and ridiculing it, without taking the time to engage 
in the actual arguments. There are plenty of sources online that you could’ve 
used to learn more, or you could have just asked in this thread. Instead, you 
posted this sarcastic, denigrating, mocking reply.

Yours isn’t the only post that crosses the line in this thread, but it is the 
worst. It is an example of the kind of discourse that is too common and that 
reflects very badly on python-dev. It creates an atmosphere of disrespect and 
abuse, and actively scares away both experienced contributors and new ones. 
This is not conjecture; I and others on the SC have heard from more than a 
dozen people that this is exactly the case.

This is not acceptable. It has to stop.

The PSF Code of Conduct WG has sent the SC a recommendation on this issue. It 
recommended a temporary ban from python-dev for Steven, and a warning for 
someone else. While the SC believes a temporary ban would certainly be a 
reasonable action, we decided to instead make this public statement, as a 
warning to everyone. The SC wants to be clear that this statement applies to 
everyone who has made uncivil and inflammatory posts.

This tone is not acceptable from anyone. The SC wants to make this very clear: 
this is a final warning to remind everyone that this behaviour will result in 
strong measures, including bans and ejections. If you want to argue something, 
do so on the merits, in good faith, and with empathy and consideration. If you 
can only ridicule someone, their motivation or their lived experience, please 
keep it to yourself.


> On Tue, Mar 09, 2021 at 08:27:19PM +, Pablo Galindo Salgado wrote:
> 
> >The Steering Council discussed renaming the master branch to main and
> >the consensus was that we should do that. 
> 
> And about time too. Can we now tackle some of the equally pressing use 
> of offensive terms that are common in the Python community, starting 
> with the name of the language itself?
> 
> Pythons are snakes, which is triggering to people with a phobia of 
> snakes. About one third of all people, or more than two *billion* 
> people, suffer from some level of phobia towards snakes.
> 
> The popular "nose" testing framework is a blatant antisemetic and 
> neo-nazi dog whistle.
> 
> "bool" is named after George Boole, a problematic white man who 
> appropriated the culture of both the Middle East and East Asia.
> 
> "dict" is confusable with, and is often abbreviated to, an offensive 
> word. And don't even get me started with the obvious sexism of "tty".
> 
> Unicode is racist because it has unified Chinese, Japanese and Korean 
> characters as if they were the same thing, and relegates non-Western 
> languages to second class status:
> 
> https://modelviewculture.com/pieces/i-can-text-you-a-pile-of-poo-but-i-cant-write-my-name
> 
> It also includes a teddy bear symbol, which is named after notorious 
> racist and imperialist Theodore Roosevelt, and no less than *six* 
> swastika symbols. Also the Cross of Jerusalem, the symbol of such openly 
> fascist groups as the Federal State of Austria during the 1930s and the 
> Russian far-right extremist organisation the People's National Party.
> 
> It even has a symbol for chains, which is associated even more closely 
> with slavery than "master".
> 
> Speaking of slavery, in the standard library we have ChainMap and 
> itertools.chain.
> 
> We have the ableist "runpy", and in the random module a function named 
> after Vilfredo Pareto, who supported the rule of fascist dictator Benito 
> Mussolini. There are the token and tokenize modules, which are offensive 
> for their association with both sexist and racist views.
> 
> The tarfile module is associated with the racist Uncle Remus stories, 
> and a derogatory term for US Blacks.
> 
> https://www.washingtonpost.com/blogs/compost/post/doug-lamborns-tar-baby-quagmire/2011/08/03/gIQAKmXnsI_blog.html
> 
> The textwrap module uses a derogatory racist and fatphobic term dozens 
> of times.
> 
> http://rsdb.org/slur/chunk
> 
> Each of these issues are just as important as the "master" issue.
> 
> 
> -- 
> Steve

___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/pytho

[Python-Dev] Re: Steering Council reply regarding conduct (was Re: Steering Council update for February)

2021-03-23 Thread Martin Dengler 

On Tue, Mar 23, 2021 at 12:02:38PM -0700, Python Steering Council wrote:

From Thomas Wouters, on behalf of and with full support of the Python Steering
Council:
[use of SC power; specifically, PEP-0013.Powers.2: 'Enforce ... code of 
conduct']


From PEP 13[^1]

To use its powers, the council votes.

[...]

Whenever possible, the council's deliberations and votes shall be held in 
public.


Please share the deliberations and votes.

Martin

[^1]: https://github.com/python/peps/blob/master/pep-0013.rst
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/WWVV7ONKG7GD73JJ7YBPHTSRH2PCRY3U/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] On the migration from master to main

2021-03-23 Thread Python Steering Council 
From Thomas Wouters, on behalf of and with full support of the Python Steering 
Council:

This discussion seems to have died down a little, but I still want to make a 
few things clear:

Yes, this is a political decision. Very many decisions are political. The 
existence of an open-source project is inherently political. The decision to 
try and make python-dev more welcoming, more open, more helpful is also a 
political decision -- one that the SC feels is absolutely necessary for the 
long-term health of the Python language. Not wanting to be bothered by 
political decisions is a political decision; it’s a decision that you’re happy 
with politics as they are. I’m afraid you can’t avoid politics.

This isn’t just about ‘master’ being rooted in slavery. This is about what the 
community sees and does. As I mentioned before, we’re not leading the pack in 
this, we’re merely following along with others (like, say, Django). There are 
undoubtedly other terms and practices that are genuinely offensive, and the 
decision on how to handle them will be taken on a case-by-case basis, weighing 
the cost and the benefit in each case.

While you may feel the benefit of this change is small and that it has no real 
impact, we believe that there is little cost to making this change. We believe 
this change, while a minor inconvenience to some, helps demonstrate our 
commitment to acting in the best interests of Python's future. Failure to make 
a small sacrifice, such as this, signals that the Python core development 
community would be unlikely to undertake real change for greater benefits.

This isn’t happening because GitHub/Microsoft made a political decision. It’s 
happening because it is incredibly easy to make this move, many projects have 
already done this, and it reflects badly on any project not making this change.

Speaking for the whole SC in this,
Thomas.

___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/J5GR6YVIVWQ2VPLISAGBQH3UQN4YWAXS/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Merge request reminder.

2021-03-23 Thread Alfred Perlstein 
Hello, a while back we hit a bug with doctest and mock.call that 
confused a few people on our team for a while.


I dug into fixing it and made doctest more defensive around 
un-unwrappable calls.


I submitted a PR (https://github.com/python/cpython/pull/22981) for an 
Issue (https://bugs.python.org/issue35753) back in October 25, 2020 
which got some review, and then have not heard for several months.


I am wondering if it can be reviewed and feedback given on a way forward.

Thank you,

-Alfred

___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/DQTVEJGORWXPBLXG2P5S6YZBLCEL2DG3/
Code of Conduct: http://python.org/psf/codeofconduct/