[Python-Dev] Final reminder: 3.7.8 / 3.6.11 cutoff Monday, last non-security bugfixes for 3.7.x

[Ned Deily]

We will be accecpting commits to the 3.7 branch until this coming Monday 06-15 
23:59 AOE. At that time, 3.7 will effectively enter security-fix mode. That 
will also be the deadline for any additional security-fixes for release in 
3.6.11. Release candidates will follow shortly thereafter. Assuming no 
release-blocker issues arise during the release candidate testing, 3.7.8 and 
3.6.11 will release on 06-27.

https://discuss.python.org/t/upcoming-cutoffs-for-3-9-3-8-3-7-and-3-6-releases-last-chance-for-3-7-bugfixes/4362

--
  Ned Deily
  n...@python.org -- []
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/EKYBL5VG6TWSTTQUNBP56VJVHDR3GRM3/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] When can we remove wchar_t* cache from string?

[Inada Naoki]

Hi, all.

Py_UNICODE has been deprecated since PEP 393 (Flexible string representation).

wchar_t* cache in the string object is used only in deprecated APIs.
It waste 1 word (8 bytes on 64bit machine) per string instance.

The deprecated APIs are documented as "Deprecated since version 3.3,
will be removed in version 4.0."
See https://docs.python.org/3/c-api/unicode.html#deprecated-py-unicode-apis

But when PEP 393 is implemented, no one expects 3.10 will be released.
Can we reschedule the removal?

My proposal is, schedule the removal on Python 3.11.  But we will postpone
the removal if we can not remove its usage until it.

I grepped the use of the deprecated APIs from top 4000 PyPI packages.

result: 
https://github.com/methane/notes/blob/master/2020/wchar-cache/deprecated-use
step: https://github.com/methane/notes/blob/master/2020/wchar-cache/README.md

I noticed:

* Most of them are generated by Cython.
  * I reported it to Cython so Cython 0.29.21 will fix them.  I expect
more than 1 year
between Cython 0.29.21 and Python 3.11rc1.
* Most of them are `PyUnicode_FromUnicode(NULL, 0);`
  * We may be able to keep PyUnicode_FromUnicode, but raise error when length>0.

Regards,

--
Inada Naoki  
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/7JVC3IKS2V73K36ISEJAAWMRFN2T4KKR/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: My take on multiple interpreters (Was: Should we be making so many changes in pursuit of PEP 554?)

[Mark Shannon]

On 11/06/2020 2:50 pm, Riccardo Ghetta wrote:

Hello Mark,
and thanks for your suggestions. However, I'm afraid I haven't explained 
our use of python well enough.


On 11/06/2020 12:59, Mark Shannon wrote:

If you need to share objects across threads, then there will be
contention, regardless of how many interpreters there are, or which
processes they are in.
As a rule, we don't use that many python objects.  Most of the time a 
script calls C++ functions, operating on C++ data.

Perhaps with a small snippet I will explain myself better :

 hcpi='INFLEUR'
 n_months=3
 base_infl=hs_base(hcpi, n_months, 0)
 im=hs_fs(hcpi,'sia','m',n_months,0)
 ip=hs_fs(hcpi,'sia','m',n_months-1,0)
 ir=im+(hs_range()[1].day-1)/month_days(hs_range()[1])*(ip-im)
 return ir/base_infl    # double

this is a part of a inflation estimation used in pricing an 
inflation-linked bond.
hcpi and n_months are really parameters of the script and the hs_ 
functions are all implemented in C++.
Some are very small and fast like hs_range, others are much more complex 
and slow (hs_fs), so we wrap them with 
Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS.
As you see, here python is used more to direct C++, than manipulate 
objects.  At GUI level things work a bit differently, but here we just 
tried to avoid building and destroying a lot of ephemeral python objects 
(unneeded anyway, because all subsequent processing is done by C++).
This python script is only a part of a larger processing done in 
parallel by several threads, each operating in distinct instruments.
Evaluating an instrument could involve zero, one, or several of those 
scripts.
During evaluation an instrument is bound to a single thread, so from the 
point of view of python threads share nothing.



If the additional resource consumption is irrelevant, what's the
objection to spinning up a new processes?
The additional resource consumption of a new python interpreter is 
irrelevant, but the process as a whole needs a lot of extra data making 
a new process rather costly.


Starting a new process is cheap. On my machine, starting a new Python 
process takes under 1ms and uses a few Mbytes.


The overhead largely comes from what you do with the process. The 
additional cost of starting a new interpreter is the same regardless of 
whether it is in the same process or not. There should be no need to 
start a new application process for a new Python interpreter.


Plus there are issues of licensing, synchronization and load balancing 
that are much easier to resolve (for our system, at least) with threads 
than processes.


Would this prevent CPython starting new processes, or is this just for 
processes managed by your application?


Still, we /do/ use multiple processes, but those tend to be across 
administrative boundaries, or for very specific issues.


Ciao,
Riccardo


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/2AG665AKKLNYOXO2RNE53TOQ626PBIM7/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: My take on multiple interpreters (Was: Should we be making so many changes in pursuit of PEP 554?)

[Paul Moore]

On Fri, 12 Jun 2020 at 09:47, Mark Shannon  wrote:
> Starting a new process is cheap. On my machine, starting a new Python
> process takes under 1ms and uses a few Mbytes.

Is that on Windows or Unix? Traditionally, process creation has been
costly on Windows, which is why threads, and in-process solutions in
general, tend to be more common on that platform. I haven't done
experiments recently, but I do tend to avoid multiprocess-type
solutions on Windows "just in case". I know that evaluating a new
feature based on unsubstantiated assumptions informed by "it used to
be like this" is ill-advised, but so is assuming that everything will
be OK based on experience on a single platform :-)

Personally, I'm in favour of multiple interpreter support mostly for
the same reasons as Petr (easy embedding, in the style of Lua).
Exposing interpreters to Python, and per-interpreter GILs, strike me
as really interesting areas for experimentation, but I'm reserving
final judgement on the practical benefits until we have working code
and some practical experience. The incremental costs for those are
low, though, as the bulk of the work is actually needed for the "easy
embedding" use case.

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZSNO3WRBWLE76W7K4CQSIEWPQIN5AJDG/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: When can we remove wchar_t* cache from string?

[Mark Shannon]

Hi,

A big +1 to exposing fewer internals of the PyUnicodeObject to C code.

Ultimately, making PyUnicodeObject immutable to C code would be a real 
bonus. It would make the code cleaner, safer and faster. A triple win!


I don't think removing Py_UNICODE API be sufficient for that, have you 
thoughts on what else would need to change?


On 12/06/2020 9:32 am, Inada Naoki wrote:

Hi, all.

Py_UNICODE has been deprecated since PEP 393 (Flexible string representation).

wchar_t* cache in the string object is used only in deprecated APIs.
It waste 1 word (8 bytes on 64bit machine) per string instance.

The deprecated APIs are documented as "Deprecated since version 3.3,
will be removed in version 4.0."
See https://docs.python.org/3/c-api/unicode.html#deprecated-py-unicode-apis

But when PEP 393 is implemented, no one expects 3.10 will be released.
Can we reschedule the removal?

My proposal is, schedule the removal on Python 3.11.  But we will postpone
the removal if we can not remove its usage until it.

I grepped the use of the deprecated APIs from top 4000 PyPI packages.

result: 
https://github.com/methane/notes/blob/master/2020/wchar-cache/deprecated-use
step: https://github.com/methane/notes/blob/master/2020/wchar-cache/README.md

I noticed:

* Most of them are generated by Cython.
   * I reported it to Cython so Cython 0.29.21 will fix them.  I expect
more than 1 year
 between Cython 0.29.21 and Python 3.11rc1.
* Most of them are `PyUnicode_FromUnicode(NULL, 0);`
   * We may be able to keep PyUnicode_FromUnicode, but raise error when 
length>0.

Regards,

--
Inada Naoki  
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/7JVC3IKS2V73K36ISEJAAWMRFN2T4KKR/
Code of Conduct: http://python.org/psf/codeofconduct/


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TSL2C4HQDHQGXBSNP2WF2AJNNV4A55S7/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: My take on multiple interpreters (Was: Should we be making so many changes in pursuit of PEP 554?)

[Edwin Zimmerman]

On 6/12/2020 5:08 AM, Paul Moore wrote:
> On Fri, 12 Jun 2020 at 09:47, Mark Shannon  wrote:
>> Starting a new process is cheap. On my machine, starting a new Python
>> process takes under 1ms and uses a few Mbytes.
> Is that on Windows or Unix? Traditionally, process creation has been
> costly on Windows, which is why threads, and in-process solutions in
> general, tend to be more common on that platform. I haven't done
> experiments recently, but I do tend to avoid multiprocess-type
> solutions on Windows "just in case". I know that evaluating a new
> feature based on unsubstantiated assumptions informed by "it used to
> be like this" is ill-advised, but so is assuming that everything will
> be OK based on experience on a single platform :-)
Here's a test on Windows 10, 4 logical cpus, 8 GB of ram:

>>> timeit.timeit("""multiprocessing.Process(target=exit).start()""",number=100,
>>>  globals=globals())
0.62975287
>>> timeit.timeit("""multiprocessing.Process(target=exit).start()""",number=1000,
>>>  globals=globals())
40.28172119964

Or this way:
>>> timeit.timeit("""os.system('python.exe -c "exit()"')""",number=100, 
>>> globals=globals())
17.46125929995

--Edwin
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/KU3ODOFVE4NMVJWXGPSJMENCZ42P5VBW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: My take on multiple interpreters (Was: Should we be making so many changes in pursuit of PEP 554?)

[Edwin Zimmerman]

On 6/12/2020 6:18 AM, Edwin Zimmerman wrote:
> On 6/12/2020 5:08 AM, Paul Moore wrote:
>> On Fri, 12 Jun 2020 at 09:47, Mark Shannon  wrote:
>>> Starting a new process is cheap. On my machine, starting a new Python
>>> process takes under 1ms and uses a few Mbytes.
>> Is that on Windows or Unix? Traditionally, process creation has been
>> costly on Windows, which is why threads, and in-process solutions in
>> general, tend to be more common on that platform. I haven't done
>> experiments recently, but I do tend to avoid multiprocess-type
>> solutions on Windows "just in case". I know that evaluating a new
>> feature based on unsubstantiated assumptions informed by "it used to
>> be like this" is ill-advised, but so is assuming that everything will
>> be OK based on experience on a single platform :-)
> Here's a test on Windows 10, 4 logical cpus, 8 GB of ram:
>
 timeit.timeit("""multiprocessing.Process(target=exit).start()""",number=100,
  globals=globals())
> 0.62975287
 timeit.timeit("""multiprocessing.Process(target=exit).start()""",number=1000,
  globals=globals())
> 40.28172119964
>
> Or this way:
 timeit.timeit("""os.system('python.exe -c "exit()"')""",number=100, 
 globals=globals())
> 17.46125929995
>
> --Edwin
For comparison, on a single core linux cloud server with 512 mb of ram:

timeit.timeit("""multiprocessing.Process(target=exit).start()""",number=100, 
globals=globals())
0.354354709998006

timeit.timeit("""multiprocessing.Process(target=exit).start()""",number=1000, 
globals=globals())
3.847851719998289

So yeah, process creation is still rather costly on Windows.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BLBNMZXKYDKRDYRFNEHYPMNHNFMOU4WG/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: My take on multiple interpreters (Was: Should we be making so many changes in pursuit of PEP 554?)

[Steve Dower]

On 12Jun2020 1008, Paul Moore wrote:

On Fri, 12 Jun 2020 at 09:47, Mark Shannon  wrote:

Starting a new process is cheap. On my machine, starting a new Python
process takes under 1ms and uses a few Mbytes.


Is that on Windows or Unix? Traditionally, process creation has been
costly on Windows, which is why threads, and in-process solutions in
general, tend to be more common on that platform. I haven't done
experiments recently, but I do tend to avoid multiprocess-type
solutions on Windows "just in case". I know that evaluating a new
feature based on unsubstantiated assumptions informed by "it used to
be like this" is ill-advised, but so is assuming that everything will
be OK based on experience on a single platform :-)


It's still like that, though I'm actively involved in trying to get it 
improved. However, it's unlikely at this point to ever get to 
equivalence with Unix - Windows just sets up too many features 
(security, isolation, etc.) at the process boundary rather than other 
parts of the lifecycle.


It's also *incredibly arrogant* to insist that users rewrite their 
applications to suit Python, rather than us doing the work to fit their 
needs. That's not how being a libraries/runtime developer works. Our 
responsibility is to humbly do the work that will benefit our users, not 
to find ways to put in the least possible effort and use the rest for 
blame-shifting. Some of us do much more talking than listening, and it 
does not pass unnoticed.


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YYBRXYCIQE4B2NDOP3UT7AYR54DQVZCQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: My take on multiple interpreters (Was: Should we be making so many changes in pursuit of PEP 554?)

[Eric V. Smith]

On 6/11/2020 6:59 AM, Mark Shannon wrote:

Hi Riccardo,

On 10/06/2020 5:51 pm, Riccardo Ghetta wrote:

Hi,
as an user, the "lua use case" is right what I need at work.
I realize that for python this is a niche case, and most users don't 
need any of this, but I hope it will useful to understand why having 
multiple independent interpreters in a single process can be an 
essential feature.
The company I work for develop and sells a big C++ financial system 
with python embedded, providing critical flexibility to our customers.
Python is used as a scripting language, with most cases having C++ 
calling a python script itself calling other C++ functions.
Most of the times those scripts are in workloads I/O bound or where 
the time spent in python is negligible. > But some workloads are 
really cpu bound and those tend to become
GIL-bound, even with massive use of C++ helpers; some to the point 
that GIL-contention makes up over 80% of running time, instead of 1-5%.
And every time our customers upgrade their server, they buy machines 
with more cores and the contention problem worsens.


Different interpreters need to operate in their own isolated address 
space, or there will be horrible race conditions.

Regardless of whether that separation is done in software or hardware,
it has to be done.


I realize this is true now, but why must it always be true? Can't we fix 
this? At least one solution has been proposed: passing around a pointer 
to the current interpreter. I realize there issues here, like callbacks 
and signals that will need to be worked out. But I don't think it's 
axiomatically true that we'll always have race conditions with multiple 
interpreters in the same address space.


Eric
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BNBNEVUN6SLRUEQQ57SUAMP6PCCONFXF/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Final reminder: 3.7.8 / 3.6.11 cutoff Monday, last non-security bugfixes for 3.7.x

[Ivan Pozdeev via Python-Dev]

Why not take a look at the list of open issues for 3.7 with PRs then? There are 
702 as of now.

https://bugs.python.org/issue?%40sort0=creation&%40sort1=&%40group0=&%40group1=&%40columns=title%2Cid%2Cactivity%2Cstatus&%40filter=status%2Cversions%2Cstage&status=1&versions=21&stage=4&%40pagesize=50&%40startwith=0

On 12.06.2020 11:17, Ned Deily wrote:

We will be accecpting commits to the 3.7 branch until this coming Monday 06-15 
23:59 AOE. At that time, 3.7 will effectively enter security-fix mode. That 
will also be the deadline for any additional security-fixes for release in 
3.6.11. Release candidates will follow shortly thereafter. Assuming no 
release-blocker issues arise during the release candidate testing, 3.7.8 and 
3.6.11 will release on 06-27.

https://discuss.python.org/t/upcoming-cutoffs-for-3-9-3-8-3-7-and-3-6-releases-last-chance-for-3-7-bugfixes/4362

--
   Ned Deily
   n...@python.org -- []
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/EKYBL5VG6TWSTTQUNBP56VJVHDR3GRM3/
Code of Conduct: http://python.org/psf/codeofconduct/
--
Regards,
Ivan

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OYXTIA7JJEAGJAH57OJWLDQKG5QTMHQZ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Final reminder: 3.7.8 / 3.6.11 cutoff Monday, last non-security bugfixes for 3.7.x

[Ned Deily]

On Jun 12, 2020, at 08:25, Ivan Pozdeev via Python-Dev  
wrote:
> Why not take a look at the list of open issues for 3.7 with PRs then? There 
> are 702 as of now.
> 
> https://bugs.python.org/issue?%40sort0=creation&%40sort1=&%40group0=&%40group1=&%40columns=title%2Cid%2Cactivity%2Cstatus&%40filter=status%2Cversions%2Cstage&status=1&versions=21&stage=4&%40pagesize=50&%40startwith=0

Looking at open PRs is a good suggestion. Unfortunately, the version tags on 
the bug tracker are not consistently maintained and updated. Just because there 
is a PR attached  to an issue doesn’t mean that it would be applicable today to 
all the versions selected in the issue.
 
A more accurate approach is to look at open PRs that are either explicitly 
targeted for 3.7 or which have the “needs backport to 3.7” label.  By those 
metrics, we currently have no open 3.7-specific PRs (the last was merged 
yesterday) and there are 60 open PRs with the 3.7 backport label. But even that 
number is high because some of those will end up not getting merged to master 
at all, and thus not backported, and others will be merged to master for 3.10 
and perhaps to 3.9 but deemed inappropriate to backport to 3.8 and/or 3.7.

https://github.com/python/cpython/pulls?utf8=✓&q=is%3Apr+is%3Aopen+base%3A3.7

https://github.com/python/cpython/pulls?utf8=✓&q=is%3Apr+is%3Aopen+label%3A%22needs+backport+to+3.7%22

  --
Ned Deily
n...@python.org -- []
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/V3JTYA7NRU7PGOOWFZY2VQQPL42WI3J5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: My take on multiple interpreters (Was: Should we be making so many changes in pursuit of PEP 554?)

[Mark Shannon]

Hi Edwin,

Thanks for providing some concrete numbers.
Is it expected that creating 100 processes takes 6.3ms per process, but 
that creating 1000 process takes 40ms per process? That's over 6 times 
as long in the latter case.


Cheers,
Mark.

On 12/06/2020 11:29 am, Edwin Zimmerman wrote:

On 6/12/2020 6:18 AM, Edwin Zimmerman wrote:

On 6/12/2020 5:08 AM, Paul Moore wrote:

On Fri, 12 Jun 2020 at 09:47, Mark Shannon  wrote:

Starting a new process is cheap. On my machine, starting a new Python
process takes under 1ms and uses a few Mbytes.

Is that on Windows or Unix? Traditionally, process creation has been
costly on Windows, which is why threads, and in-process solutions in
general, tend to be more common on that platform. I haven't done
experiments recently, but I do tend to avoid multiprocess-type
solutions on Windows "just in case". I know that evaluating a new
feature based on unsubstantiated assumptions informed by "it used to
be like this" is ill-advised, but so is assuming that everything will
be OK based on experience on a single platform :-)

Here's a test on Windows 10, 4 logical cpus, 8 GB of ram:


timeit.timeit("""multiprocessing.Process(target=exit).start()""",number=100, 
globals=globals())

0.62975287

timeit.timeit("""multiprocessing.Process(target=exit).start()""",number=1000, 
globals=globals())

40.28172119964

Or this way:

timeit.timeit("""os.system('python.exe -c "exit()"')""",number=100, 
globals=globals())

17.46125929995

--Edwin

For comparison, on a single core linux cloud server with 512 mb of ram:

timeit.timeit("""multiprocessing.Process(target=exit).start()""",number=100, 
globals=globals())
0.354354709998006

timeit.timeit("""multiprocessing.Process(target=exit).start()""",number=1000, 
globals=globals())
3.847851719998289

So yeah, process creation is still rather costly on Windows.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BLBNMZXKYDKRDYRFNEHYPMNHNFMOU4WG/
Code of Conduct: http://python.org/psf/codeofconduct/


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/XKAMVXHG3FJDNAGFWYCLUGM3L7VFV52P/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Are extension types allowed to implement both nb_add and sq_concat?

[Eric Wieser]

Hi all,

In the `operator` module, both `operator.concat` and `operator.add` are 
described as performing `a + b`.
When defining a type in pure python, it is not possible to give these different 
meanings.
However, when defining a C extension type, it is possible to fill both 
`tp_as_number->nb_add` and `tp_as_sequence->sq_concat` with different 
implementations to obtain different behaviors.

Is the fact that `operator.concat` (`PySequence_Concat`) and `operator.add` 
(`PyNumber_Add`) can have different behaviors an implementation detail of 
CPython, or is it intended behavior in the language itself?

I ask because [a recent PR for numpy][1] proposes implementing both of these 
slots, and discussion turned to whether it was intended to be possible to do 
this in the first place, and if so whether a patch will be needed to PyPy.

It seems that there was at least some vague intent for this to be possible - In 
bpo-29139, Serhiy said

> Third-party classes (maybe NumPy arrays, I don't know) can have different 
> implementations of `sq_concat` and `nb_add`.

It seems to me there are at least three stances that could be taken here:

* Specifying both is considered invalid: python should consider emitting a 
warning in `Type_READY` if both are filled.
* Specifying both is considered an implementation detail specific to CPython: 
the [C API docs for the type slots][2] should indicate this
* Specifying both is explicitly allowed and considered a language feature. 
`__concat__` should be added as a slot_wrapper around `sq_concat` to allow the 
language feature to be accessed without writing C extensions.

Eric Wieser

Apologies if this is not the right list - this didn't feel right for 
python-ideas or bpo.
I'm more than happy to repost this elsewhere if asked.

[1]: https://github.com/numpy/numpy/issues/16489
[2]: https://docs.python.org/3/c-api/typeobj.html
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/HCXMI7LPWZEKKPRIJTVHN5ZRWIUXM5C3/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: My take on multiple interpreters (Was: Should we be making so many changes in pursuit of PEP 554?)

[Riccardo Ghetta]

On 12/06/2020 10:45, Mark Shannon wrote:

On 11/06/2020 2:50 pm, Riccardo Ghetta wrote:
On 11/06/2020 12:59, Mark Shannon wrote:



If the additional resource consumption is irrelevant, what's the
objection to spinning up a new processes?

The additional resource consumption of a new python interpreter is
irrelevant, but the process as a whole needs a lot of extra data making
a new process rather costly.

Starting a new process is cheap. On my machine, starting a new Python
process takes under 1ms and uses a few Mbytes.
Sorry, I wasn't clear here.  I was talking about starting one of our 
server processes, /with python embedded/.
Since python routines are called by our C++ code and need to call other 
C++ routines, it cannot work alone and is surrounded by a lot of data 
needed for the C++ part.  A python interpreter by itself would be like a 
cpu chip for someone needing a server.  A critical component, sure, but  
only a small part of the whole.

Plus there are issues of licensing, synchronization and load balancing
that are much easier to resolve (for our system, at least) with threads
than processes.

Would this prevent CPython starting new processes, or is this just for
processes managed by your application?
Is only for application processes, but because python is always embedded 
there is little practical difference.


I hope to not come out arrogant or dismissive, but can we take it from 
granted that multiprocessing is not a viable solution for our 
application, or at least that it would be impractical and too expensive 
rebuilding it from scratch to change paradigm ?
At the same time, I realize that ours is a somewhat niche case and it 
may not be deemed interesting for python evolution.
I just wanted to present a real world example of someone using python 
today and who would benefit immensely if python would permit multiple, 
separate, interpreters in a single process.
Or any other solution removing the bottlenecks that currently so limit 
multithreaded python performance.


Ciao,
Riccardo
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JSVOIN77TCOSDLRI7OALBZGGTQCPOKNE/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: My take on multiple interpreters (Was: Should we be making so many changes in pursuit of PEP 554?)

[Mark Shannon]

Hi Steve,

On 12/06/2020 12:43 pm, Steve Dower wrote:

On 12Jun2020 1008, Paul Moore wrote:

On Fri, 12 Jun 2020 at 09:47, Mark Shannon  wrote:

Starting a new process is cheap. On my machine, starting a new Python
process takes under 1ms and uses a few Mbytes.


Is that on Windows or Unix? Traditionally, process creation has been
costly on Windows, which is why threads, and in-process solutions in
general, tend to be more common on that platform. I haven't done
experiments recently, but I do tend to avoid multiprocess-type
solutions on Windows "just in case". I know that evaluating a new
feature based on unsubstantiated assumptions informed by "it used to
be like this" is ill-advised, but so is assuming that everything will
be OK based on experience on a single platform :-)


It's still like that, though I'm actively involved in trying to get it 
improved. However, it's unlikely at this point to ever get to 
equivalence with Unix - Windows just sets up too many features 
(security, isolation, etc.) at the process boundary rather than other 
parts of the lifecycle. >
It's also *incredibly arrogant* to insist that users rewrite their 
applications to suit Python, rather than us doing the work to fit their 
needs. That's not how being a libraries/runtime developer works. Our 
responsibility is to humbly do the work that will benefit our users, not 
to find ways to put in the least possible effort and use the rest for 
blame-shifting. Some of us do much more talking than listening, and it 
does not pass unnoticed.
I don't think anyone is suggesting that users rewrite their code to work 
with existing features.
Using any new feature is going to take some work on the users part, and 
we are talking about a new feature.


Developer time is a finite resource and any time spent helping one set 
of users is not spent helping others.
Likewise, optimizing for one use case may be hurting performance for 
other use cases.


Personally, I have no idea what is important for other people, but I 
would like any discussion to have sound technical underpinnings.

Once we have those, it becomes possible to have a meaningful discussion.

Cheers,
Mark.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/4U26HEVBZTETSOXKHETH23VZUXHFHAU2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: My take on multiple interpreters (Was: Should we be making so many changes in pursuit of PEP 554?)

[Eric Snow]

On Fri, Jun 12, 2020 at 2:49 AM Mark Shannon  wrote:
> The overhead largely comes from what you do with the process. The
> additional cost of starting a new interpreter is the same regardless of
> whether it is in the same process or not.

FWIW, there's more to it than that:

* there is some overhead to starting the runtime and main interpreter
that does not apply to additional in-process interpreters
* I don't see why we shouldn't be able to come up with a strategy for
interpreter startup that does not involve copying or sharing a lot of
interpreter state, thus reducing startup time and memory consumption
* I'm guessing that re-importing builtin/extension modules is faster
than importing then new in a separate process

-eric
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/M7FZL6LVEP2CRMDKGZE4BA6G7WOS542H/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: My take on multiple interpreters (Was: Should we be making so many changes in pursuit of PEP 554?)

[Mark Shannon]

Hi Eric,

On 12/06/2020 4:17 pm, Eric Snow wrote:

On Fri, Jun 12, 2020 at 2:49 AM Mark Shannon  wrote:

The overhead largely comes from what you do with the process. The
additional cost of starting a new interpreter is the same regardless of
whether it is in the same process or not.


FWIW, there's more to it than that:

* there is some overhead to starting the runtime and main interpreter
that does not apply to additional in-process interpreters


You seem to be implying that there would be more overhead for a new 
interpreter that operates in a different O/S process. What would that be?



* I don't see why we shouldn't be able to come up with a strategy for
interpreter startup that does not involve copying or sharing a lot of
interpreter state, thus reducing startup time and memory consumption


Indeed, that would be beneficial regardless of which process the 
interpreter is in.



* I'm guessing that re-importing builtin/extension modules is faster
than importing then new in a separate process


Each new interpreter need to re-import the modules. The overhead could 
be reduced by making more of the module immutable, allowing some 
sharing. For linux, at least, that benefit would apply to multiple 
processes as well.


Cheers,
Mark.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/FUN2IKFHOYSUYY6UJA7C73AWK3YMF7QD/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Are extension types allowed to implement both nb_add and sq_concat?

[Guido van Rossum]

But how would you invoke them, other than using operator.add and
operator.concat? A better UI would be to define new methods with more
discoverable names.

On Fri, Jun 12, 2020 at 7:29 AM Eric Wieser 
wrote:

> Hi all,
>
> In the `operator` module, both `operator.concat` and `operator.add` are
> described as performing `a + b`.
> When defining a type in pure python, it is not possible to give these
> different meanings.
> However, when defining a C extension type, it is possible to fill both
> `tp_as_number->nb_add` and `tp_as_sequence->sq_concat` with different
> implementations to obtain different behaviors.
>
> Is the fact that `operator.concat` (`PySequence_Concat`) and
> `operator.add` (`PyNumber_Add`) can have different behaviors an
> implementation detail of CPython, or is it intended behavior in the
> language itself?
>
> I ask because [a recent PR for numpy][1] proposes implementing both of
> these slots, and discussion turned to whether it was intended to be
> possible to do this in the first place, and if so whether a patch will be
> needed to PyPy.
>
> It seems that there was at least some vague intent for this to be possible
> - In bpo-29139, Serhiy said
>
> > Third-party classes (maybe NumPy arrays, I don't know) can have
> different implementations of `sq_concat` and `nb_add`.
>
> It seems to me there are at least three stances that could be taken here:
>
> * Specifying both is considered invalid: python should consider emitting a
> warning in `Type_READY` if both are filled.
> * Specifying both is considered an implementation detail specific to
> CPython: the [C API docs for the type slots][2] should indicate this
> * Specifying both is explicitly allowed and considered a language feature.
> `__concat__` should be added as a slot_wrapper around `sq_concat` to allow
> the language feature to be accessed without writing C extensions.
>
> Eric Wieser
>
> Apologies if this is not the right list - this didn't feel right for
> python-ideas or bpo.
> I'm more than happy to repost this elsewhere if asked.
>
> [1]: https://github.com/numpy/numpy/issues/16489
> [2]: https://docs.python.org/3/c-api/typeobj.html
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/HCXMI7LPWZEKKPRIJTVHN5ZRWIUXM5C3/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/FB5N6U7ZNFAMHIJIK374RWE2F5WCJ2VI/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: When can we remove wchar_t* cache from string?

[MRAB]

On 2020-06-12 09:32, Inada Naoki wrote:

Hi, all.

Py_UNICODE has been deprecated since PEP 393 (Flexible string representation).

wchar_t* cache in the string object is used only in deprecated APIs.
It waste 1 word (8 bytes on 64bit machine) per string instance.

The deprecated APIs are documented as "Deprecated since version 3.3,
will be removed in version 4.0."
See https://docs.python.org/3/c-api/unicode.html#deprecated-py-unicode-apis

But when PEP 393 is implemented, no one expects 3.10 will be released.
Can we reschedule the removal?

My proposal is, schedule the removal on Python 3.11.  But we will postpone
the removal if we can not remove its usage until it.

I grepped the use of the deprecated APIs from top 4000 PyPI packages.

result: 
https://github.com/methane/notes/blob/master/2020/wchar-cache/deprecated-use
step: https://github.com/methane/notes/blob/master/2020/wchar-cache/README.md

I noticed:

* Most of them are generated by Cython.
   * I reported it to Cython so Cython 0.29.21 will fix them.  I expect
more than 1 year
 between Cython 0.29.21 and Python 3.11rc1.
* Most of them are `PyUnicode_FromUnicode(NULL, 0);`
   * We may be able to keep PyUnicode_FromUnicode, but raise error when 
length>0.

I think it would be strange to keep PyUnicode_FromUnicode but complain 
unless length == 0. If it's going to be removed, then remove it and 
suggest a replacement for that use-case, such as PyUnicode_FromString 
with a NULL argument. (I'm not sure if PyUnicode_FromString will accept 
NULL, but if it currently doesn't, then maybe it should in future be 
treated as being equivalent to PyUnicode_FromString("").)

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/3CMLZWRTOBTA6NMOZRYS5MF2SY7S5GKM/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Are extension types allowed to implement both nb_add and sq_concat?

[Barry Warsaw]

On Jun 12, 2020, at 04:21, Eric Wieser  wrote:

> It seems to me there are at least three stances that could be taken here:
> 
> * Specifying both is considered invalid: python should consider emitting a 
> warning in `Type_READY` if both are filled.
> * Specifying both is considered an implementation detail specific to CPython: 
> the [C API docs for the type slots][2] should indicate this
> * Specifying both is explicitly allowed and considered a language feature. 
> `__concat__` should be added as a slot_wrapper around `sq_concat` to allow 
> the language feature to be accessed without writing C extensions.

If you can define the behavior at the C layer and not at the Python layer, then 
I think almost by definition it’s an implementation detail of CPython.  Whether 
that’s intentional or not would help determine whether this should be demoted 
to a bug with an emitted warning, or promoted to a full-on feature supported in 
the language.

Cheers,
-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/DO5J63K3ACFCV63SRD6QAHT6PAIMLNKR/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: My take on multiple interpreters (Was: Should we be making so many changes in pursuit of PEP 554?)

[Emily Bowman]

On Fri, Jun 12, 2020 at 7:19 AM Mark Shannon  wrote:

> Hi Edwin,
>
> Thanks for providing some concrete numbers.
> Is it expected that creating 100 processes takes 6.3ms per process, but
> that creating 1000 process takes 40ms per process? That's over 6 times
> as long in the latter case.
>
> Cheers,
> Mark.
>
> On 12/06/2020 11:29 am, Edwin Zimmerman wrote:
> > On 6/12/2020 6:18 AM, Edwin Zimmerman wrote:
> >> On 6/12/2020 5:08 AM, Paul Moore wrote:
> >>> On Fri, 12 Jun 2020 at 09:47, Mark Shannon  wrote:
>  Starting a new process is cheap. On my machine, starting a new Python
>  process takes under 1ms and uses a few Mbytes.
> >>> Is that on Windows or Unix? Traditionally, process creation has been
> >>> costly on Windows, which is why threads, and in-process solutions in
> >>> general, tend to be more common on that platform. I haven't done
> >>> experiments recently, but I do tend to avoid multiprocess-type
> >>> solutions on Windows "just in case". I know that evaluating a new
> >>> feature based on unsubstantiated assumptions informed by "it used to
> >>> be like this" is ill-advised, but so is assuming that everything will
> >>> be OK based on experience on a single platform :-)
> >> Here's a test on Windows 10, 4 logical cpus, 8 GB of ram:
> >>
> >
> timeit.timeit("""multiprocessing.Process(target=exit).start()""",number=100,
> globals=globals())
> >> 0.62975287
> >
> timeit.timeit("""multiprocessing.Process(target=exit).start()""",number=1000,
> globals=globals())
> >> 40.28172119964
> >>
> >> Or this way:
> > timeit.timeit("""os.system('python.exe -c "exit()"')""",number=100,
> globals=globals())
> >> 17.46125929995
> >>
> >> --Edwin
> > For comparison, on a single core linux cloud server with 512 mb of ram:
> >
> >
> timeit.timeit("""multiprocessing.Process(target=exit).start()""",number=100,
> globals=globals())
> > 0.354354709998006
> >
> >
> timeit.timeit("""multiprocessing.Process(target=exit).start()""",number=1000,
> globals=globals())
> > 3.847851719998289
> >
> > So yeah, process creation is still rather costly on Windows.
>

I was wondering that too, some tests show that process creation/destruction
starts to seriously bog down after a few hundred in a row. I guess it's
hitting some resource limits it has to clean up, though creating hundreds
of processes at once sounds like an antipattern that doesn't really deserve
too much consideration. It would be rare that fork is more than a
negligible part of any workload. (With A/V on, though, it's _much_ slower
out the gate. I'm seeing over 100ms per process with Kaspersky running.)

Em
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/S47P3DFRW46VRVWLSKDASREHLYOFE6L4/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: My take on multiple interpreters (Was: Should we be making so many changes in pursuit of PEP 554?)

[Edwin Zimmerman]

My previous timings were slightly inaccurate, as they compared spawning 
processes on Windows to forking on Linux.  Also, I changed my timing code to 
run all process synchronously, to avoid hitting resource limits.

 

Updated Windows (Windows 7 this time, on a four core processor):

>>> timeit.timeit('x=multiprocessing.Process(target=exit);x.start();x.join()', 
>>> number=1000,globals = globals())

84.7111053659259

 

Updated Linux with spawn (single core processor):

>>> ctx = multiprocessing.get_context('spawn')

>>> timeit.timeit('x=ctx.Process(target=exit);x.start();x.join()', 
>>> number=1000,globals = globals())

60.01154333699378

 

Updated Linux with fork:
>>> timeit.timeit('x=multiprocessing.Process(target=exit);x.start();x.join()', 
>>> number=1000,globals = globals())

4.402019854984246

 

Compare this to subinterpreters on my linux machine:

>>> timeit.timeit('s=_xxsubinterpreters.create();_xxsubinterpreters.destroy(s)',number=1000,
>>>  globals=globals())

13.47043095799745

 

This shows that is speed is all that matters, multiprocessing comes out way 
ahead of subinterpreters on linux, but way behind on Windows.

I need to time subinterpreters on Windows yet for the full picture, but that 
will be tomorrow till I get that done.

 

--Edwin

 

From: Emily Bowman [mailto:silverback...@gmail.com] 
Sent: Friday, June 12, 2020 12:44 PM
To: Mark Shannon 
Cc: Python Dev 
Subject: [Python-Dev] Re: My take on multiple interpreters (Was: Should we be 
making so many changes in pursuit of PEP 554?)

 

On Fri, Jun 12, 2020 at 7:19 AM Mark Shannon mailto:m...@hotpy.org> > wrote:

Hi Edwin,

Thanks for providing some concrete numbers.
Is it expected that creating 100 processes takes 6.3ms per process, but 
that creating 1000 process takes 40ms per process? That's over 6 times 
as long in the latter case.

Cheers,
Mark.

On 12/06/2020 11:29 am, Edwin Zimmerman wrote:
> On 6/12/2020 6:18 AM, Edwin Zimmerman wrote:
>> On 6/12/2020 5:08 AM, Paul Moore wrote:
>>> On Fri, 12 Jun 2020 at 09:47, Mark Shannon >>  > wrote:
 Starting a new process is cheap. On my machine, starting a new Python
 process takes under 1ms and uses a few Mbytes.
>>> Is that on Windows or Unix? Traditionally, process creation has been
>>> costly on Windows, which is why threads, and in-process solutions in
>>> general, tend to be more common on that platform. I haven't done
>>> experiments recently, but I do tend to avoid multiprocess-type
>>> solutions on Windows "just in case". I know that evaluating a new
>>> feature based on unsubstantiated assumptions informed by "it used to
>>> be like this" is ill-advised, but so is assuming that everything will
>>> be OK based on experience on a single platform :-)
>> Here's a test on Windows 10, 4 logical cpus, 8 GB of ram:
>>
> timeit.timeit("""multiprocessing.Process(target=exit).start()""",number=100,
>  globals=globals())
>> 0.62975287
> timeit.timeit("""multiprocessing.Process(target=exit).start()""",number=1000,
>  globals=globals())
>> 40.28172119964
>>
>> Or this way:
> timeit.timeit("""os.system('python.exe -c "exit()"')""",number=100, 
> globals=globals())
>> 17.46125929995
>>
>> --Edwin
> For comparison, on a single core linux cloud server with 512 mb of ram:
> 
> timeit.timeit("""multiprocessing.Process(target=exit).start()""",number=100, 
> globals=globals())
> 0.354354709998006
> 
> timeit.timeit("""multiprocessing.Process(target=exit).start()""",number=1000, 
> globals=globals())
> 3.847851719998289
> 
> So yeah, process creation is still rather costly on Windows.

 

I was wondering that too, some tests show that process creation/destruction 
starts to seriously bog down after a few hundred in a row. I guess it's hitting 
some resource limits it has to clean up, though creating hundreds of processes 
at once sounds like an antipattern that doesn't really deserve too much 
consideration. It would be rare that fork is more than a negligible part of any 
workload. (With A/V on, though, it's _much_ slower out the gate. I'm seeing 
over 100ms per process with Kaspersky running.)

 

Em

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OGUXOQZWQLEERNCAHQ4PP6CMUCU3WNF3/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Final reminder: 3.7.8 / 3.6.11 cutoff Monday, last non-security bugfixes for 3.7.x

[Ivan Pozdeev via Python-Dev]

On 12.06.2020 16:08, Ned Deily wrote:

On Jun 12, 2020, at 08:25, Ivan Pozdeev via Python-Dev  
wrote:

Why not take a look at the list of open issues for 3.7 with PRs then? There are 
702 as of now.

https://bugs.python.org/issue?%40sort0=creation&%40sort1=&%40group0=&%40group1=&%40columns=title%2Cid%2Cactivity%2Cstatus&%40filter=status%2Cversions%2Cstage&status=1&versions=21&stage=4&%40pagesize=50&%40startwith=0

Looking at open PRs is a good suggestion. Unfortunately, the version tags on 
the bug tracker are not consistently maintained and updated. Just because there 
is a PR attached  to an issue doesn’t mean that it would be applicable today to 
all the versions selected in the issue.
  
A more accurate approach is to look at open PRs that are either explicitly targeted for 3.7 or which have the “needs backport to 3.7” label.  By those metrics, we currently have no open 3.7-specific PRs (the last was merged yesterday) and there are 60 open PRs with the 3.7 backport label. But even that number is high because some of those will end up not getting merged to master at all, and thus not backported, and others will be merged to master for 3.10 and perhaps to 3.9 but deemed inappropriate to backport to 3.8 and/or 3.7.


I would doubt the quality of tags maintenance at Github, too. E.g. https://github.com/python/cpython/pull/12131 is labeled 3.7 and 3.8 at 
BPO but has no backport tags whatsoever at GH.

So the search you gave is clearly insufficient.


https://github.com/python/cpython/pulls?utf8=✓&q=is%3Apr+is%3Aopen+base%3A3.7

https://github.com/python/cpython/pulls?utf8=✓&q=is%3Apr+is%3Aopen+label%3A%22needs+backport+to+3.7%22

   --
 Ned Deily
 n...@python.org -- []
--
Regards,
Ivan

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/55NOEOVOTQUJYXCKGEAULIDY7I22BPRJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Summary of Python tracker Issues

[Python tracker]

ACTIVITY SUMMARY (2020-06-05 - 2020-06-12)
Python tracker at https://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open7447 ( +3)
  closed 45258 (+78)
  total  52705 (+81)

Open issues with patches: 3009 


Issues opened (57)
==

#29782: Use __builtin_clzl for bits_in_digit if available
https://bugs.python.org/issue29782  reopened by vstinner

#40878: Use c99 on the aixtools bot
https://bugs.python.org/issue40878  opened by skrah

#40882: memory leak in multiprocessing.shared_memory.SharedMemory in W
https://bugs.python.org/issue40882  opened by eryksun

#40884: Added defaults parameter for logging.Formatter
https://bugs.python.org/issue40884  opened by bar.harel

#40885: Cannot pipe GzipFile into subprocess
https://bugs.python.org/issue40885  opened by Nehal Patel

#40886: Add PYTHONLOGGING environment variable and -L cmdline argument
https://bugs.python.org/issue40886  opened by bar.harel

#40891: Use pep573 in functools
https://bugs.python.org/issue40891  opened by shihai1991

#40892: IDLE: use rlcompleter suffixed for completions
https://bugs.python.org/issue40892  opened by terry.reedy

#40893: tkinter integrate TkDND support
https://bugs.python.org/issue40893  opened by epaine

#40894: asyncio.gather() cancelled() always False
https://bugs.python.org/issue40894  opened by timmwagener

#40896: Missing links to Source Code in Documentation pages
https://bugs.python.org/issue40896  opened by edison.abahurire

#40897: Inheriting from class that defines __new__ causes inspect.sign
https://bugs.python.org/issue40897  opened by ezyang

#40899: Document exceptions raised by importlib.import
https://bugs.python.org/issue40899  opened by j13r

#40900: uuid module build fix on FreeBSD proposal
https://bugs.python.org/issue40900  opened by devnexen

#40901: It's not clear what "interface name" means in socket if_namein
https://bugs.python.org/issue40901  opened by jstasiak

#40905: IDLE relabel Save on close
https://bugs.python.org/issue40905  opened by terry.reedy

#40906: Unable to import module due to python unable to resolve depend
https://bugs.python.org/issue40906  opened by sabakauser

#40912: _PyOS_SigintEvent is never closed on Windows
https://bugs.python.org/issue40912  opened by steve.dower

#40913: time.sleep ignores errors on Windows
https://bugs.python.org/issue40913  opened by steve.dower

#40914: tarfile creates output that appears to omit files
https://bugs.python.org/issue40914  opened by mcr314

#40915: multiple problems with mmap.resize()  in Windows
https://bugs.python.org/issue40915  opened by eryksun

#40916: Proposed tweak to allow for per-task async generator semantics
https://bugs.python.org/issue40916  opened by Joshua Oreman

#40923: Thread safety : disable intel’s compiler autopar where itâ€
https://bugs.python.org/issue40923  opened by Laël Cellier

#40924: Recent importlib change breaks most recent certifi == 2020.4.5
https://bugs.python.org/issue40924  opened by ned.deily

#40925: Remove redundant macros used for stack manipulation in interpr
https://bugs.python.org/issue40925  opened by Mark.Shannon

#40926: command line interface of symtable module is broken
https://bugs.python.org/issue40926  opened by BTaskaya

#40927: ./python -m test test___all__ test_binhex fails
https://bugs.python.org/issue40927  opened by remi.lapeyre

#40928: OS X: malloc(): set default diagnostics to DEBUG_WRITE_ON_CRAS
https://bugs.python.org/issue40928  opened by remi.lapeyre

#40930: zoneinfo gives incorrect dst() in Pacific/Rarotonga between 19
https://bugs.python.org/issue40930  opened by p-ganssle

#40931: zoneinfo gives incorrect dst() in Europe/Madrid in 1938
https://bugs.python.org/issue40931  opened by p-ganssle

#40932: subprocess docs should warn of shlex use on Windows
https://bugs.python.org/issue40932  opened by Stephen Farris

#40933: zoneinfo may give incorrect dst() in Europe/Minsk in 1942
https://bugs.python.org/issue40933  opened by p-ganssle

#40934: Default RLock does not work well for Manager Condition
https://bugs.python.org/issue40934  opened by Misko Dzamba

#40935: Links to Python3 docs for some libs return 404
https://bugs.python.org/issue40935  opened by edison.abahurire

#40936: Remove deprecated functions from gettext
https://bugs.python.org/issue40936  opened by remi.lapeyre

#40937: Remove deprecated functions marked for removal in 3.10
https://bugs.python.org/issue40937  opened by remi.lapeyre

#40938: urllib.parse.urlunsplit makes relative path to absolute (http:
https://bugs.python.org/issue40938  opened by op368

#40939: Remove the old parser
https://bugs.python.org/issue40939  opened by pablogsal

#40941: Merge generator.gi_running and frame executing flag into singl
https://bugs.python.org/issue40941  opened by Mark.Shannon

#40942: BaseManager cannot start with local manager
https://bugs.python.org/issue40942  opened by Mike Jarvis

[Python-Dev] Re: My take on multiple interpreters (Was: Should we be making so many changes in pursuit of PEP 554?)

[Chris Angelico]

On Sat, Jun 13, 2020 at 3:50 AM Edwin Zimmerman  wrote:
>
> My previous timings were slightly inaccurate, as they compared spawning 
> processes on Windows to forking on Linux.  Also, I changed my timing code to 
> run all process synchronously, to avoid hitting resource limits.
>
> Updated Windows (Windows 7 this time, on a four core processor):
>
> >>> timeit.timeit('x=multiprocessing.Process(target=exit);x.start();x.join()',
> >>>  number=1000,globals = globals())
> 84.7111053659259
>

Thanks, I was actually going to ask about joining the processes, since
you don't really get a good indication of timings from asynchronous
operations like that. Another interesting data point is that starting
and joining in batches makes a fairly huge difference to performance,
at least on my Linux system. Starting with your example and rescaling
the number by ten to compensate for performance differences:

>>> timeit.timeit('x=multiprocessing.Process(target=exit);x.start();x.join()', 
>>> number=1,globals = globals())
14.261007152497768

Just for completeness and consistency, confirmed that adding a list
comp around it doesn't change the timings:

>>> timeit.timeit('xx=[multiprocessing.Process(target=exit) for _ in 
>>> range(1)];[x.start() for x in xx];[x.join() for x in xx]', 
>>> number=1,globals = globals())
14.030426062643528

But doing a hundred at a time and then joining them all cuts the time in half!

>>> timeit.timeit('xx=[multiprocessing.Process(target=exit) for _ in 
>>> range(100)];[x.start() for x in xx];[x.join() for x in xx]', 
>>> number=100,globals = globals())
5.470761131495237

The difference is even more drastic with spawn, although since it's
slower, I also lowered the number of iterations.

>>> ctx = multiprocessing.get_context('spawn')
>>> timeit.timeit('x=ctx.Process(target=exit);x.start();x.join()', 
>>> number=1000,globals = globals())
40.82687543518841
>>> timeit.timeit('xx=[ctx.Process(target=exit) for _ in range(100)];[x.start() 
>>> for x in xx];[x.join() for x in xx]', number=10,globals = 
>>> globals())8.566341979429126
8.566341979429126

Would be curious to know if that's the same on Windows.

ChrisA
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YFRM3LNO37B5JXNYPO2T7CAVYQRGAYES/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Are extension types allowed to implement both nb_add and sq_concat?

[Eric Wieser]

>  But how would you invoke them, other than using operator.add and
operator.concat?

The case I was thinking of was just a direct call to `operator.concat`.
The motivation here is simply to make `operator.concat` do
something non-surprising on numpy arrays (be that `np.concatenate(a, b)`,
or simply `raise TypeError`).
I doubt numpy would actually advocate using it, but we can at least remove
the foot-gun.

Eric

On Fri, 12 Jun 2020 at 17:18, Guido van Rossum  wrote:

> But how would you invoke them, other than using operator.add and
> operator.concat? A better UI would be to define new methods with more
> discoverable names.
>
> On Fri, Jun 12, 2020 at 7:29 AM Eric Wieser 
> wrote:
>
>> Hi all,
>>
>> In the `operator` module, both `operator.concat` and `operator.add` are
>> described as performing `a + b`.
>> When defining a type in pure python, it is not possible to give these
>> different meanings.
>> However, when defining a C extension type, it is possible to fill both
>> `tp_as_number->nb_add` and `tp_as_sequence->sq_concat` with different
>> implementations to obtain different behaviors.
>>
>> Is the fact that `operator.concat` (`PySequence_Concat`) and
>> `operator.add` (`PyNumber_Add`) can have different behaviors an
>> implementation detail of CPython, or is it intended behavior in the
>> language itself?
>>
>> I ask because [a recent PR for numpy][1] proposes implementing both of
>> these slots, and discussion turned to whether it was intended to be
>> possible to do this in the first place, and if so whether a patch will be
>> needed to PyPy.
>>
>> It seems that there was at least some vague intent for this to be
>> possible - In bpo-29139, Serhiy said
>>
>> > Third-party classes (maybe NumPy arrays, I don't know) can have
>> different implementations of `sq_concat` and `nb_add`.
>>
>> It seems to me there are at least three stances that could be taken here:
>>
>> * Specifying both is considered invalid: python should consider emitting
>> a warning in `Type_READY` if both are filled.
>> * Specifying both is considered an implementation detail specific to
>> CPython: the [C API docs for the type slots][2] should indicate this
>> * Specifying both is explicitly allowed and considered a language
>> feature. `__concat__` should be added as a slot_wrapper around `sq_concat`
>> to allow the language feature to be accessed without writing C extensions.
>>
>> Eric Wieser
>>
>> Apologies if this is not the right list - this didn't feel right for
>> python-ideas or bpo.
>> I'm more than happy to repost this elsewhere if asked.
>>
>> [1]: https://github.com/numpy/numpy/issues/16489
>> [2]: https://docs.python.org/3/c-api/typeobj.html
>> ___
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-le...@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-dev@python.org/message/HCXMI7LPWZEKKPRIJTVHN5ZRWIUXM5C3/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
>
> --
> --Guido van Rossum (python.org/~guido)
> *Pronouns: he/him **(why is my pronoun here?)*
> 
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/357WAPFVVDES26PJFMWBMMHI2VJ3M3HE/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Are extension types allowed to implement both nb_add and sq_concat?

[Guido van Rossum]

On Fri, Jun 12, 2020 at 12:56 PM Eric Wieser 
wrote:

> >  But how would you invoke them, other than using operator.add and
> operator.concat?
>
> The case I was thinking of was just a direct call to `operator.concat`.
> The motivation here is simply to make `operator.concat` do
> something non-surprising on numpy arrays (be that `np.concatenate(a, b)`,
> or simply `raise TypeError`).
> I doubt numpy would actually advocate using it, but we can at least remove
> the foot-gun.
>

Oh, that makes sense, as a CPython specific behavior. What does it do
currently? Is the problem that it does the same thing as add? I'd vote for
raising -- you shouldn't encourage or enable code that uses this pattern.
(In general I am a big fan of just never using the operator module.)

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/4OBAZMPYMK6GW2ORCUCKLA3AK7P56UZU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] PEP 387: backwards compatibility policy

[Brett Cannon]

Started at a discussion at
https://discuss.python.org/t/pep-387-backwards-compatibilty-policy/4421 to
try to finally resolve this PEP and our backwards compatibility policy.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/MPEQZZMUX72WA7JYAOIE6EDPGKV7CTAK/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: My take on multiple interpreters (Was: Should we be making so many changes in pursuit of PEP 554?)

[Edwin Zimmerman]

On 6/12/2020 2:17 PM, Chris Angelico wrote:
> On Sat, Jun 13, 2020 at 3:50 AM Edwin Zimmerman  
> wrote:
>> My previous timings were slightly inaccurate, as they compared spawning 
>> processes on Windows to forking on Linux.  Also, I changed my timing code to 
>> run all process synchronously, to avoid hitting resource limits.
>>
>> Updated Windows (Windows 7 this time, on a four core processor):
>>
> timeit.timeit('x=multiprocessing.Process(target=exit);x.start();x.join()',
>  number=1000,globals = globals())
>> 84.7111053659259
>>
> Thanks, I was actually going to ask about joining the processes, since
> you don't really get a good indication of timings from asynchronous
> operations like that. Another interesting data point is that starting
> and joining in batches makes a fairly huge difference to performance,
> at least on my Linux system. Starting with your example and rescaling
> the number by ten to compensate for performance differences:
>
 timeit.timeit('x=multiprocessing.Process(target=exit);x.start();x.join()', 
 number=1,globals = globals())
> 14.261007152497768
>
> Just for completeness and consistency, confirmed that adding a list
> comp around it doesn't change the timings:
>
 timeit.timeit('xx=[multiprocessing.Process(target=exit) for _ in 
 range(1)];[x.start() for x in xx];[x.join() for x in xx]', 
 number=1,globals = globals())
> 14.030426062643528
>
> But doing a hundred at a time and then joining them all cuts the time in half!
>
 timeit.timeit('xx=[multiprocessing.Process(target=exit) for _ in 
 range(100)];[x.start() for x in xx];[x.join() for x in xx]', 
 number=100,globals = globals())
> 5.470761131495237
>
> The difference is even more drastic with spawn, although since it's
> slower, I also lowered the number of iterations.
>
 ctx = multiprocessing.get_context('spawn')
 timeit.timeit('x=ctx.Process(target=exit);x.start();x.join()', 
 number=1000,globals = globals())
> 40.82687543518841
 timeit.timeit('xx=[ctx.Process(target=exit) for _ in 
 range(100)];[x.start() for x in xx];[x.join() for x in xx]', 
 number=10,globals = globals())8.566341979429126
> 8.566341979429126
>
> Would be curious to know if that's the same on Windows.
Yea, it's the same.  Watch your cpu utilization, and you will realize that your 
list comprehension is parallelizing the process startups.
> ChrisA
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/YFRM3LNO37B5JXNYPO2T7CAVYQRGAYES/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CICKBHTOAUOW3ARZ2Z4AYAKOWVGKWKVU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Final reminder: 3.7.8 / 3.6.11 cutoff Monday, last non-security bugfixes for 3.7.x

[Łukasz Langa]

> On 12 Jun 2020, at 19:51, Ivan Pozdeev via Python-Dev  
> wrote:
> 
> I would doubt the quality of tags maintenance at Github, too. E.g. 
> https://github.com/python/cpython/pull/12131 
>  is labeled 3.7 and 3.8 at BPO 
> but has no backport tags whatsoever at GH.
> So the search you gave is clearly insufficient.

Tags maintenance isn't about quality but rather about automation. Things 
without backport tags won't get backported unless somebody does it by hand. And 
even automatically created backports need to be created off of committed pull 
requests and need to pass tests to be mergeable. Given the limited time until 
the Monday cutoff, I'd say that almost all of your 700+ BPO open issues that 
list 3.7 as affected will miss the boat on the fixes.

- Ł___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LFAWYBRRIHR3D7VWSGOOBO6PLRA6SLXJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Final reminder: 3.7.8 / 3.6.11 cutoff Monday, last non-security bugfixes for 3.7.x

[Ivan Pozdeev via Python-Dev]

On 13.06.2020 3:49, Łukasz Langa wrote:



On 12 Jun 2020, at 19:51, Ivan Pozdeev via Python-Dev mailto:python-dev@python.org>> wrote:

I would doubt the quality of tags maintenance at Github, too. E.g.https://github.com/python/cpython/pull/12131is labeled 3.7 and 3.8 at 
BPO but has no backport tags whatsoever at GH.

So the search you gave is clearly insufficient.


Tags maintenance isn't about quality but rather about automation. Things without backport tags won't get backported unless somebody does 
it by hand. And even automatically created backports need to be created off of committed pull requests and need to pass tests to be 
mergeable. Given the limited time until the Monday cutoff, I'd say that almost all of your 700+ BPO open issues that list 3.7 as affected 
will miss the boat on the fixes.


Except they are not "mine" but everyone's.


I'll have some downtime next week so I could look into adding stuff to automatically update labels on existing issues and PRs upon release 
of a new major version to, say, bedevere. The lack of this is the reason for the discrepancy.





- Ł
--
Regards,
Ivan
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/MLDOEOYTKNZHCKTWMGI772Q7YLOVR4CZ/
Code of Conduct: http://python.org/psf/codeofconduct/