date:20220207

Thanks for trying it! I'm curious why it would be slower (perhaps less
locality? perhaps the ...Id... APIs have some other trick up their sleeve?)
but since it's also messier and less backwards compatible than just leaving
_Py_IDENTIFIER alone and just not using it, I'd say let's not spend more
time on that alternative and just focus on the two other horses still in
the race: immortal objects or what you have now.

On Mon, Feb 7, 2022 at 5:41 PM Eric Snow 
wrote:

> On Fri, Feb 4, 2022 at 8:25 PM Eric Snow 
> wrote:
> > On Fri, Feb 4, 2022, 16:03 Guido van Rossum  wrote:
> >> I wonder if a better solution than that PR wouldn't be to somehow
> change the implementation of _Py_IDENTIFIER() to do that,
> >
> > Yeah, I had the same realization today.  I'm going to try it out.
>
> I updated _Py_IDENTIFIER() to use a statically initialized string
> object and it isn't too bad.  The tricky thing is that PyASCIIObject
> expects to the data to be an array after the object.  So the field
> must be a pre-sized array (like I did in gh-30928).  That makes things
> messier.  The alternative is to do what Steve is suggesting.
>
> I ran the benchmarks and making _Py_IDENTIFIER() a statically
> initialized object makes things 2% slower (instead of 1% faster).
> There are a few things I could do to speed that up a little, but at
> best we'd get back to performance-neutral.
>
> -eric
>


-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/M47DIA75SZJO3ZCTJPLRTOUBYFIIZH4N/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: It's now time to deprecate the stdlib urllib module

2022-02-07 Thread Jim J. Jewett

There are problems with urllib.  With hindsight, it would have been nice to do 
a few things differently.  But that doesn't make migrating away from it any 
easier.

This thread has mentioned several "better" alternatives -- but with the 
exception of 3rd party Requests, the docs don't even mention them.

Saying "You can do better, but we won't tell you how" is pretty rude to 
beginners, and we should not do it.

Delegating to the operating system may be sensible for a production system, and 
there is nothing wrong with saying so in the docs, and it would be great if we 
made that easy.  But it is absolutely not a reasonable replacement for a 
straightforward (possibly inefficient and non-scalable) implementation written 
in python that people can read and use for reference.  urllib shouldn't be 
deprecated until we have a better solution to *that* use case that is also in 
the stdlib.  (That might well be worth doing, but it should happen before the 
deprecation.)

-jJ
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JI5CFS3WYXQEXKSEZH2ZTE3JJJ7AUAMW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Moving away from _Py_IDENTIFIER().

2022-02-07 Thread Eric Snow

On Fri, Feb 4, 2022 at 8:25 PM Eric Snow  wrote:
> On Fri, Feb 4, 2022, 16:03 Guido van Rossum  wrote:
>> I wonder if a better solution than that PR wouldn't be to somehow change the 
>> implementation of _Py_IDENTIFIER() to do that,
>
> Yeah, I had the same realization today.  I'm going to try it out.

I updated _Py_IDENTIFIER() to use a statically initialized string
object and it isn't too bad.  The tricky thing is that PyASCIIObject
expects to the data to be an array after the object.  So the field
must be a pre-sized array (like I did in gh-30928).  That makes things
messier.  The alternative is to do what Steve is suggesting.

I ran the benchmarks and making _Py_IDENTIFIER() a statically
initialized object makes things 2% slower (instead of 1% faster).
There are a few things I could do to speed that up a little, but at
best we'd get back to performance-neutral.

-eric
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/DDWOJLFOTXTZ35LMBCPH2DHFMCSVLHH5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Should we require IEEE 754 floating-point for CPython?

2022-02-07 Thread Gregory P. Smith

On Mon, Feb 7, 2022 at 4:52 PM Christopher Barker 
wrote:

> From the perspective of some that writes a lot of computational code:
>
> On Mon, Feb 7, 2022 at 10:25 AM Mark Dickinson  wrote:
>
>> - Should we require the presence of NaNs in order for CPython to build?
>> - Should we require IEEE 754 floating-point for
>> CPython-the-implementation?
>>
>
> Yes, and yes, together, as Mark suggests
>
> - Should we require IEEE 754 floating-point for Python-the-language?
>>
>
> I would say no, but it should be recommended -- see Victor's example of
> Micro Python -- though does anyone have an authority over that? IIUC, Micro
> Python already has a few differences -- but is anyone saying they shouldn't
> call it Python? Though, with a quick perusal of teh docs, micro Python does
> seem to support NaN and Inf, at least -- not sure about the rest of 754.
>

PEP 7 and PEP 11 are only about CPython. Other Python VMs full or partial
are free to do what they want with how they implement floats if at all.
(they're build time optional in MicroPython as are many Python language
syntax features)

While we're at it are  64 bit floats required for either cPython or Python
> the language?
>

CPython: yes.  we use a double.
Python the language: no.  (float is single precision on many micropython
platforms as it saves precious ram and performance, plus microcontroller
fpu hardware like an M4 is usually single precision 32bit)
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/X5LWPTTQHS7BZFARSJ3BO5PDO27CRKWO/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Should we require IEEE 754 floating-point for CPython?

2022-02-07 Thread Jim J. Jewett

- Should we require the presence of NaNs in order for CPython to build?
- Should we require IEEE 754 floating-point for CPython-the-implementation?
- Should we require IEEE 754 floating-point for Python-the-language?

I don't have strong opinions on the first two, but for the language definition, 
I think the most we should say is "if an implementation does not support IEEE 
754 floating-point, this must be mentioned in the documentation as an 
implementation limit."
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YEXX363XX6DS7ZC653RBLIPNQIHBVYTK/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Require a C compiler supporting C99 to build Python 3.11

2022-02-07 Thread Inada Naoki

https://www.ibm.com/docs/en/SSGH3R_16.1.0/pdf/getstart.pdf

As far as reading page 3,
xlclang fully supports C89/C99/C11.
xlc fully supports C89/C99, and partially supports C11.

On Tue, Feb 8, 2022 at 8:57 AM Brett Cannon  wrote:
>
>
>
> On Mon, Feb 7, 2022 at 9:12 AM Victor Stinner  wrote:
>>
>> Hi,
>>
>> I made a change to require C99  "NAN" constant and I'm was
>> asked to update the PEP 7 to clarify the C subset is needed to build
>> Python 3.11.
>>
>> Python 3.6 requires a subset of the C99 standard to build defined by the PEP 
>> 7:
>> https://www.python.org/dev/peps/pep-0007/
>>
>> I modified Python 3.11 to require more C99 features of the  header:
>>
>> * bpo-45440: copysign(), hypot(), isfinite(), isinf(), isnan(), round()
>> * bpo-46640: NAN
>>
>> After my NAN change (bpo-46640), Petr Viktorin asked me to update the
>> PEP 7. I proposed a change to simply say that "Python 3.11 and newer
>> versions use C99":
>> https://github.com/python/peps/pull/2309
>>
>> I would prefer to not have to give an exhaustive list of C99 features
>> used by CPython, since it's unclear to me what belongs to C99 or to
>> ISO C89. As I wrote before, Python already uses C99 features since
>> Python 3.6.
>>
>> On my PEP PR, Guido van Rossum asked me to escalate the discussion to
>> python-dev, so here I am :-)
>>
>> In "C99", the number "99" refers to the year 1999, the standard is now
>> 23 years old:
>> https://en.wikipedia.org/wiki/C99
>>
>> In 2022, C99 is now well supported by C compilers supported by Python:
>> GCC, clang, MSVC.
>
>
> I think if those compilers fully C99 at this point we should consider just 
> moving completely over to C99.
>
> -Brett
>
>>
>>
>> I don't know if AIX XLC supports C99. AIX provides a "c99" compiler
>> compatible with C99. It also seems like GCC is usable on AIX.
>>
>> I don't know if ICC supports C99. Python doesn't officially the ICC
>> compiler, the ICC buildbots are gone a few years ago. But sometimes I
>> make some changes to enhance the ICC support, when the change is small
>> enough.
>>
>> Note: Python also uses C11 , but it's not required: there
>> are fallbacks for compilers which don't support it.
>>
>> Victor
>> --
>> Night gathers, and now my watch begins. It shall not end until my death.
>> ___
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-le...@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at 
>> https://mail.python.org/archives/list/python-dev@python.org/message/J5FSP6J4EITPY5C2UJI7HSL2GQCTCUWN/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/ZLDOBJUVMTIRETVRNHPWWO5MBHTXYEW3/
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Inada Naoki  
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/XIMQWFVP7RJW7CTV74YPWI74L7ZUU2PX/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: RFC on PEP 675: Arbitrary Literal Strings

2022-02-07 Thread S Pradeep Kumar

On Mon, Feb 7, 2022 at 3:55 PM Brett Cannon  wrote:

> Can I suggest adding a "Specification" section (see
> https://www.python.org/dev/peps/pep-0012/#suggested-sections for the
> suggested sections to have in a PEP)?
>

Brett: I'm sorry about this oversight. I've fixed it in
https://github.com/python/peps/pull/2313

-- 
S Pradeep Kumar
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/IBIFCXEBTXJ4YM6PVC24BO6MRSG2DZ6R/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Should we require IEEE 754 floating-point for CPython?

2022-02-07 Thread Christopher Barker

>From the perspective of some that writes a lot of computational code:

On Mon, Feb 7, 2022 at 10:25 AM Mark Dickinson  wrote:

> - Should we require the presence of NaNs in order for CPython to build?
> - Should we require IEEE 754 floating-point for CPython-the-implementation?
>

Yes, and yes, together, as Mark suggests

- Should we require IEEE 754 floating-point for Python-the-language?
>

I would say no, but it should be recommended -- see Victor's example of
Micro Python -- though does anyone have an authority over that? IIUC, Micro
Python already has a few differences -- but is anyone saying they shouldn't
call it Python? Though, with a quick perusal of teh docs, micro Python does
seem to support NaN and Inf, at least -- not sure about the rest of 754.

While we're at it are  64 bit floats required for either cPython or Python
the language?

IIUC, the main platform that this was an issue for, for computationally
heavy code, was Cray -- but it seems they're 754 compatible now, yes?

Thanks for doing this,

-CHB

-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YBTMWKI2WCI3MBFVBSAXJ43LCZFUE6TY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Should we require IEEE 754 floating-point for CPython?

On Tue, Feb 8, 2022 at 12:19 AM Brett Cannon  wrote:
> Do we have a buildbot that has a CPU or OS that can't handle IEEE-754? What 
> are the chances we will get one? If the answers are "none" and "slim", then 
> it seems reasonable to require NaN and IEEE-754.

All CPU architectures of current buildbot workers use IEEE 754 (little
or big endian).

I looked at the following CPU architectures:

* x86 (Intel 32-bit)
* x86-64 (Intel 64-bit)
* armv7l (ARM 32-bit)
* aarch64 (ARM 64-bit)
* ppc64, ppc64le (PowerPC 64-bit)
* s390x (IBM)

There is also a "SPARCv9 Oracle Solaris 11.4" worker, but sadly it
seems to be disconnected for 3 months. The SPARCv9 architecture uses
IEEE 754.

On buildbots, I checked float.__getformat__("double"). It is logged as
"builtins.float.double_format" in the "pythoninfo" step of buidbots.

Little-endian examples:

* x86-64 (AMD64 Arch Linux Asan Debug 3.x): IEEE, little-endian
* x86 (Gentoo Non-Debug with X 3.x): IEEE, little-endian
* aarch64 (CentOS9 LTO + PGO 3.10): IEEE, little-endian
* aarch64 (Fedora Rawhide Clang 3.x): IEEE, little-endian
* aarch64: MacOS M1 (ARM64 macOS 3.x): IEEE, little-endian
* aarch64 (ARM64 Windows 3.x): IEEE, little-endian
* PPC64LE (CentOS9 3.10): IEEE, little-endian
* armv7l (ARM Raspbian 3.x): IEEE, little-endian

Big-endian examples:

* PPC64 (AIX 3.x): IEEE, big-endian
* s390x (Debian 3.x): IEEE, big-endian

float.__getformat__("double") is initialized by _PyFloat_InitState()
at Python startup with this code:
---
#if SIZEOF_DOUBLE == 8
{
double x = 9006104071832581.0;
if (memcmp(, "\x43\x3f\xff\x01\x02\x03\x04\x05", 8) == 0)
detected_double_format = ieee_big_endian_format;
else if (memcmp(, "\x05\x04\x03\x02\x01\xff\x3f\x43", 8) == 0)
detected_double_format = ieee_little_endian_format;
else
detected_double_format = unknown_format;
}
#else
detected_double_format = unknown_format;
#endif
---

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/4PJP7CB7UXQ6KRRWBBQUIQCLO2LL5JNO/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Require a C compiler supporting C99 to build Python 3.11

On Mon, Feb 7, 2022 at 9:12 AM Victor Stinner  wrote:

> Hi,
>
> I made a change to require C99  "NAN" constant and I'm was
> asked to update the PEP 7 to clarify the C subset is needed to build
> Python 3.11.
>
> Python 3.6 requires a subset of the C99 standard to build defined by the
> PEP 7:
> https://www.python.org/dev/peps/pep-0007/
>
> I modified Python 3.11 to require more C99 features of the  header:
>
> * bpo-45440: copysign(), hypot(), isfinite(), isinf(), isnan(), round()
> * bpo-46640: NAN
>
> After my NAN change (bpo-46640), Petr Viktorin asked me to update the
> PEP 7. I proposed a change to simply say that "Python 3.11 and newer
> versions use C99":
> https://github.com/python/peps/pull/2309
>
> I would prefer to not have to give an exhaustive list of C99 features
> used by CPython, since it's unclear to me what belongs to C99 or to
> ISO C89. As I wrote before, Python already uses C99 features since
> Python 3.6.
>
> On my PEP PR, Guido van Rossum asked me to escalate the discussion to
> python-dev, so here I am :-)
>
> In "C99", the number "99" refers to the year 1999, the standard is now
> 23 years old:
> https://en.wikipedia.org/wiki/C99
>
> In 2022, C99 is now well supported by C compilers supported by Python:
> GCC, clang, MSVC.


I think if those compilers fully C99 at this point we should consider just
moving completely over to C99.

-Brett


>
> I don't know if AIX XLC supports C99. AIX provides a "c99" compiler
> compatible with C99. It also seems like GCC is usable on AIX.
>
> I don't know if ICC supports C99. Python doesn't officially the ICC
> compiler, the ICC buildbots are gone a few years ago. But sometimes I
> make some changes to enhance the ICC support, when the change is small
> enough.
>
> Note: Python also uses C11 , but it's not required: there
> are fallbacks for compilers which don't support it.
>
> Victor
> --
> Night gathers, and now my watch begins. It shall not end until my death.
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/J5FSP6J4EITPY5C2UJI7HSL2GQCTCUWN/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZLDOBJUVMTIRETVRNHPWWO5MBHTXYEW3/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: RFC on PEP 675: Arbitrary Literal Strings

Can I suggest adding a "Specification" section (see
https://www.python.org/dev/peps/pep-0012/#suggested-sections for the
suggested sections to have in a PEP)?

When you first pull up the PEP it goes from "Rationale" to "Valid Locations
for LiteralString". As a reader that doesn't point me at a valid start
point to capture the key details I need to know to understand what changes
to Python are being proposed. For instance, is LiteralString supported to
be a new built-in? Is it a subclass of str? I have no way to know from the
first section of the PEP that isn't trying to convince me why this PEP is
worth reading.

On Mon, Feb 7, 2022 at 3:38 PM Pradeep Kumar Srinivasan 
wrote:

> PEP 675 [1] introduces a supertype for precise literal string types, such
> as `Literal["foo"]`, called `LiteralString`.
>
> The PEP allows libraries to distinguish the type of command strings from
> data. Powerful, command-executing libraries try to prevent security
> vulnerabilities by accepting arguments separately from the SQL query or
> shell command. However, these libraries have no way to prevent programmers
> from, say, using f-strings to embed arguments within the command string,
> which can allow malicious users to execute arbitrary commands (called
> "SQL/shell injection"). With this PEP, libraries can rely on type checkers
> to prevent such common, undesired uses.
>
> Scala has a very similar concept that is used to prevent SQL injection at
> compile time. [5] We also discuss the drawbacks of alternative approaches,
> such as security linters, full taint analysis, and NewTypes [6].
>
> Since we have reached consensus on the PEP in typing-sig [2], we wanted to
> get your comments and suggestions before submitting to the Steering
> Council.
>
> This PEP is mainly useful for type checking. There are no changes to
> Python syntax or runtime behavior.
>
> Discussions:
> + typing-sig: [2]
> + BPO issue suggesting using this PEP to prevent vulnerabilities in
> loggers: [4]
> + Reddit thread on r/Python discussing this PEP: [3]
>
> Thanks,
> S Pradeep Kumar
> Graham Bleaney
>
> [1]: https://www.python.org/dev/peps/pep-0675/
> [2]:
> https://mail.python.org/archives/list/typing-...@python.org/thread/VB74EHNM4RODDFM64NEEEBJQVAUAWIAW/
> [3]:
> https://www.reddit.com/r/Python/comments/r71lzi/pep_675_arbitrary_literal_strings/
> [4]: https://bugs.python.org/issue46200
> [5]:
> https://www.python.org/dev/peps/pep-0675/#literal-string-types-in-scala
> [6]: https://www.python.org/dev/peps/pep-0675/#rejected-alternatives
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/AXR3ZBEZF2UTBC2RFCASBTQB4ZYNQGET/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/7O66E32XD6ZWLQHW55OPWBJ6AQNF7SYT/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: It's now time to deprecate the stdlib urllib module

On Mon, Feb 7, 2022 at 4:56 AM Steve Dower  wrote:

> On 2/6/2022 4:44 PM, Christian Heimes wrote:
> > If I had the power and time, then I would replace urllib with a simpler,
> > reduced HTTP client that uses platform's HTTP library under the hood
> > (WinHTTP on Windows, NSURLSession (?) on macOS, Web API for Emscripten,
> > maybe curl on Linux/BSD). For non-trivial HTTP requests, httpx or
> > aiohttp are much better suited than urllib.
>
> I'm +1 on this, though I think it would have to be in place before the
> "two releases until removal" kicked in for urllib.request.
>

Yes, we definitely couldn't deprecate anything regarding downloading over
HTTP w/o having a replacement in place.

I am not even considering deprecating urllib.parse.

>
> The stdlib can't get by without at least the basic functionality of curl
> built in natively. But we can do this on most platforms without
> vendoring OpenSSL, which is a HUGE win. Then our default behaviour could
> correctly use proxies (including auto-config), CA certificate bundles,
> integrated authentication, and other OS features that are currently
> ignored by our core.
>

I also agree this is the best of the 2 options, although I would also
accept Christian's other option of a more targeted, tight,
standards-compliant solution if that would somehow lead to less maintenance
overhead. And when I say "less maintenance overhead," I really mean it: I
would question whether following redirects as an option is worth the
overhead in this scenario. I'm very much thinking of this from a
bootstrap/script/learning scenario and pushing people towards e.g. httpx
for anything fancier.

>
> Chances are we could keep simple urlopen() calls in place, and use the
> deprecation as a "potential change of behaviour" without necessarily
> having to break the API. I'm yet to come across a case where making a
> trivial urlopen() call _better_ would break things (the cases I've seen
> that would break are things like "using an OpenSSL environment variable
> to configure something that I wish had been automatic").
>

We could try to get fancy and only raise DeprecationWarning in cases where
things won't work to extend when we consider pushing people to the better
API.

>
> The nature of network/internet access is that we have to break things
> periodically anyway, because all the code that was written over the last
> 30+ years is eventually going to be found to be exploitable. I'd be
> quite happy to say "Python gives you what your OS gives you; update the
> OS for fixes".
>

Exactly. My guideline for this whole idea would be that if it doesn't make
sense in a beginner course that says to "download an HTML page and count
all the anchor tags," then it's too fancy for the stdlib. And that should
be enough to bootstrap installers which then get you httpx. Otherwise the
networking stack moves too fast (from a security POV) and requires unique
knowledge to get right that we have simply not kept up as much as we would
like. I think it's okay to admit it might be time to trim with part of the
stdlib down to something that we can manage easily (but we *cannot* drop
the ability to download something over HTTPS).
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/VP2WXOBWPGAX7UIH25DWRSYWFEDNINNU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] RFC on PEP 675: Arbitrary Literal Strings

2022-02-07 Thread Pradeep Kumar Srinivasan

PEP 675 [1] introduces a supertype for precise literal string types, such as 
`Literal["foo"]`, called `LiteralString`.

The PEP allows libraries to distinguish the type of command strings from data. 
Powerful, command-executing libraries try to prevent security vulnerabilities 
by accepting arguments separately from the SQL query or shell command. However, 
these libraries have no way to prevent programmers from, say, using f-strings 
to embed arguments within the command string, which can allow malicious users 
to execute arbitrary commands (called "SQL/shell injection"). With this PEP, 
libraries can rely on type checkers to prevent such common, undesired uses.

Scala has a very similar concept that is used to prevent SQL injection at 
compile time. [5] We also discuss the drawbacks of alternative approaches, such 
as security linters, full taint analysis, and NewTypes [6].

Since we have reached consensus on the PEP in typing-sig [2], we wanted to get 
your comments and suggestions before submitting to the Steering Council. 

This PEP is mainly useful for type checking. There are no changes to Python 
syntax or runtime behavior.

Discussions:
+ typing-sig: [2]
+ BPO issue suggesting using this PEP to prevent vulnerabilities in loggers: [4]
+ Reddit thread on r/Python discussing this PEP: [3]

Thanks,
S Pradeep Kumar
Graham Bleaney

[1]: https://www.python.org/dev/peps/pep-0675/
[2]: 
https://mail.python.org/archives/list/typing-...@python.org/thread/VB74EHNM4RODDFM64NEEEBJQVAUAWIAW/
[3]: 
https://www.reddit.com/r/Python/comments/r71lzi/pep_675_arbitrary_literal_strings/
[4]: https://bugs.python.org/issue46200
[5]: https://www.python.org/dev/peps/pep-0675/#literal-string-types-in-scala
[6]: https://www.python.org/dev/peps/pep-0675/#rejected-alternatives
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/AXR3ZBEZF2UTBC2RFCASBTQB4ZYNQGET/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Require a C compiler supporting C99 to build Python 3.11

2022-02-07 Thread Jeremiah Vivian

On Mon, Feb 7, 2022 at 5:11 PM Victor Stinner vstin...@python.org wrote:
> I made a change to require C99  "NAN" constant and I'm was
> asked to update the PEP 7 to clarify the C subset is needed to build
> Python 3.11.
What if it was built with MSVC which is usually just C90?
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JSQX3OY4XMTBLLCERP3JDJXPPT2BMQQ6/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Replace debug runtime checks in release mode with assertions in debug mode

On Mon, Feb 7, 2022 at 8:59 AM Victor Stinner  wrote:

> On Mon, Feb 7, 2022 at 5:48 PM Guido van Rossum  wrote:
> > So you're proposing to completely get rid of those three?
>
> I don't propose to remove them, but only call them if Python is built
> in debug mode. Or remove them from the release build, unless
> ./configure --with-assertions is used.
>
>
> > And you're sure that each and every single call to any of those is
> better off being an assert()?
>
> For many years, many C extensions raised an exception *and* returned a
> result: that's a bug. The strange part is that in some cases, the
> exceptions is somehow ignored and the program continues running fine.
>
> That's why I added _Py_CheckFunctionResult() and _Py_CheckSlotResult()
> which helped to catch such bugs. But before that, these programs were
> running fine :-)
>
> So it's not fully clear to me there was really a bug or it's just that
> Python became more pedantic :-)
>
>
> About PyErr_BadInternalCall(): in 10 years, I saw a few SystemError
> raised by this function, but usually when I hacked on Python. It's
> really rare to hit such bug.
>
>
> > (I still haven't gotten into the habit of building in debug mode by
> default, in part because it *isn't* the default when you invoke ./configure
> or PCbuild/build.bat.)
>
> If you don't develop C extensions, the release mode is faster and enough
> ;-)
>
> Ah. I don't know if CIs like GitHub Actions and Azure Pipelines
> provide Python debug builds. If if it's not the case, it would be nice
> to have the choice :-)
>

They do not:
https://github.com/actions/python-versions/blob/797eb71c41e47d194f563c7ef01790d734534788/builders/ubuntu-python-builder.psm1#L35-L38
.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/VRISRYZVU47PQRWKHN77CV5545SKUI5O/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Should we require IEEE 754 floating-point for CPython?

On Mon, Feb 7, 2022 at 12:31 PM Gregory P. Smith  wrote:

>
>
> On Mon, Feb 7, 2022 at 11:06 AM Victor Stinner 
> wrote:
>
>> Hi Mark,
>>
>> Aha, good, you posted an email to python-dev, good :-) Last days, I
>> was trying to collect more data about this topic, especially find
>> platforms which *don't* support IEEE 754, before posting to
>> python-dev.
>>
>> Nowadays, outside museums, it's hard to find computers which don't
>> implement IEEE 754.
>>
>> * Since 1998, IBM S/390 supports IEEE 754.
>> * Compaq/DEC VAX didn't use IEEE 754, but are no longer built since
>> 2000 (you cant still use Python 3.10 if it can be built on it!)
>> * Some Cray computers like Cray SV1 (1998) don't implement IEEE 754,
>> but other Cray computers like Cray T90 (1995) implement IEEE 754.
>>
>> There are embedded devices with no hardware FPU: in this case, the FPU
>> is implemented in software. I expect it to implement IEEE 754. Is
>> CPython a good target for such small CPUs which have no hardware FPU?
>> MicroPython may better fit their needs.
>>
>> On the other side, all moderns CPU architectures support IEEE 754:
>> Intel x86, ARM, IBM Power and PowerPC, Compaq/DEC Alpha, HP PA-RISC,
>> Motorola 68xxx and 88xxx, SGI R-, Sun SPARC.
>>
>> Sources:
>>
>> *
>> https://en.wikipedia.org/wiki/Floating-point_arithmetic#IEEE_754:_floating_point_in_modern_computers
>> *
>> https://stackoverflow.com/questions/2234468/do-any-real-world-cpus-not-use-ieee-754
>>
>>
>> On Mon, Feb 7, 2022 at 7:25 PM Mark Dickinson  wrote:
>> > - Should we require the presence of NaNs in order for CPython to build?
>> > - Should we require IEEE 754 floating-point for
>> CPython-the-implementation?
>>
>> In the past, when we deprecated the support for an old platform, we
>> didn't suddenly remove the code. We made sure that it's no longer
>> possible to build on it. So if anyone notices, it's easy to revert
>> (ex: remove the few lines in configure).
>>
>> Would it make sense to trigger a build error on plaforms which don't
>> support IEEE 754 in Python 3.11, and later decide if it's time to
>> remove the code in in Python 3.12?
>>
>> Well. If you ask *me*, I'm in favor of removing the code right now. If
>> someone needs to support a platform which doesn't support IEEE 754,
>> the support can be maintained *outside* the Python source code, as
>> external patches or as a Git fork, no?
>>
>> Honestly, I never got access to any machine which doesn't support IEEE
>> 754 (or nobody told me!).
>>
>>
>> > - Should we require IEEE 754 floating-point for Python-the-language?
>>
>> Right now, I have no opinion on this question.
>>
>>
>> > Note that on the current main branch there's a Py_NO_NAN macro that
>> builders can define to indicate that NaNs aren't supported, but the Python
>> build is currently broken if Py_NO_NAN is defined (see
>> https://bugs.python.org/issue46656). If the answer to the first question
>> is "No", then we need to fix the build under Py_NO_NAN. That's not a big
>> deal - perhaps a couple of hours of work.
>>
>> My comment on bpo-46656: "Python uses Py_NAN without "#ifdef Py_NAN"
>> guard since 2008. Building Python with Py_NO_NAN never worked. Nobody
>> reported such build failure in the last 14 years..."
>>
>> If anyone would try building/using Python on a platform without NAN, I
>> would expect that we would get a bug report or an email. I'm not aware
>> of anything like that, so it seems like nobody uses Python on such
>> platform.
>>
>> Victor
>>
>
> It's 2022. We should just require both NaN and IEEE-754.
>
> By the time a system is large enough to build and run CPython, it is
> exceedingly unlikely that it will not be able to do that. Even if it means
> software emulation of IEEE-754 floating point on something large yet odd
> enough not to have a FPU.
>
> The places where non-IEEE-754 and non-NaN floating point are likely to
> exist are in specialized parallelized coprocessor hardware. Not the general
> purpose CPU running CPython.
>

Do we have a buildbot that has a CPU or OS that can't handle IEEE-754? What
are the chances we will get one? If the answers are "none" and "slim", then
it seems reasonable to require NaN and IEEE-754.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QTJAUZ476W7UXL2LQR3VWZBB6CVQ7OA6/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Should we require IEEE 754 floating-point for CPython?

On Mon, Feb 7, 2022 at 11:02 AM Victor Stinner  wrote:

> Hi Mark,
>
> Aha, good, you posted an email to python-dev, good :-) Last days, I
> was trying to collect more data about this topic, especially find
> platforms which *don't* support IEEE 754, before posting to
> python-dev.
>
> Nowadays, outside museums, it's hard to find computers which don't
> implement IEEE 754.
>
> * Since 1998, IBM S/390 supports IEEE 754.
> * Compaq/DEC VAX didn't use IEEE 754, but are no longer built since
> 2000 (you cant still use Python 3.10 if it can be built on it!)
> * Some Cray computers like Cray SV1 (1998) don't implement IEEE 754,
> but other Cray computers like Cray T90 (1995) implement IEEE 754.
>
> There are embedded devices with no hardware FPU: in this case, the FPU
> is implemented in software. I expect it to implement IEEE 754. Is
> CPython a good target for such small CPUs which have no hardware FPU?
> MicroPython may better fit their needs.
>
> On the other side, all moderns CPU architectures support IEEE 754:
> Intel x86, ARM, IBM Power and PowerPC, Compaq/DEC Alpha, HP PA-RISC,
> Motorola 68xxx and 88xxx, SGI R-, Sun SPARC.
>

RISC-V also supports the 2008 IEEE-754 spec:
https://riscv.org/technical/specifications/ .

-Brett



>
> Sources:
>
> *
> https://en.wikipedia.org/wiki/Floating-point_arithmetic#IEEE_754:_floating_point_in_modern_computers
> *
> https://stackoverflow.com/questions/2234468/do-any-real-world-cpus-not-use-ieee-754
>
>
> On Mon, Feb 7, 2022 at 7:25 PM Mark Dickinson  wrote:
> > - Should we require the presence of NaNs in order for CPython to build?
> > - Should we require IEEE 754 floating-point for
> CPython-the-implementation?
>
> In the past, when we deprecated the support for an old platform, we
> didn't suddenly remove the code. We made sure that it's no longer
> possible to build on it. So if anyone notices, it's easy to revert
> (ex: remove the few lines in configure).
>
> Would it make sense to trigger a build error on plaforms which don't
> support IEEE 754 in Python 3.11, and later decide if it's time to
> remove the code in in Python 3.12?
>
> Well. If you ask *me*, I'm in favor of removing the code right now. If
> someone needs to support a platform which doesn't support IEEE 754,
> the support can be maintained *outside* the Python source code, as
> external patches or as a Git fork, no?
>
> Honestly, I never got access to any machine which doesn't support IEEE
> 754 (or nobody told me!).
>
>
> > - Should we require IEEE 754 floating-point for Python-the-language?
>
> Right now, I have no opinion on this question.
>
>
> > Note that on the current main branch there's a Py_NO_NAN macro that
> builders can define to indicate that NaNs aren't supported, but the Python
> build is currently broken if Py_NO_NAN is defined (see
> https://bugs.python.org/issue46656). If the answer to the first question
> is "No", then we need to fix the build under Py_NO_NAN. That's not a big
> deal - perhaps a couple of hours of work.
>
> My comment on bpo-46656: "Python uses Py_NAN without "#ifdef Py_NAN"
> guard since 2008. Building Python with Py_NO_NAN never worked. Nobody
> reported such build failure in the last 14 years..."
>
> If anyone would try building/using Python on a platform without NAN, I
> would expect that we would get a bug report or an email. I'm not aware
> of anything like that, so it seems like nobody uses Python on such
> platform.
>
> Victor
> --
> Night gathers, and now my watch begins. It shall not end until my death.
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/PPVBMCKL5E2MVTWVFGSI4VTT6IAJ5QBF/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/WLYIMCEBOKWHYX4XIYLRRX4VHGZGMSJO/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Should we require IEEE 754 floating-point for CPython?

2022-02-07 Thread Gregory P. Smith

On Mon, Feb 7, 2022 at 11:06 AM Victor Stinner  wrote:

> Hi Mark,
>
> Aha, good, you posted an email to python-dev, good :-) Last days, I
> was trying to collect more data about this topic, especially find
> platforms which *don't* support IEEE 754, before posting to
> python-dev.
>
> Nowadays, outside museums, it's hard to find computers which don't
> implement IEEE 754.
>
> * Since 1998, IBM S/390 supports IEEE 754.
> * Compaq/DEC VAX didn't use IEEE 754, but are no longer built since
> 2000 (you cant still use Python 3.10 if it can be built on it!)
> * Some Cray computers like Cray SV1 (1998) don't implement IEEE 754,
> but other Cray computers like Cray T90 (1995) implement IEEE 754.
>
> There are embedded devices with no hardware FPU: in this case, the FPU
> is implemented in software. I expect it to implement IEEE 754. Is
> CPython a good target for such small CPUs which have no hardware FPU?
> MicroPython may better fit their needs.
>
> On the other side, all moderns CPU architectures support IEEE 754:
> Intel x86, ARM, IBM Power and PowerPC, Compaq/DEC Alpha, HP PA-RISC,
> Motorola 68xxx and 88xxx, SGI R-, Sun SPARC.
>
> Sources:
>
> *
> https://en.wikipedia.org/wiki/Floating-point_arithmetic#IEEE_754:_floating_point_in_modern_computers
> *
> https://stackoverflow.com/questions/2234468/do-any-real-world-cpus-not-use-ieee-754
>
>
> On Mon, Feb 7, 2022 at 7:25 PM Mark Dickinson  wrote:
> > - Should we require the presence of NaNs in order for CPython to build?
> > - Should we require IEEE 754 floating-point for
> CPython-the-implementation?
>
> In the past, when we deprecated the support for an old platform, we
> didn't suddenly remove the code. We made sure that it's no longer
> possible to build on it. So if anyone notices, it's easy to revert
> (ex: remove the few lines in configure).
>
> Would it make sense to trigger a build error on plaforms which don't
> support IEEE 754 in Python 3.11, and later decide if it's time to
> remove the code in in Python 3.12?
>
> Well. If you ask *me*, I'm in favor of removing the code right now. If
> someone needs to support a platform which doesn't support IEEE 754,
> the support can be maintained *outside* the Python source code, as
> external patches or as a Git fork, no?
>
> Honestly, I never got access to any machine which doesn't support IEEE
> 754 (or nobody told me!).
>
>
> > - Should we require IEEE 754 floating-point for Python-the-language?
>
> Right now, I have no opinion on this question.
>
>
> > Note that on the current main branch there's a Py_NO_NAN macro that
> builders can define to indicate that NaNs aren't supported, but the Python
> build is currently broken if Py_NO_NAN is defined (see
> https://bugs.python.org/issue46656). If the answer to the first question
> is "No", then we need to fix the build under Py_NO_NAN. That's not a big
> deal - perhaps a couple of hours of work.
>
> My comment on bpo-46656: "Python uses Py_NAN without "#ifdef Py_NAN"
> guard since 2008. Building Python with Py_NO_NAN never worked. Nobody
> reported such build failure in the last 14 years..."
>
> If anyone would try building/using Python on a platform without NAN, I
> would expect that we would get a bug report or an email. I'm not aware
> of anything like that, so it seems like nobody uses Python on such
> platform.
>
> Victor
>

It's 2022. We should just require both NaN and IEEE-754.

By the time a system is large enough to build and run CPython, it is
exceedingly unlikely that it will not be able to do that. Even if it means
software emulation of IEEE-754 floating point on something large yet odd
enough not to have a FPU.

The places where non-IEEE-754 and non-NaN floating point are likely to
exist are in specialized parallelized coprocessor hardware. Not the general
purpose CPU running CPython.

-gps
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UQXG7E6ONOQFCIES4PQIMJFZDXNGQJTM/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Should we require IEEE 754 floating-point for CPython?

Hi Mark,

Aha, good, you posted an email to python-dev, good :-) Last days, I
was trying to collect more data about this topic, especially find
platforms which *don't* support IEEE 754, before posting to
python-dev.

Nowadays, outside museums, it's hard to find computers which don't
implement IEEE 754.

* Since 1998, IBM S/390 supports IEEE 754.
* Compaq/DEC VAX didn't use IEEE 754, but are no longer built since
2000 (you cant still use Python 3.10 if it can be built on it!)
* Some Cray computers like Cray SV1 (1998) don't implement IEEE 754,
but other Cray computers like Cray T90 (1995) implement IEEE 754.

There are embedded devices with no hardware FPU: in this case, the FPU
is implemented in software. I expect it to implement IEEE 754. Is
CPython a good target for such small CPUs which have no hardware FPU?
MicroPython may better fit their needs.

On the other side, all moderns CPU architectures support IEEE 754:
Intel x86, ARM, IBM Power and PowerPC, Compaq/DEC Alpha, HP PA-RISC,
Motorola 68xxx and 88xxx, SGI R-, Sun SPARC.

Sources:

* 
https://en.wikipedia.org/wiki/Floating-point_arithmetic#IEEE_754:_floating_point_in_modern_computers
* 
https://stackoverflow.com/questions/2234468/do-any-real-world-cpus-not-use-ieee-754

On Mon, Feb 7, 2022 at 7:25 PM Mark Dickinson  wrote:
> - Should we require the presence of NaNs in order for CPython to build?
> - Should we require IEEE 754 floating-point for CPython-the-implementation?

In the past, when we deprecated the support for an old platform, we
didn't suddenly remove the code. We made sure that it's no longer
possible to build on it. So if anyone notices, it's easy to revert
(ex: remove the few lines in configure).

Would it make sense to trigger a build error on plaforms which don't
support IEEE 754 in Python 3.11, and later decide if it's time to
remove the code in in Python 3.12?

Well. If you ask *me*, I'm in favor of removing the code right now. If
someone needs to support a platform which doesn't support IEEE 754,
the support can be maintained *outside* the Python source code, as
external patches or as a Git fork, no?

Honestly, I never got access to any machine which doesn't support IEEE
754 (or nobody told me!).

> - Should we require IEEE 754 floating-point for Python-the-language?

Right now, I have no opinion on this question.

> Note that on the current main branch there's a Py_NO_NAN macro that builders 
> can define to indicate that NaNs aren't supported, but the Python build is 
> currently broken if Py_NO_NAN is defined (see 
> https://bugs.python.org/issue46656). If the answer to the first question is 
> "No", then we need to fix the build under Py_NO_NAN. That's not a big deal - 
> perhaps a couple of hours of work.

My comment on bpo-46656: "Python uses Py_NAN without "#ifdef Py_NAN"
guard since 2008. Building Python with Py_NO_NAN never worked. Nobody
reported such build failure in the last 14 years..."

If anyone would try building/using Python on a platform without NAN, I
would expect that we would get a bug report or an email. I'm not aware
of anything like that, so it seems like nobody uses Python on such
platform.

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PPVBMCKL5E2MVTWVFGSI4VTT6IAJ5QBF/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Should we require IEEE 754 floating-point for CPython?

2022-02-07 Thread Mark Dickinson

On Mon, Feb 7, 2022 at 5:11 PM Victor Stinner  wrote:

> I made a change to require C99  "NAN" constant [...]


There's a separate discussion topic lurking here. It's equally in need of
discussion here (IMO), but it's orthogonal to the "should we require C99"
discussion. I've changed the subject line accordingly to try to avoid
derailing that discussion.

Unlike the other things Victor mentions ("copysign", "round", etc.), the
NAN macro is not required to be present by C99. Instead, the standard says
that "NAN is defined if and only if the implementation supports quiet NaNs
for the float type" (C99 §7.12p5).

Victor is proposing in GH-31160
 to require the presence of
the NAN macro in order for CPython to build, which under C99 is equivalent
to requiring that the C float type supports quiet NaNs. That's not the same
as requiring IEEE 754 floating-point, but it's not far off - there aren't
many non-IEEE 754 floating-point formats that support NaNs. (Historically,
there are essentially none, but it seems quite likely that there will be at
least some non-IEEE 754 formats in the future that support NaNs; Google's
bfloat16 format is one example.)

So there (at least) three questions here:

- Should we require the presence of NaNs in order for CPython to build?
- Should we require IEEE 754 floating-point for CPython-the-implementation?
- Should we require IEEE 754 floating-point for Python-the-language?

For the first two, I'd much prefer either to not require NaNs, or to go the
whole way and require IEEE 754 for CPython. Requiring NaNs but not IEEE 754
feels like an awkward halfway house: in practice, it would be just as
restrictive as requiring IEEE 754, but without the benefits of making that
requirement explicit (e.g., being able to get rid of non-IEEE 754 paths in
existing code, and being able to tell users that they can reasonably expect
IEEE 754-conformant behaviour).

Note that on the current main branch there's a Py_NO_NAN macro that
builders can define to indicate that NaNs aren't supported, but the Python
build is currently broken if Py_NO_NAN is defined (see
https://bugs.python.org/issue46656). If the answer to the first question is
"No", then we need to fix the build under Py_NO_NAN. That's not a big deal
- perhaps a couple of hours of work.

-- 
Mark
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/GUIR2HZHFV2TDS7GUQHAHFSA4IC3QLMZ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Replace debug runtime checks in release mode with assertions in debug mode

2022-02-07 Thread Terry Reedy


On 2/7/2022 11:48 AM, Guido van Rossum wrote:

(I still haven't gotten into the habit of building in debug mode by 
default, in part because it *isn't* the default when you invoke 
./configure or PCbuild/build.bat.)


On Windows, I have build.bat in the directory containing the repository 
and maintenance workspaces.  It calls PCbuild\build.bat -d.  I usually 
build with "..\build" rather than the direct call above.  Easier to type 
and does the usual right thing.


--
Terry Jan Reedy

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6QTVXJQBKM2GU35ZJ5BMC4RL5MY5K74M/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Require a C compiler supporting C99 to build Python 3.11

Hi,

I made a change to require C99 "NAN" constant and I'm was
asked to update the PEP 7 to clarify the C subset is needed to build
Python 3.11.

Python 3.6 requires a subset of the C99 standard to build defined by the PEP 7:
https://www.python.org/dev/peps/pep-0007/

I modified Python 3.11 to require more C99 features of the header:

* bpo-45440: copysign(), hypot(), isfinite(), isinf(), isnan(), round()
* bpo-46640: NAN

After my NAN change (bpo-46640), Petr Viktorin asked me to update the
PEP 7. I proposed a change to simply say that "Python 3.11 and newer
versions use C99":
https://github.com/python/peps/pull/2309

I would prefer to not have to give an exhaustive list of C99 features
used by CPython, since it's unclear to me what belongs to C99 or to
ISO C89. As I wrote before, Python already uses C99 features since
Python 3.6.

On my PEP PR, Guido van Rossum asked me to escalate the discussion to
python-dev, so here I am :-)

In "C99", the number "99" refers to the year 1999, the standard is now
23 years old:
https://en.wikipedia.org/wiki/C99

In 2022, C99 is now well supported by C compilers supported by Python:
GCC, clang, MSVC.

I don't know if AIX XLC supports C99. AIX provides a "c99" compiler
compatible with C99. It also seems like GCC is usable on AIX.

I don't know if ICC supports C99. Python doesn't officially the ICC
compiler, the ICC buildbots are gone a few years ago. But sometimes I
make some changes to enhance the ICC support, when the change is small
enough.

Note: Python also uses C11 , but it's not required: there
are fallbacks for compilers which don't support it.

Victor
--
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/J5FSP6J4EITPY5C2UJI7HSL2GQCTCUWN/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Replace debug runtime checks in release mode with assertions in debug mode

On Mon, Feb 7, 2022 at 5:48 PM Guido van Rossum  wrote:
> So you're proposing to completely get rid of those three?

I don't propose to remove them, but only call them if Python is built
in debug mode. Or remove them from the release build, unless
./configure --with-assertions is used.

> And you're sure that each and every single call to any of those is better off 
> being an assert()?

For many years, many C extensions raised an exception *and* returned a
result: that's a bug. The strange part is that in some cases, the
exceptions is somehow ignored and the program continues running fine.

That's why I added _Py_CheckFunctionResult() and _Py_CheckSlotResult()
which helped to catch such bugs. But before that, these programs were
running fine :-)

So it's not fully clear to me there was really a bug or it's just that
Python became more pedantic :-)

About PyErr_BadInternalCall(): in 10 years, I saw a few SystemError
raised by this function, but usually when I hacked on Python. It's
really rare to hit such bug.

> (I still haven't gotten into the habit of building in debug mode by default, 
> in part because it *isn't* the default when you invoke ./configure or 
> PCbuild/build.bat.)

If you don't develop C extensions, the release mode is faster and enough ;-)

Ah. I don't know if CIs like GitHub Actions and Azure Pipelines
provide Python debug builds. If if it's not the case, it would be nice
to have the choice :-)

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/KJNUCD3X37ZPHE2UYSJBO5ZHYFQTMPZA/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Replace debug runtime checks in release mode with assertions in debug mode

So you're proposing to completely get rid of those three? And you're sure
that each and every single call to any of those is better off being an
assert()?

(I still haven't gotten into the habit of building in debug mode by
default, in part because it *isn't* the default when you invoke ./configure
or PCbuild/build.bat.)

On Mon, Feb 7, 2022 at 8:45 AM Victor Stinner  wrote:

> On Mon, Feb 7, 2022 at 5:38 PM Guido van Rossum  wrote:
> > ISTM this is better discussed on a case-by-case basis than as a blanket
> policy change. (The latter could end up causing a flood of trivial PRs from
> wannabe-contributors who found and fix yet another violation of the policy,
> which is both a nuisance for reviewers and a risk of introducing bugs due
> to being over-zealous.)
>
> That's why I propose to only change code using these 3 functions:
>
> * PyErr_BadInternalCall(),
> * _Py_CheckFunctionResult()
> * _Py_CheckSlotResult()
>
> Victor
> --
> Night gathers, and now my watch begins. It shall not end until my death.
>


-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/MGVQ4EAXBH7EYZB27Z3TGT3IUU4IU6GK/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: It's now time to deprecate the stdlib urllib module

2022-02-07 Thread Stéfane Fermigier

My two cents:

1. I’ve grepped ("ag-ed" acually) through all my code bases, and as noted
by others, urllib.parse is used in many places. urllib.request never is, as
we've been using requests or httpx instead.

IMHO and for the context I'm using it (YMMV), urllib.parse is useful and
should be kept (could be replaced by third-party libs like furl or yarl,
but probably not a great idea).

2. Obviously bootstrapping pip or similar package management tools should
be a concern.

3. Overall, I think the days were "battery included" was a positive
argument are over. I'd rather make the standard library leaner, and
focussed on core language constructs. The only advantage that I can see is
that having stuff in the standard lib can reduce fragmentation, at least
initially, and ensure a very high level of quality and support, but at some
point in the future history has shown us that usually a better alternative
ends up emerging.

4. When deprecating and removing stuff from the stdlib and if there are no
dependency issues, it should be possible to more the components to their
own dedicated packages, maybe under an "extra" or "legacy" organisation.
The question of support stays open, though.

Regards,

  S.

On 6 Feb 2022 at 23:39:55, Gregory P. Smith  wrote:

>
>
> On Sun, Feb 6, 2022 at 9:13 AM Paul Moore  wrote:
>
>> On Sun, 6 Feb 2022 at 16:51, Christian Heimes 
>> wrote:
>>
>> > The urllib package -- and to some degree also the http package -- are
>> > constant source of security bugs. The code is old and the parsers for
>> > HTTP and URLs don't handle edge cases well. Python core lacks a true
>> > maintainer of the code. To be honest, we have to admit defeat and be up
>> > front that urllib is not up to the task for this decade. It was designed
>> > written during a more friendly, less scary time on the internet.
>> >
>> > If I had the power and time, then I would replace urllib with a simpler,
>> > reduced HTTP client that uses platform's HTTP library under the hood
>> > (WinHTTP on Windows, NSURLSession (?) on macOS, Web API for Emscripten,
>> > maybe curl on Linux/BSD). For non-trivial HTTP requests, httpx or
>> > aiohttp are much better suited than urllib.
>> >
>> > The second best option is to reduce the feature set of urllib to core
>> > HTTP (no ftp, proxy, HTTP auth) and a partial rewrite with stricter,
>> > more standard conform parsers for urls, query strings, and RFC 2822
>> > instead of RFC 822 for headers.
>>
>> I'd likely be fine with either of these two options. I'm not worried
>> about supporting "advanced" uses. But having no way of getting a file
>> from the internet without relying on 3rd party packages seems like a
>> huge gap in functionality for a modern language. And having to use a
>> 3rd party library to parse URLs will simply push more people to use
>> home-grown regexes rather than something safe and correct. Remember
>> that a lot of Python users are not professional software developers,
>> but scientists, data analysts, and occasional users, for whom the
>> existence of something in the stdlib is the *only* reason they have
>> any idea that URLs need specialised parsing in the first place.
>>
>> And while we all like to say 3rd party modules are great, the reality
>> is that they provide a genuine problem for many of these
>> non-specialist users - and I say that as a packaging specialist and
>> pip maintainer. The packaging ecosystem is *not* newcomer-friendly in
>> the way that core Python is, much as we're trying to improve that
>> situation.
>>
>> I've said it previously, but I'll reiterate - IMO this *must* have a
>> PEP, and that PEP must be clear that the intention is to *remove*
>> urllib, not simply to "deprecate and then think about it". That could
>> be making it part of PEP 594, or a separate PEP, but one way or
>> another it needs a PEP.
>>
>
> This would need to be it's own PEP.  urllib et. al. are used by virtually
> everybody.  They're highly used batteries.
>
> I'm -1 on deprecating it for that reason alone.
>
> Christian proposes that having a simpler scope rewrite of it might be
> nice, but I think disruption to the world and loss of trust in Python would
> be similar either way.
>
> -gps
>
>
>>
>> Paul
>> ___
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-le...@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-dev@python.org/message/KT6TGUBTLLETHES2OVVGZWSGYC5JCEKC/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/QMSFZBQJFWKFFE3LFQLQE2AT6WKMLPGL/
>

[Python-Dev] Re: Replace debug runtime checks in release mode with assertions in debug mode

On Mon, Feb 7, 2022 at 5:38 PM Guido van Rossum  wrote:
> ISTM this is better discussed on a case-by-case basis than as a blanket 
> policy change. (The latter could end up causing a flood of trivial PRs from 
> wannabe-contributors who found and fix yet another violation of the policy, 
> which is both a nuisance for reviewers and a risk of introducing bugs due to 
> being over-zealous.)

That's why I propose to only change code using these 3 functions:

* PyErr_BadInternalCall(),
* _Py_CheckFunctionResult()
* _Py_CheckSlotResult()

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/HLLNK35HIF3LYBUS35TOATEV6A6LHOXT/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Replace debug runtime checks in release mode with assertions in debug mode

ISTM this is better discussed on a case-by-case basis than as a blanket
policy change. (The latter could end up causing a flood of trivial PRs from
wannabe-contributors who found and fix yet another violation of the policy,
which is both a nuisance for reviewers and a risk of introducing bugs due
to being over-zealous.)

On Mon, Feb 7, 2022 at 7:59 AM Victor Stinner  wrote:

> Hi,
>
> I would like to replace runtime checks on C function arguments with
> assertions. For example, replace:
>
> PyObject *
> PyDict_GetItemWithError(PyObject *op, PyObject *key)
> {
> if (!PyDict_Check(op)) {
> PyErr_BadInternalCall();
> return NULL;
> }
> ...
> }
>
> with:
>
> PyObject *
> PyDict_GetItemWithError(PyObject *op, PyObject *key)
> {
> assert(PyDict_Check(op));
> ...
> }
>
> I'm not asking to remove all of them. For example,
> PyList_GetItem(list, index) will continuing raising an IndexError if
> the index is out of range.
>
> I'm asking to replace runtime checks with assertions when the C API is
> "obviously" misused: replace PyErr_BadInternalCall(),
> _Py_CheckFunctionResult() and _Py_CheckSlotResult() with assertions.
> The exact scope should be defined.
>
> --
>
> Python tries to be nice to C extensions authors and calls
> PyErr_BadInternalCall() to emit a nice SystemError exception when the
> C API is misused. There are many sanity checks run at runtime. We pay
> the cost of these runtime checks even if a C extension is well written
> (has no bug).
>
> In Python 3.6, I added the _Py_CheckFunctionResult() function to check
> the result of any function call. A function must return a non-NULL
> object and not raises an exception, or return NULL and raises an
> exception. Otherwise, SystemError is raised. For example,
> PyObject_Call() uses it to check the PyTypeObject.tp_call result.
>
> In Python 3.10, I added a similar _Py_CheckSlotResult() function to
> check the result of slot functions. For example, PyObject_Size()
> checks the result of the PyTypeObject.tp_as_sequence.sq_length slot
> function.
>
> --
>
> I modified Python 3.8 to be able to use a debug Python build as a
> drop-in replacement of a Python release build: debug and release
> builds are now ABI compatible. It is no longer needed to rebuild C
> extensions in debug mode. My first goal was to be able to use gdb on a
> Python binary built with -O0:
>
> https://developers.redhat.com/articles/2021/09/08/debugging-python-c-extensions-gdb
>
> A debug Python build adds many other debug features useful to develop
> C extensions:
> https://docs.python.org/dev/using/configure.html#python-debug-build
>
> IMO with Python 3.11, it's now easy enough to get a Python debug build
> to develop or debug your C extensions. End users should not have to
> pay the price of these runtime checks.
>
> On most Linux distribution, a Python debug build is available as
> "python3-debug" executable. To check if you are using a debug build,
> check if the sys.gettotalrefcount() function is available.
>
> By the way, it is also possible to build Python in release mode with
> assertions using "./configure --with-assertions":
> https://docs.python.org/dev/using/configure.html#cmdoption-with-assertions
>
> So, what do you think?
>
> --
>
> In 2019, I already proposed this idea on the bug tracker, but I
> abandoned my idea:
> https://bugs.python.org/issue37406
>
> Victor
> --
> Night gathers, and now my watch begins. It shall not end until my death.
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/RR3DJ4IZVSFKZUE5AV6OUCJ3QPYXD26Z/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YLXFI7L3XTY5HPLNQL56WGDJY5YUBJD4/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Moving away from _Py_IDENTIFIER().

Yes, please build it; I can't quite follow what you're getting at (probably
since I don't know the Windows APIs).

On Mon, Feb 7, 2022 at 4:39 AM Steve Dower  wrote:

> On 2/5/2022 4:09 PM, Guido van Rossum wrote:
> > On Sat, Feb 5, 2022 at 5:18 AM Steve Dower  > > wrote:
> >
> > The gap this has with what I'd like to see is that it will only work
> > for
> > compile-time strings. If instead it could work for an arbitrary
> uint8_t
> > pointer, rather than an embedded array, that allows even runtime
> > strings
> > to be very cheaply passed into the interpreter (and yes, the caller
> has
> > to manage the lifetime, but that's pretty easy).
> >
> >
> > What's the use case for that, that isn't covered by
> PyUnicode_FromString()?
>
> No dynamic memory allocation (and hence, no deallocation) for any case
> where the original string is already managed, which is really quite
> common. Access to the memory manager is what requires thread
> affinity/the GIL for primitive object construction, and string copies -
> especially with transcoding - are the expensive part.
>
> For cases where strings are just passed around but never manipulated
> (e.g. a lot of filesystem operations, or runtime/interpreter
> configuration), the string may never have to be decoded at all. It's
> almost as good as a tagged pointer, but without breaking our existing
> object model (assuming all the PyUnicode_* functions learn how to handle
> them, which is necessary).
>
> But it's purely a transparent optimisation for most users, with an added
> opportunity for those using native APIs and probably decent complexity
> for us as maintainers. There are a lot of edge cases to handle that I'm
> sure people will use to shoot down the idea, which is why rather than
> debate details here I'd rather build it and define its boundaries of
> usefulness, though for now there's plenty of stuff I'd rather do than
> both of these, so it remains an idea for now :)
>
> Cheers,
> Steve
>


-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/I5KB6CADRNVBF6ZUAC7BPKEEY4ORQ4Z4/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Python-Dev] Replace debug runtime checks in release mode, with assertions in debug mode

2022-02-07 Thread Matti Picus




On 7/2/22 17:55, Victor Stinner wrote:

End users should not have to
pay the price of these runtime checks.



Is the cost of the checks measurable and significant? Surely sanitizing 
arguments to prevent segfaults is worth a few nano-seconds?



Matti

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OK43VQKWNXOCPLXQQI3TH5KUXHUUZZLM/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Replace debug runtime checks in release mode with assertions in debug mode

Hi,

I would like to replace runtime checks on C function arguments with
assertions. For example, replace:

PyObject *
PyDict_GetItemWithError(PyObject *op, PyObject *key)
{
if (!PyDict_Check(op)) {
PyErr_BadInternalCall();
return NULL;
}
...
}

with:

PyObject *
PyDict_GetItemWithError(PyObject *op, PyObject *key)
{
assert(PyDict_Check(op));
...
}

I'm not asking to remove all of them. For example,
PyList_GetItem(list, index) will continuing raising an IndexError if
the index is out of range.

I'm asking to replace runtime checks with assertions when the C API is
"obviously" misused: replace PyErr_BadInternalCall(),
_Py_CheckFunctionResult() and _Py_CheckSlotResult() with assertions.
The exact scope should be defined.

--

Python tries to be nice to C extensions authors and calls
PyErr_BadInternalCall() to emit a nice SystemError exception when the
C API is misused. There are many sanity checks run at runtime. We pay
the cost of these runtime checks even if a C extension is well written
(has no bug).

In Python 3.6, I added the _Py_CheckFunctionResult() function to check
the result of any function call. A function must return a non-NULL
object and not raises an exception, or return NULL and raises an
exception. Otherwise, SystemError is raised. For example,
PyObject_Call() uses it to check the PyTypeObject.tp_call result.

In Python 3.10, I added a similar _Py_CheckSlotResult() function to
check the result of slot functions. For example, PyObject_Size()
checks the result of the PyTypeObject.tp_as_sequence.sq_length slot
function.

--

I modified Python 3.8 to be able to use a debug Python build as a
drop-in replacement of a Python release build: debug and release
builds are now ABI compatible. It is no longer needed to rebuild C
extensions in debug mode. My first goal was to be able to use gdb on a
Python binary built with -O0:
https://developers.redhat.com/articles/2021/09/08/debugging-python-c-extensions-gdb

A debug Python build adds many other debug features useful to develop
C extensions:
https://docs.python.org/dev/using/configure.html#python-debug-build

IMO with Python 3.11, it's now easy enough to get a Python debug build
to develop or debug your C extensions. End users should not have to
pay the price of these runtime checks.

On most Linux distribution, a Python debug build is available as
"python3-debug" executable. To check if you are using a debug build,
check if the sys.gettotalrefcount() function is available.

By the way, it is also possible to build Python in release mode with
assertions using "./configure --with-assertions":
https://docs.python.org/dev/using/configure.html#cmdoption-with-assertions

So, what do you think?

--

In 2019, I already proposed this idea on the bug tracker, but I
abandoned my idea:
https://bugs.python.org/issue37406

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/RR3DJ4IZVSFKZUE5AV6OUCJ3QPYXD26Z/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 674 – Disallow using macros as l-value (version 2)

On Mon, Jan 31, 2022 at 3:57 PM Petr Viktorin  wrote:
> Remove the point from the Abstract?
> (...)
> I think you should make it clear that the percentage was taken after
> fixing many of the projects.

Thanks for your review.

I pushed a change to take your remarks in account:
https://github.com/python/peps/commit/e9456773643c7dcca584ae0100db42639b93f04e

I simplified the Abstract and added a new "Backward compatibility >
Statistics" section to show all numbers at one place:
https://python.github.io/peps/pep-0674/#statistics

I also moved projects already fixed in the "Top 5000 PyPI" category or
in a new "Other affected projects" category, depending if they are in
the top 5000 PyPI projects or not. It should now be easier to
understand statistics.

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CJRZSM53LVDKJPVEEFJYUERW7B3LDF5L/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Slowly bend the C API towards the limited API to get a stable ABI for everyone

2022-02-07 Thread Petr Viktorin


On 07. 02. 22 15:29, Victor Stinner wrote:

On Mon, Feb 7, 2022 at 2:26 PM Victor Stinner  wrote:

CPython is also affected by these issues, but the benefits of PEP 674
(alone) are too indirect, so I chose to avoid mentioning CPython
issues directly, to avoid confusion.


A concrete example of problem caused by exposing structures in the C
API (unrelated to PEP 674). It's a tricky problem...

typedef struct {
 PyObject_VAR_HEAD
 Py_hash_t ob_shash;
 char ob_sval[1];
} PyBytesObject;

The "char ob_sval[1];" syntax used to declare an array is an undefined
behavior if the array is longer in memory. On a bytes object of 4
bytes, accessing ob_sval[3] works, but is an undefined behavior.

=> see https://bugs.python.org/issue40120 for details

The problem can be solved by using "char ob_sval[];" syntax, but we
cannot use this syntax in the public C header, since it causes
compiler errors if the header is built with a C++ compiler (not to
build Python itself, but build a C++ extension using the Python C
API). Removing the structure from the public C API would solve the C++
issue.


That sounds like a good candidate for deprecation, and adding proper 
getters if we didn't have them. But I don't think we actually need to 
remove the deprecated struct from the public C API.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/3NYGRFIDNBL2MBR66O4HKYVZYS2WEVP5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Slowly bend the C API towards the limited API to get a stable ABI for everyone

On Mon, Feb 7, 2022 at 2:26 PM Victor Stinner  wrote:
> CPython is also affected by these issues, but the benefits of PEP 674
> (alone) are too indirect, so I chose to avoid mentioning CPython
> issues directly, to avoid confusion.

A concrete example of problem caused by exposing structures in the C
API (unrelated to PEP 674). It's a tricky problem...

typedef struct {
PyObject_VAR_HEAD
Py_hash_t ob_shash;
char ob_sval[1];
} PyBytesObject;

The "char ob_sval[1];" syntax used to declare an array is an undefined
behavior if the array is longer in memory. On a bytes object of 4
bytes, accessing ob_sval[3] works, but is an undefined behavior.

=> see https://bugs.python.org/issue40120 for details

The problem can be solved by using "char ob_sval[];" syntax, but we
cannot use this syntax in the public C header, since it causes
compiler errors if the header is built with a C++ compiler (not to
build Python itself, but build a C++ extension using the Python C
API). Removing the structure from the public C API would solve the C++
issue.

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LXZCN6IZYZH6XTHXKWSRIFJOHTCB4MAN/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Slowly bend the C API towards the limited API to get a stable ABI for everyone

2022-02-07 Thread Petr Viktorin

On 07. 02. 22 14:26, Victor Stinner wrote:

On Mon, Feb 7, 2022 at 2:08 PM Petr Viktorin wrote:

Basically, instead of "We'll remove this API now because it prevents
moving to a hypothetical moving garbage collector", it should be "Here
is a moving garbage collector that speeds Python up by 30%, but to add
it we need to remove these 30 deprecated APIs". The deprecation can be
proactive, but not the removal.

PEP 674 gives 3 concrete examples of issues already affecting the
CPython nogil fork, HPy and GraalPython. They are not hypothetical.

HPy and GraalPython are different projects, and their issues don't
affect current users of CPython.
It would be great to make an easy-to-use API that would be compatible
with HPy and GraalPython. But not at the cost of breaking existing code.
Are the advantages of moving to HPy or Graalpython worth porting the
code? IMO that's a question each extension author should be able to ask.
They shouldn't be forced to do the port by CPython.

As for nogil, that's a promising experiment. If it turns out to be
successful, let's remove the parts that are blocking it. Do we already
know which parts those are?

CPython is also affected by these issues, but the benefits of PEP 674
(alone) are too indirect, so I chose to avoid mentioning CPython
issues directly, to avoid confusion.

IMO, CPython issues are the ones most relevant to CPython. If yo're
bringing them up, could you be more specific about them?
If we don't discuss them, how do we know if there better ways to solve
them than what you're proposing?

It's possible to workaround them: more or less copy/paste CPython
inefficient code, as PyPy did years ago. The problem is that the
workaround is inefficient and so PyPy cpyext remains slow. Well, HPy
address the cpyext performance problem for PyPy and GraalPython ;-)

Hooray! So, the issue is addressed, and we don't need to break code that
doesn't care about performance on PyPy! Sounds perfect!
More seriously: I expect that there are a lot of extension authors that
value API stability over performance on PyPy. They're doing nothing
wrong and they don't deserve to be punished for using API they had
available (and docs they had available) some years ago.

I don't think that the question is if there is a real problem or not.
The question is what's the best migration plan to move existing C
extensions towards a better API which don't suffer from these
problems.

Once 95% of C extensions will use the limited C API, we would still
not be able to change Python internals, because of the 5% remaining C
extensions which are stuck at the legacy C API.

And don't forget extensions that use the "bad" parts of limited API,
like using Py_TYPE as l-value in Python 3.10-.

IMO, the required ABI changes will be more drastic than the API changes.

When we want to change CPython internals, IMO we should know the exact
set of APIs that'll need to be removed. Without an actual proposed
change, I don't think we can know that.
HPy is a good indicator of what API should be deprecated (i.e. where we
should find a better way of doing things), but I don't think it gives
good reasons for breaking changes.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/4A7X7WT2MGXXIZUFEMUE6QBVRWPUMZC2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Slowly bend the C API towards the limited API to get a stable ABI for everyone