[Python-Dev] Re: PEP 667: Consistent views of namespaces

2021-08-23 Thread Gregory P. Smith
Just adding a datapoint, searching our internal codebase at work, I only
found two things that I'd consider to be uses of PyEval_GetLocals() outside
of CPython itself:

https://github.com/earwig/mwparserfromhell/blob/develop/src/mwparserfromhell/parser/ctokenizer/tokenizer.c#L220
https://github.com/numba/numba/blob/master/numba/_dispatcher.cpp#L664

The bulk of uses are naturally with CPython's own ceval.c and similar.

-gps

On Mon, Aug 23, 2021 at 9:02 AM Guido van Rossum  wrote:

> On Mon, Aug 23, 2021 at 8:46 AM Mark Shannon  wrote:
>
>> Hi Guido,
>>
>> On 23/08/2021 3:53 pm, Guido van Rossum wrote:
>> > On Mon, Aug 23, 2021 at 4:38 AM Mark Shannon > > > wrote:
>> >
>> > Hi Nick,
>> >
>> > On 22/08/2021 4:51 am, Nick Coghlan wrote:
>> >
>> >  > If Mark's claim that PyEval_GetLocals() could not be fixed was
>> > true then
>> >  > I would be more sympathetic to his proposal, but I know it isn't
>> > true,
>> >  > because it still works fine in the PEP 558 implementation (it
>> even
>> >  > immediately sees changes made via proxies, and proxies see
>> > changes to
>> >  > extra variables). The only truly unfixable public API is
>> >  > PyFrame_LocalsToFast().
>> >
>> > You are making claims that seem inconsistent with each other.
>> > Namely, you are claiming that:
>> >
>> > 1. That the result of locals() is ephemeral.
>> > 2. That PyEval_GetLocals() returns a borrowed reference.
>> >
>> > This seems impossible, as you can't return a borrowed reference to
>> > an emphemeral object. That's just a pointer to freed memory.
>> >
>> > Do `locals()` and `PyEval_GetLocals()` behave differently?
>> >
>> >
>> > That is my understanding, yes. in PEP 558 locals() returns a snapshot
>> > dict, the Python-level f_locals property returns a fresh proxy that has
>> > no state except a pointer to the frame, and PyEval_GetLocals() returns
>> a
>> > borrowed reference to the dict that's stored on the frame's C-level
>> > f_locals attribute
>>
>> Can we avoid describing the C structs in any of these PEPs?
>>
>> It confuses readers having Python attributes and "C-level attributes"
>> (C struct fields?).
>> It also restricts the implementation unnecessarily.
>>
>> (E.g. the PyFrameObject doesn't have a `f_locals` field in 3.11:
>>
>> https://github.com/python/cpython/blob/main/Include/cpython/frameobject.h#L7
>> )
>>
>
> I'd be happy to. Nick's PEP still references it (and indeed it is very
> confusing) and I took it from him. And honestly it would be nice to have a
> specific short name for it, rather than circumscribing it with "an internal
> dynamic snapshot stored on the frame object " :-)
>
>
>> >
>> > (In my "crazy" proposal all that is the same.)
>>
>> >
>> > Is the result of `PyEval_GetLocals()` cached, but `locals()` not?
>> >
>> >
>> > I wouldn't call it a cache -- deleting it would affect the semantics,
>> > not just the performance. But yes, it returns a reference to an object
>> > that is owned by the frame, just as it does in 3.10 and before.
>> >
>> > If that were the case, then it is a bit confusing, but could work.
>> >
>> >
>> > Yes, see my "crazy" proposal.
>> >
>> > Would PyEval_GetLocals() be defined as something like this?
>> >
>> > (add _locals_cache attribute to the frame which is initialized to
>> NULL).
>> >
>> > def PyEval_GetLocals():
>> >   frame._locals_cache attribute = locals()
>> >   return borrow(frame._locals_cache attribute)
>> >
>> >
>> > Nah, the dict returned by PyEval_GetLocals() is stored in the frame's
>> > C-level f_locals attribute, which is consulted by the Python-level
>> > f_locals proxy -- primarily to store "extra" variables, but IIUC in
>> > Nick's latest version it is also still used to cache by that proxy.
>> > Nick's locals() just returns dict(sys._getframe().f_locals).
>>
>> The "extra" variables must be distinct from the result of locals() as
>> that includes both extras and "proper" variables.
>> If we want to cache the locals(), it needs to be distinct from the extra
>> variables.
>>
>
> I don't care that much about caching locals(), but it seems we're bound to
> cache any non-NULL result from PyEval_GetLocals(), since it returns a
> borrowed reference. So they may be different things, with different
> semantics, if we don't cache locals().
>
>
>> A debugger setting extra variables in a function that that is also
>> accessed by a C call to PyEval_GetLocals() is going to be incredibly
>> rare. Let's not worry about efficiency here.
>>
>
> Agreed.
>
> >
>> > None of this is clear (at least not to me) from PEP 558.
>> >
>> >
>> > One problem with PEP 558 is that it's got too many words, and it's
>> > lacking a section that crisply describes the semantics of the proposed
>> > implementation. I've suggested to Nick that he add a section with
>> > pseudo-code for the implementation, like you did in yours.
>> >
>> > (PS, did you read my PS 

[Python-Dev] Re: PEP 467 feedback from the Steering Council

2021-08-22 Thread Gregory P. Smith
On Tue, Aug 10, 2021 at 3:48 PM Christopher Barker 
wrote:

> On Tue, Aug 10, 2021 at 3:00 PM  wrote:
>
>> The history of bytes/bytearray is a dual-purpose view.  It can be used in
>> a string-like way to emulate Python 2 string handling (hence all the usual
>> string methods and a repr that displays in a string-like fashion).  It can
>> also be used as an array of numbers, 0 to 255 (hence the list methods and
>> having an iterator of ints).  ISTM that the authors of this PEP reject or
>> want to discourage the latter use cases.
>>
>
> I didn't read it that way, but if so, please no, I"d rather see the former
> use cases discouraged. ISTM that the Py2 string handling is still needed
> for working with mixed binary / text data -- but that should be a pretty
> specialized use case. spelling the way to create a byte, byte() sure makes
> more sense in any other context.
>
>
>> ... anything where a C programmer would an array of unsigned chars).
>>
>
> or any programmer would use an array of unsigned 8bit integers :-) numpy
> spells it: `np.uint8`, and the the type in the C99 stdint.h is `uint8_t`.
> My point is that for anyone not an "old time" C programmer, or even a
> Python2 programmer, the  "character is an unsigned 8 bit int" concept is
> alien and confusing, not a helpful mnemonic.
>
>
>> For example, creating a single byte with bytes([0x1f]) isn't pleasant,
>> obvious, or fast.
>>
>
> no, though bytes([31]) isn't horrible ;-)   (despite coding for over four
> decades, I'm still not comfortable with hex notation)
>
> I say it's not horrible, because bytes is a Sequence of bytes (or integer
> values between 0 and 255), initializing it with an iterable seems pretty
> reasonable, that's how we initialize most (all?) other sequences after all.
> And compatible with array.array and numpy arrays.
>

I consider bytes([31]) notation to be horrible API design because a simple
easy to make typo of omitting the [] or using () and forgetting the
tupleizing comma turns it into a different valid call with an entirely
different meaning.  bytes([31]) vs bytes((31)) vs bytes(31).

It's also ugly to anyone who thinks about what bytecode is generated and
executed in order to do it.  an entire new list object with a single
element referring to a tiny int is created and destroyed just to create a
b'\037' object?  An optimizer pass to fix that up at the bytecode level
isn't easy as it can only be done when it can prove that `bytes` has not
been reassigned to something other than the builtin.  Near impossible in a
lot of code.  bytes.fromint(31) isn't much better in the bytecode regard,
but at least a temporary list is not being created.

As much as I think that bytes(size: int) was a bad idea to have as an API -
bytearray(size: int) is fine and useful as it is mutable - that ship sailed
and getting rid of it would break some odd code.  It doesn't have much use,
so adding fromsize(size: int) methods don't sound very compelling as it
just adds yet another way to do the same thing.  we should just live with
that specific wart.

`bchr` as a builtin... I'm with the others on saying no to any new builtin
that isn't expected to see frequent use.  bchr won't see frequent use.

`bytes.fromint` seems fine.  others are proposing `bytes.byte` for that.  I
don't *like* to argue over names (the last stage of anything) but I do need
to point out how that sounds to read.  It falls victim to API stuttering.
"bytes dot byte" or "bytes byte" doesn't convey much to a reader in English
as the difference is a subtle "s".  "bytes dot from int" or "bytes from
int" is quite clear.  (avoiding stuttering in API design was popularized by
golang - it's a good thing to strive for in any language)  It's times like
this that i wish Python had chosen consistent camelCase, CapWords, or
snake_case in all API names as conjoinedwords aren't great. But they are
sadly consistent with our past sins.

One thing never mentioned in the PEP.  If you expect a primary use of the
fromint (aka bchr builtin that isn't going to happen) to be called on
constant values often.  Why are we adding name lookups and function calls
to this?  Why not address the elephant in the room and allow for decimal
values to be written as an escape sequence within bytes literals?

b'\d31' for example to say "decimal byte 31".  Proposal: Only values 0-255
with no leading zero should be accepted when parsing such an escape.  (Do
not bother adding the same feature for codepoints in unicode strs; leave
that to later if someone shows actual demand).  This can't address the
bytearray need, but that's been true of bytearray for ages, a common way to
create them is via a copy from transient bytes objects.  bytearray(b'\d31')
isn't much different than bytearray.fromint(31).  one less name lookup.

Why not add a \d escape? Introducing a new escape is fraught with peril as
existing \d's within b'' literals in code could change meaning.  backwards
compatibility fail.  But one that is easy to check for 

[Python-Dev] Re: Making code object APIs unstable

2021-08-17 Thread Gregory P. Smith
+cc: cython-devel

background reading for those new to the thread:
https://mail.python.org/archives/list/python-dev@python.org/thread/ZWTBR5ESYR26BUIVMXOKPFRLGGYDJSFC/

On Tue, Aug 17, 2021 at 9:47 AM Victor Stinner  wrote:

> Since Cython is a common consumer of this C API, can somone please dig
> into Cython to see exactly what it needs in terms of API? How does
> Cython create all arguments of the __Pyx_PyCode_New() macro? Does it
> copy an existing function to only override some fields, something like
> CodeType.replace(field=new_value)?
>
> If possible, I would prefer that Cython only uses the *public* C API.
> Otherwise, it will be very likely that Cython will break at every
> single Python release. Cython has a small team to maintain the code
> base, whereas CPython evolves much faster with a larger team.
>
> Victor
>

I don't claim knowledge of Cython internals, but the two places it appears
to call it's __Pyx_PyCode_New macro are:

https://github.com/cython/cython/blob/master/Cython/Utility/Exceptions.c#L769
in __Pyx_CreateCodeObjectForTraceback() - this one already has a `#if
CYTHON_COMPILING_IN_LIMITED_API` code path option in it.

 and

https://github.com/cython/cython/blob/master/Cython/Compiler/ExprNodes.py#L9722
in CodeObjectNode.generate_result_code()  that creates PyCodeObject's for
CyFunction instances per its comment.  Slightly described in this comment
http://google3/third_party/py/cython/files/Cython/Compiler/ExprNodes.py?l=397.
I don't see anything obvious mentioning the limited API in that code
generator.

it'd be best to loop in Cython maintainers for more of an idea of Cython's
intents and needs with PyCode_New APIs.  I've cc'd cython-de...@python.org.

-Greg


> On Tue, Aug 17, 2021 at 8:51 AM Gregory P. Smith  wrote:
> >
> > Doing a search of a huge codebase (work), the predominant user of
> PyCode_New* APIs appears to be checked in Cython generated code (in all
> sorts of third_party OSS projects). It's in the boilerplate that Cython
> extensions make use of via it's __Pyx_PyCode_New macro.
> https://github.com/cython/cython/blob/master/Cython/Utility/ModuleSetupCode.c#L470
> >
> > I saw very few non-Cython uses.  There are some, but at a very quick
> first glance they appear simple - easy enough to reach out to the projects
> with a PR to update their code.
> >
> > The Cython use will require people to upgrade Cython and regenerate
> their code before they can use the Python version that changes these. That
> is not an uncommon thing for Cython. It's unfortunate that many projects on
> ship generated sources rather than use Cython at build time, but that isn't
> _our_ problem to solve. The more often we change internal APIs that things
> depend on, the more people will move their projects towards doing the right
> thing with regards to either not using said APIs or rerunning an up to date
> code generator as part of their build instead of checking in generated
> unstable API using sources.
> >
> > -gps
> >
> >
> > On Mon, Aug 16, 2021 at 8:04 PM Guido van Rossum 
> wrote:
> >>
> >> On Mon, Aug 16, 2021 at 4:44 PM Nick Coghlan 
> wrote:
> >>>
> >>> [...]
> >>> A cloning-with-replacement API that accepted the base code object and
> the "safe to modify" fields could be a good complement to the API
> deprecation proposal.
> >>
> >>
> >> Yes (I forgot to mention that).
> >>
> >>>
> >>> Moving actual "from scratch" code object creation behind the
> Py_BUILD_CORE guard with an underscore prefix on the name would also make
> sense, since it defines a key piece of the compiler/interpreter boundary.
> >>
> >>
> >> Yeah, we have _PyCode_New() for that.
> >>
> >>>
> >>> Cheers,
> >>> Nick.
> >>>
> >>> P.S. Noting an idea that won't work, in case anyone else reading the
> thread was thinking the same thing: a "PyType_FromSpec" style API won't
> help here, as the issue is that the compiler is now doing more work up
> front and recording that extra info in the code object for the interpreter
> to use. There is no way to synthesise that info if it isn't passed to the
> constructor, as it isn't intrinsically recorded in the opcode sequence.
> >>
> >>
> >> That's the API style that _PyCode_New() uses (thanks to Eric who IIRC
> pushed for this and implemented it). You gave me an idea now: the C
> equivalent to .replace() could use the same input structure; one can leave
> fields NULL that should be copied from the original unmodified.
> >>
> >> --
> >> --Guido van Rossum (python

[Python-Dev] Re: Making code object APIs unstable

2021-08-17 Thread Gregory P. Smith
Doing a search of a huge codebase (work), the predominant user of
PyCode_New* APIs appears to be checked in Cython generated code (in all
sorts of third_party OSS projects). It's in the boilerplate that Cython
extensions make use of via it's __Pyx_PyCode_New macro.
https://github.com/cython/cython/blob/master/Cython/Utility/ModuleSetupCode.c#L470

I saw very few non-Cython uses.  There are some, but at a very quick first
glance they appear simple - easy enough to reach out to the projects with a
PR to update their code.

The Cython use will require people to upgrade Cython and regenerate their
code before they can use the Python version that changes these. That is not
an uncommon thing for Cython. It's unfortunate that many projects on ship
generated sources rather than use Cython at build time, but that isn't
_our_ problem to solve. The more often we change internal APIs that things
depend on, the more people will move their projects towards doing the right
thing with regards to either not using said APIs or rerunning an up to date
code generator as part of their build instead of checking in generated
unstable API using sources.

-gps


On Mon, Aug 16, 2021 at 8:04 PM Guido van Rossum  wrote:

> On Mon, Aug 16, 2021 at 4:44 PM Nick Coghlan  wrote:
>
>> [...]
>> A cloning-with-replacement API that accepted the base code object and the
>> "safe to modify" fields could be a good complement to the API deprecation
>> proposal.
>>
>
> Yes (I forgot to mention that).
>
>
>> Moving actual "from scratch" code object creation behind the
>> Py_BUILD_CORE guard with an underscore prefix on the name would also make
>> sense, since it defines a key piece of the compiler/interpreter boundary.
>>
>
> Yeah, we have _PyCode_New() for that.
>
>
>> Cheers,
>> Nick.
>>
>> P.S. Noting an idea that won't work, in case anyone else reading the
>> thread was thinking the same thing: a "PyType_FromSpec" style API won't
>> help here, as the issue is that the compiler is now doing more work up
>> front and recording that extra info in the code object for the interpreter
>> to use. There is no way to synthesise that info if it isn't passed to the
>> constructor, as it isn't intrinsically recorded in the opcode sequence.
>>
>
> That's the API style that _PyCode_New() uses (thanks to Eric who IIRC
> pushed for this and implemented it). You gave me an idea now: the C
> equivalent to .replace() could use the same input structure; one can leave
> fields NULL that should be copied from the original unmodified.
>
> --
> --Guido van Rossum (python.org/~guido)
> *Pronouns: he/him **(why is my pronoun here?)*
> 
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/NWYMCDAMS4YRJ7ESXNWQ6MIBSRAZEXEM/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/67DMIW7NQE6M6LEPLANXKZQEFOFVPBBL/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Does anyone use threading debug PYTHONTHREADDEBUG=1 env var? Can I remove it?

2021-07-07 Thread Gregory P. Smith
On Wed, Jul 7, 2021 at 2:28 AM Victor Stinner  wrote:

> Hi,
>
> Does anyone use threading debug PYTHONTHREADDEBUG=1 env var on a
> Python debug build? If not, can I just remove it?
>
> --
>
> To fix a race condition at thread exit on Linux using the glibc, I
> removed calls to pthread_exit() (PyThread_exit_thread()) in the
> _thread module:
>
>https://bugs.python.org/issue44434
>
> A side effect of this change is the removal of the
> "PyThread_exit_thread called" threading debug log when using
> PYTHONTHREADDEBUG=1 environment variable.
>
> I never used PYTHONTHREADDEBUG. I just tried it and it produces tons
> of output in stdout about locks. It looks basically useless because it
> produces way too many logs, and it pollutes stdout (ex: most Python
> tests fail when it's enabled).
>
> This debug mode requires to build Python in debug mode (./configure
> --with-pydebug):
>
>https://docs.python.org/dev/using/configure.html#python-debug-build
>
> IMO there are enough external debugging tools to debug threading
> issues. Python no longer has to come with its built-in logs.
>
> I propose to deprecate the feature in Python 3.11 and remove it in 2
> releases (Python 3.13).
>

I agree with its removal.


> Victor
> --
> Night gathers, and now my watch begins. It shall not end until my death.
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/NMLGCDRUKLZSTK4UICJTKR54WRXU2ZGJ/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/F3MFU5YD6H6NIPW5ULIRNGAMSPSWXYPN/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Proposal: declare "unstable APIs"

2021-06-03 Thread Gregory P. Smith
Overall agreement. Your list of ast and code objects and bytecode
instructions are things that I'd _hope_ people already consider unstable so
declaring them as such just makes sense if we're not doing that already.

The ideal way to declare an API as unstable is to constantly change it in a
breaking manner.  With every release and potentially even within some patch
releases when the point really needs to be made.  Even when you didn't have
a reason to change anything.  If you don't do that, people are going to
find it convenient, discover stability, assume it exists, depend on it, and
complain about breakage no matter what was stated.
https://www.hyrumslaw.com/

One obvious now in hindsight question: Why are any of these APIs even
public? They all deserve underscore prefixed names to highlight their
private-ness and potential instability.

-gps

On Thu, Jun 3, 2021 at 10:46 AM Guido van Rossum  wrote:

> In practice, provisional APIs have been quite stable. The term
> "provisional" was introduced for PEPs that introduce new modules, where we
> wanted to allow some wiggle room for changes based on experience with using
> the new module during the first release cycle where it's made available.
> You can think of it as a sort of extended beta period for that module only.
> Generally provisional status only lasts for one release cycle.
>
> "Unstable" has a different meaning -- it's for APIs (including modules)
> that are likely to change in every release (or most releases, anyway).
> Users are not discouraged from using these, but they *must* be mindful of
> their code breaking with every new release.
>
> I could imagine some unstability to allow incompatible changes in bugfix
> releases, though for my main use case it would be sufficient to only allow
> those in minor releases.
>
> On Thu, Jun 3, 2021 at 10:32 AM Senthil Kumaran 
> wrote:
>
>> On Thu, Jun 03, 2021 at 10:10:53AM -0700, Guido van Rossum wrote:
>> > This is not a complete thought yet, but it occurred to me that while we
>> have
>> > deprecated APIs (which will eventually go away), and provisional APIs
>> (which
>> > must mature a little before they're declared stable), and stable APIs
>> (which
>> > everyone can rely on), it might be good to also have something like
>> *unstable*
>> > APIs, which will continually change without ever going away or
>> stabilizing.
>>
>> The first grey area will between Provisional API vs Unstable API.
>>
>> Do developers consider provisional APIs as stable and start relying upon
>> heavily? I am not sure.
>>
>> I also lack the experience for the use-cases that you are thinking
>> about.
>>
>> --
>> Senthil
>>
>
>
> --
> --Guido van Rossum (python.org/~guido)
> *Pronouns: he/him **(why is my pronoun here?)*
> 
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/ZUTAVE3S3QMWNIBGTBDOTJ7M62CTO57R/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/NCWBNN5AH4D26ECGL4DYRI2DKZBIKV6C/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: GDB not breaking at the right place

2021-05-25 Thread Gregory P. Smith
On Tue, May 25, 2021 at 7:49 PM Inada Naoki  wrote:

> On Tue, May 25, 2021 at 5:38 AM Guido van Rossum  wrote:
> >
> > To the contrary, I think if you want the CI jobs to be faster you should
> add the CFLAGS to the configure call used to run the CI jobs.
> >
>
> -Og makes it faster not only CI jobs, but also everyday "edit code and
> run `make test` with all assertions" cycles.
>
> I don't have opinion which should be default. (+0 for -O0).
> I use -Og by default and use -O0 only when I need anyway.
>

Agreed, what we do today is already fine.  -Og or -O1 are decent options
for fast unoptimized builds that lead to increased productivity in the
common case.
Actually firing up a debugger on CPython's C code is not the common thing
for a developer to do.
When someone wants to do that, they should build with the relevant compiler
for that purpose.  ie: Skip should do this.  If there is confusion about
the meaning of --with-pydebug, that's just a documentation/help-text update
to be made.

-gps

FWIW, we can disable optimization per-file basis during debugging.
>
>   // Put this line on files you want to debug.
>   #pragma GCC optimize ("O0")
>
> Regards,
>
> --
> Inada Naoki  
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/OJJZKWS446PJPXHUBNUVIYE756D5HHP4/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PMOP7CQHRHB7C5737E5BKRIBXHX5PBHX/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Future PEP: Include Fine Grained Error Locations in Tracebacks

2021-05-11 Thread Gregory P. Smith
On Tue, May 11, 2021 at 3:33 PM Mike Miller  wrote:

>
> On 5/11/21 1:57 AM, Baptiste Carvello wrote:
> > Le 11/05/2021 à 09:35, Steven D'Aprano a écrit :
> >> On Mon, May 10, 2021 at 09:44:05PM -0400, Terry Reedy wrote:
> >>
> >>> The vanilla interpreter could be updated to recognize when it is
> running
> >>> on a similated 35-year-old terminal that implements ansi-vt100 color
> >>> codes rather than a similated 40+-year-old black-and-white
> teletype-like
> >>> terminal.
> >>
> >> This is what is called "scope creep", although in this case
> >> perhaps "scope gallop" is more appropriate *wink*
> >> [...]
> >
> > Also: people paste tracebacks into issue reports, so all information has
> > to survive copy-pasting.
> >
>
> The first ANSI standard supported underlined text, didn't it?  The VT100
> did.
> That would make it part of the 40+ year old subset from the late 70's.
>
> While color might stand out more, underline suits the problem well, also
> without
> increasing the line count.
>
> There are a number of terminal emulators that support rich text copies,
> but not
> all of them.  This is added information however, so it not being
> copy-pastable
> everywhere shouldn't be a blocking requirement imho.
>

fancier REPL frontends have supported things like highlighting and such in
their tracebacks, I expect they'll adopt column information and render it
as such.

There's a difference between tracebacks dumped as plain text (utf-8) by
traceback.print_exc() appearing on stderr or directed into log files and
what can be displayed within a terminal.  It is highly unusual to emit
terminal control characters into log files.

-G


>
> -Mike
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/W44D2BWWICNJTWPQOZUWVQEIJ6T3QWYM/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/7AQFQOCZCN44ML3UDY5RNWJJHOEDS4JN/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Future PEP: Include Fine Grained Error Locations in Tracebacks

2021-05-09 Thread Gregory P. Smith
On Sun, May 9, 2021 at 9:13 AM Antoine Pitrou  wrote:

> On Sun, 09 May 2021 02:16:02 -
> "Jim J. Jewett"  wrote:
> > Antoine Pitrou wrote:
> > > On Sat, 8 May 2021 02:58:40 +
> > > Neil Schemenauer nas-pyt...@arctrix.com wrote:
> >
> > > > It would be cool if we could mmap the pyc files and have the VM run
> > > > code without an unmarshal step.
> > > > What happens if another process mutates or truncates the file while
> the
> > > CPython VM is executing code from the mapped file?  Crash?
> >
> > Why would this be any different than whatever happens now?
>
> What happens now is that the pyc file is transferred at once to memory
> using regular IO.  So the chance is really slim that you read invalid
> data due to concurrent mutation.
>

concurrent mutation isn't even what I was talking about.  We don't protect
against that today as that isn't a concern.  But POSIX semantics on the
bulk of systems where this would ever matter do software updates by moving
new files into place.  Because that is an idempotent inode change.  So the
existing open file already in the process of being read is not changed.
But as soon as you do a new open call on the pathname you get a different
file than the last time that path was opened.

This is not theoretical.  I've seen production problems as a result
(zipimport - https://bugs.python.org/issue19081) making the incorrect
assumption that they can reopen a file that they've read once at a later
point in time.  So if we do open files later, we must code defensively and
assume they might not contain what we thought.

We already have this problem with source code lines displayed in tracebacks
today as those are read on demand.  But as that is debugging information
only the wrong source lines being shown next to the filename +
linenumber in a traceback is something people just learn to ignore in these
situations.  We have the data to prevent this, we just never have.
https://bugs.python.org/issue44091 filed to track that.

Given this context, M.-A. Lemburg's alternative idea could have some merit
as it would synchronize our source skew behavior with our additional
debugging information behavior.  My initial reaction is that it's falling
into the trap of bundling too into one place though.

quoting M.-A. Lemburg:
> Create a new file format which supports enhanced debugging. This
> would include the source code in a indexed format, the AST and
> mappings between byte code, AST node, lines and columns.
>
> Python would then only use and load this file when it needs
> to print a traceback - much like it does today with the source
> code.
>
> The advantage is that you can add even more useful information
> for debugging while not making the default code distribution
> format take more memory (both disk and RAM).

Realistically: This is going to take more disk space in the common case
because in addition to the py, pyc, pyc.opt-1, pyc.opt-2 that some distros
apparently include all of today, there'd be a new pyc.debuginfo to go along
side it. The only benefit is that it isn't resident in ram. And someone
*could* choose to filter these out of their distro or container or
whatever-the-heck-their-package-format-is. But I really doubt that'll be
the default.

Not having debugging information when a problem you're trying to hunt down
and reproduce but only happens once in a blue moon is extraordinarily
frustrating.  Which is why people who value engineering time deploy with
debugging info.

There are environments where people intentionally do not deploy source
code.  But do want to get debugging data from tracebacks that they can then
correlate to their sources later for analysis (they're tracking exactly
which versions of pycs from which versions of sources were deployed).  It'd
be a shame to exclude column information for this scenario.

-gps
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6E7UZ5SFUAADUJUQ6DKPJIGO6CCGCNFU/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Future PEP: Include Fine Grained Error Locations in Tracebacks

2021-05-08 Thread Gregory P. Smith
On Sat, May 8, 2021 at 2:40 PM Jonathan Goble  wrote:

> On Sat, May 8, 2021 at 5:08 PM Pablo Galindo Salgado 
> wrote:
>
>> > Why not put in it -O instead?  Then -O means lose asserts and lose
>> fine-grained tracebacks, while -OO continues to also
>> strip out doc strings.
>>
>> What if someone wants to keep asserts but do not want the extra data?
>>
>
> What if I want to keep asserts and docstrings but don't want the extra
> data?
>
> Or actually, consider this. I *need* to keep asserts (because rightly or
> wrongly, I have a dependency, or my own code, that relies on them), but I
> *don't* want docstrings (because they're huge and I don't want the overhead
> in production), and I *don't* want the extra data in production either.
>
> Now what?
>
> I think what this illustrates is that the entire concept of optimizations
> in Python needs a complete rethink. It's already fundamentally broken for
> someone who wants to keep asserts but remove docstrings. Adding a third
> layer to this is a perfect opportunity to reconsider the whole paradigm.
>

Reconsidering "the whole paradigm" is always possible, but is a much larger
effort. It should not be something that blocks this enhancement from
happening.

We have discussed the -O mess before, on list and at summits and sprints.
-OO and the __pycache__ and longer .pyc names and versioned names were
among the results of that.  But we opted not to try and make life even more
complicated by expanding the test matrix of possible generated bytecode
even larger.

I'm getting off-topic here, and this should probably be a thread of its
> own, but perhaps what we should introduce is a compiler directive, similar
> to future statements but not that, that one can place at the top of a
> source file to tell the compiler "this file depends on asserts, don't
> optimize them out". Same for each thing that can be optimized that has a
> runtime behavior effect, including docstrings.
>

This idea has merit.  Worth keeping in mind for the future.  But agreed,
this goes beyond this threads topic so I'll leave it at that.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PCZGEWFIPS2YPMJWTILVANJYT6VWS27B/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Future PEP: Include Fine Grained Error Locations in Tracebacks

2021-05-08 Thread Gregory P. Smith
On Sat, May 8, 2021 at 2:09 PM Pablo Galindo Salgado 
wrote:

> > Why not put in it -O instead?  Then -O means lose asserts and lose
> fine-grained tracebacks, while -OO continues to also
> strip out doc strings.
>
> What if someone wants to keep asserts but do not want the extra data?
>

exactly my theme.  our existing -O and -OO already don't serve all user
needs.  (I've witnessed people who need asserts but don't want docstrings
wasting ram jump through hacky hoops to do that).  Complicating these
options more by combining additional actions on them them doesn't help.

The reason we have -O and -OO generate their own special opt-1 and opt-2
pyc files is because both of those change the generated bytecode and
overall flow of the program by omitting instructions and data.  code using
those will run slightly faster as there are fewer instructions.

The change we're talking about here doesn't do that.  It just adds
additional metadata to whatever instructions are generated.  So it doesn't
feel -O related.

While some people aren't going to like the overhead, I'm happy not offering
the choice.

> Greg, what do you think if instead of not writing it to the pyc file with
-OO or adding a header entry to decide to read/write, we place None in the
field? That way we can
> leverage the option that we intend to add to deactivate displaying the
traceback new information to reduce the data in the pyc files. The only
problem
> is that there will be still a tiny bit of overhead: an extra object per
code object (None), but that's much much better than something that scales
with the
> number of instructions :)
>
> What's your opinion on this?

I don't understand the pyc structure enough to comment on how that works,
but that sounds fine from a way to store less data if these are stored as a
side table rather than intermingled with each instruction itself.  *If
anyone even cares about storing less data.*  I would not activate
generation of that in py_compile and compileall based on the -X flag to
disable display of tracebacks though.  A flag changing a setting of the
current runtime regarding traceback printing detail level should not change
the metadata in pyc files it emits.  I realize -O and -OO behave this way,
but I don't view those as a great example. We're not writing new uniquely
named pyc files, I suggest making this an explicit option for py_compile
and compileall if we're going to support generation of pyc files without
column data at all.

I'm unclear on what the specific goals are with all of these option
possibilities.

Who non-hypothetically cares about a 22% pyc file size increase?  I don't
think we should be concerned.  I'm in favor of always writing them and the
20% size increase that results in.  If pyc size is an issue that should be
its own separate enhancement PEP.  When it comes to pyc files there is more
data we may want to store in the future for performance reasons - I don't
see them shrinking without an independent effort.

Caring about additional data retained in memory at runtime makes more sense
to me as ram cost is much greater than storage cost and is paid repeatedly
per process.  Storing an additional reference to None on code objects where
a column information table is perfectly fine.  That can be a -X style
interpreter startup option.  It isn't something that needs to impacted by
the pyc files.  Pass that option to the interpreter, and it just discards
column info tables on code objects after loading them or compiling them.
If people want to optimize for a shared pyc situation with memory mapping
techniques, that is also something that should be a separate enhancement
PEP and not involved here.  People writing code to use the column
information should always check it for None first, that'd be something we
document with the new feature.

-gps


>
> On Sat, 8 May 2021 at 22:05, Ethan Furman  wrote:
>
>> On 5/8/21 1:31 PM, Pablo Galindo Salgado wrote:
>>  >> We can't piggy back on -OO as the only way to disable this, it needs
>> to
>>  >> have an option of its own.  -OO is unusable as code that relies on
>> "doc"
>>  >> strings as application data such as
>> http://www.dabeaz.com/ply/ply.html
>>  >> exists.
>>  >
>>  > -OO is the only sensible way to disable the data. There are two things
>> to disable:
>>  >
>>  > * The data in pyc files
>>  > * Printing the exception highlighting
>>
>> Why not put in it -O instead?  Then -O means lose asserts and lose
>> fine-grained tracebacks, while -OO continues to also
>> strip out doc strings.
>>
>> --
>> ~Ethan~
>> ___
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-le...@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-dev@python.org/message/BEE4BGUZHXBTVDPOW5R4DC3S463XC3EJ/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
> 

[Python-Dev] Re: Future PEP: Include Fine Grained Error Locations in Tracebacks

2021-05-08 Thread Gregory P. Smith
On Sat, May 8, 2021 at 1:32 PM Pablo Galindo Salgado 
wrote:

> > We can't piggy back on -OO as the only way to disable this, it needs to
> have an option of its own.  -OO is unusable as code that relies on
> "doc"strings as application data such as
> http://www.dabeaz.com/ply/ply.html exists.
>
> -OO is the only sensible way to disable the data. There are two things to
> disable:
>

nit: I wouldn't choose the word "sensible" given that -OO is already
fundamentally unusable without knowing if any code in your entire
transitive dependencies might depend on the presence of docstrings...


>
> * The data in pyc files
> * Printing the exception highlighting
>
> Printing the exception highlighting can be disabled via combo of
> environment variable / -X option but collecting the data can only be
> disabled by -OO. The reason is that this will end in pyc files
> so when the data is not there, a different kind of pyc files need to be
> produced and I really don't want to have another set of pyc file extension
> just to deactivate this. Notice that also a configure
> time variable won't work because it will cause crashes when reading pyc
> files produced by the interpreter compiled without the flag.
>

I don't think the optional existence of column number information needs a
different kind of pyc file.  Just a flag in a pyc file's header at most.
It isn't a new type of file.


> On Sat, 8 May 2021 at 21:13, Gregory P. Smith  wrote:
>
>>
>>
>> On Sat, May 8, 2021 at 11:58 AM Pablo Galindo Salgado <
>> pablog...@gmail.com> wrote:
>>
>>> Hi Brett,
>>>
>>> Just to be clear, .pyo files have not existed for a while:
>>>> https://www.python.org/dev/peps/pep-0488/.
>>>
>>>
>>> Whoops, my bad, I wanted to refer to the pyc files that are generated
>>> with -OO, which have the "opt-2" prefix.
>>>
>>> This only kicks in at the -OO level.
>>>
>>>
>>> I will correct the PEP so it reflex this more exactly.
>>>
>>> I personally prefer the idea of dropping the data with -OO since if
>>>> you're stripping out docstrings you're already hurting introspection
>>>> capabilities in the name of memory. Or one could go as far as to introduce
>>>> -Os to do -OO plus dropping this extra data.
>>>
>>>
>>> This is indeed the plan, sorry for the confusion. The opt-out mechanism
>>> is using -OO, precisely as we are already dropping other data.
>>>
>>
>> We can't piggy back on -OO as the only way to disable this, it needs to
>> have an option of its own.  -OO is unusable as code that relies on
>> "doc"strings as application data such as
>> http://www.dabeaz.com/ply/ply.html exists.
>>
>> -gps
>>
>>
>>>
>>> Thanks for the clarifications!
>>>
>>>
>>>
>>> On Sat, 8 May 2021 at 19:41, Brett Cannon  wrote:
>>>
>>>>
>>>>
>>>> On Fri, May 7, 2021 at 7:31 PM Pablo Galindo Salgado <
>>>> pablog...@gmail.com> wrote:
>>>>
>>>>> Although we were originally not sympathetic with it, we may need to
>>>>> offer an opt-out mechanism for those users that care about the impact of
>>>>> the overhead of the new data in pyc files
>>>>> and in in-memory code objectsas was suggested by some folks (Thomas,
>>>>> Yury, and others). For this, we could propose that the functionality will
>>>>> be deactivated along with the extra
>>>>> information when Python is executed in optimized mode (``python -O``)
>>>>> and therefore pyo files will not have the overhead associated with the
>>>>> extra required data.
>>>>>
>>>>
>>>> Just to be clear, .pyo files have not existed for a while:
>>>> https://www.python.org/dev/peps/pep-0488/.
>>>>
>>>>
>>>>> Notice that Python
>>>>> already strips docstrings in this mode so it would be "aligned" with
>>>>> the current mechanism of optimized mode.
>>>>>
>>>>
>>>> This only kicks in at the -OO level.
>>>>
>>>>
>>>>>
>>>>> Although this complicates the implementation, it certainly is still
>>>>> much easier than dealing with compression (and more useful for those that
>>>>> don't want the feature). Notice that we also
>>>>> expect pessimistic results from compression as offsets would be quite
>

[Python-Dev] Re: Future PEP: Include Fine Grained Error Locations in Tracebacks

2021-05-08 Thread Gregory P. Smith
On Sat, May 8, 2021 at 11:58 AM Pablo Galindo Salgado 
wrote:

> Hi Brett,
>
> Just to be clear, .pyo files have not existed for a while:
>> https://www.python.org/dev/peps/pep-0488/.
>
>
> Whoops, my bad, I wanted to refer to the pyc files that are generated
> with -OO, which have the "opt-2" prefix.
>
> This only kicks in at the -OO level.
>
>
> I will correct the PEP so it reflex this more exactly.
>
> I personally prefer the idea of dropping the data with -OO since if you're
>> stripping out docstrings you're already hurting introspection capabilities
>> in the name of memory. Or one could go as far as to introduce -Os to do -OO
>> plus dropping this extra data.
>
>
> This is indeed the plan, sorry for the confusion. The opt-out mechanism is
> using -OO, precisely as we are already dropping other data.
>

We can't piggy back on -OO as the only way to disable this, it needs to
have an option of its own.  -OO is unusable as code that relies on
"doc"strings as application data such as http://www.dabeaz.com/ply/ply.html
exists.

-gps


>
> Thanks for the clarifications!
>
>
>
> On Sat, 8 May 2021 at 19:41, Brett Cannon  wrote:
>
>>
>>
>> On Fri, May 7, 2021 at 7:31 PM Pablo Galindo Salgado 
>> wrote:
>>
>>> Although we were originally not sympathetic with it, we may need to
>>> offer an opt-out mechanism for those users that care about the impact of
>>> the overhead of the new data in pyc files
>>> and in in-memory code objectsas was suggested by some folks (Thomas,
>>> Yury, and others). For this, we could propose that the functionality will
>>> be deactivated along with the extra
>>> information when Python is executed in optimized mode (``python -O``)
>>> and therefore pyo files will not have the overhead associated with the
>>> extra required data.
>>>
>>
>> Just to be clear, .pyo files have not existed for a while:
>> https://www.python.org/dev/peps/pep-0488/.
>>
>>
>>> Notice that Python
>>> already strips docstrings in this mode so it would be "aligned" with
>>> the current mechanism of optimized mode.
>>>
>>
>> This only kicks in at the -OO level.
>>
>>
>>>
>>> Although this complicates the implementation, it certainly is still much
>>> easier than dealing with compression (and more useful for those that don't
>>> want the feature). Notice that we also
>>> expect pessimistic results from compression as offsets would be quite
>>> random (although predominantly in the range 10 - 120).
>>>
>>
>> I personally prefer the idea of dropping the data with -OO since if
>> you're stripping out docstrings you're already hurting introspection
>> capabilities in the name of memory. Or one could go as far as to introduce
>> -Os to do -OO plus dropping this extra data.
>>
>> As for .pyc file size, I personally wouldn't worry about it. If someone
>> is that space-constrained they either aren't using .pyc files or are only
>> shipping a single set of .pyc files under -OO and skipping source code. And
>> .pyc files are an implementation detail of CPython so there  shouldn't be
>> too much of a concern for other interpreters.
>>
>> -Brett
>>
>>
>>>
>>> On Sat, 8 May 2021 at 01:56, Pablo Galindo Salgado 
>>> wrote:
>>>
 One last note for clarity: that's the increase of size in the stdlib,
 the increase of size
 for pyc files goes from 28.471296MB to 34.750464MB, which is an
 increase of 22%.

 On Sat, 8 May 2021 at 01:43, Pablo Galindo Salgado 
 wrote:

> Some update on the numbers. We have made some draft implementation to
> corroborate the
> numbers with some more realistic tests and seems that our original
> calculations were wrong.
> The actual increase in size is quite bigger than previously advertised:
>
> Using bytes object to encode the final object and marshalling that to
> disk (so using uint8_t) as the underlying
> type:
>
> BEFORE:
>
> ❯ ./python -m compileall -r 1000 Lib > /dev/null
> ❯ du -h Lib -c --max-depth=0
> 70M Lib
> 70M total
>
> AFTER:
> ❯ ./python -m compileall -r 1000 Lib > /dev/null
> ❯ du -h Lib -c --max-depth=0
> 76M Lib
> 76M total
>
> So that's an increase of 8.56 % over the original value. This is
> storing the start offset and end offset with no compression
> whatsoever.
>
> On Fri, 7 May 2021 at 22:45, Pablo Galindo Salgado <
> pablog...@gmail.com> wrote:
>
>> Hi there,
>>
>> We are preparing a PEP and we would like to start some early
>> discussion about one of the main aspects of the PEP.
>>
>> The work we are preparing is to allow the interpreter to produce more
>> fine-grained error messages, pointing to
>> the source associated to the instructions that are failing. For
>> example:
>>
>> Traceback (most recent call last):
>>
>>   File "test.py", line 14, in 
>>
>> lel3(x)
>>
>> ^^^
>>
>>   File "test.py", line 12, in lel3
>>

[Python-Dev] Re: a name for the ExceptHandler.type when it is a literal tuple of types

2021-05-08 Thread Gregory P. Smith
On Sat, May 8, 2021 at 8:54 AM Thomas Grainger  wrote:

> That's this bit:
>
> ```
> except (A, B):
>^^
> ```
>
> bpo-43149 currently calls it an "exception group", but that conflicts with
> PEP 654 -- Exception Groups and except*
>
> ```
>
>>>> try:
>...   pass
>... except A, B:
>...   pass
>Traceback (most recent call last):
>SyntaxError: exception group must be parenthesized
> ```
>
> some alternatives:
>
> exception classinfo must be parenthesized (classinfo so named from the
> parameter to issubclass)
> exception sequence must be parenthesized
>
> see also:
>
> - https://github.com/python/cpython/pull/24467#discussion_r628756347
> - https://www.python.org/dev/peps/pep-0654/


Given it requires ()s it is probably better to call it an "exception
sequence" or even go fully to "exception tuple" in order to avoid confusion
and tie in with the other meanings of the required syntax.

-gps
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/Y7SPN4WSXXFPAZITS2PMF2PRSVX3H5SE/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Speeding up CPython

2021-05-07 Thread Gregory P. Smith
On Fri, May 7, 2021 at 6:51 PM Steven D'Aprano  wrote:

> On Tue, Oct 20, 2020 at 01:53:34PM +0100, Mark Shannon wrote:
> > Hi everyone,
> >
> > CPython is slow. We all know that, yet little is done to fix it.
> >
> > I'd like to change that.
> > I have a plan to speed up CPython by a factor of five over the next few
> > years. But it needs funding.
>
> I've noticed a lot of optimization-related b.p.o. issues created by
> Mark, which is great. What happened with Mark's proposal here? Did the
> funding issue get sorted?
>

I believe Guido has Mark contracting on Python performance through
Microsoft?

-Greg
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5RTMMJLGZE5FHW3SAYWSKUYOLEUZ2RFX/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Future PEP: Include Fine Grained Error Locations in Tracebacks

2021-05-07 Thread Gregory P. Smith
On Fri, May 7, 2021 at 3:24 PM Pablo Galindo Salgado 
wrote:

> Thanks a lot Gregory for the comments!
>
> An additional cost to this is things that parse text tracebacks not
>> knowing how to handle it and things that log tracebacks
>> generating additional output.
>
> We should provide a way for people to disable the feature on a process as
>> part of this while they address tooling and logging issues.  (via the usual
>> set of command line flag + python env var + runtime API)
>
>
> Absolutely! We were thinking about that and that's easy enough as that is
> a single conditional on the display function + the extra init configuration.
>
> Neither of those is large. While I'd lean towards uint8_t instead of
>> uint16_t because not even humans can understand a 255 character line so why
>> bother being pretty about such a thing... Just document the caveat and move
>> on with the lower value. A future pyc format could change it if a
>> compelling argument were ever found.
>
>
> I very much agree with you here but is worth noting that I have heard the
> counter-argument that the longer the line is, the more important may be to
> distinguish what part of the line is wrong.
>

haha, true... Does our parser even have a maximum line length? (I'm not
suggesting being unlimited or matching that if huge, 64k is already
ridiculous)


>
> A compromise if you want to handle longer lines: A single uint16_t.
>> Represent the start column in the 9 bits and width in the other 7 bits. (or
>> any variations thereof)  it's all a matter of what tradeoff you want to
>> make for space reasons.  encoding as start + width instead of start + end
>> is likely better anyways if you care about compression as the width byte
>> will usually be small and thus be friendlier to compression.  I'd
>> personally ignore compression entirely.
>
>
> I would personally prefer not to implement very tricky compression
> algorithms because tools may need to parse this and I don't want to
> complicate the logic a lot. Handling lnotab is already a bit painful and
> when bugs ocur it makes debugging very tricky. Having the possibility to
> index something based on the index of the instruction is quite a good API
> in my opinion.
>
> Overall doing this is going to be a big win for developer productivity!
>
>
> Thanks! We think that this has a lot of potential indeed! :)
>
> Pablo
>
>
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/E7OM3GA4GNMRXAXOFAIZCCNTBWFUJAEP/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Future PEP: Include Fine Grained Error Locations in Tracebacks

2021-05-07 Thread Gregory P. Smith
On Fri, May 7, 2021 at 2:50 PM Pablo Galindo Salgado 
wrote:

> Hi there,
>
> We are preparing a PEP and we would like to start some early discussion
> about one of the main aspects of the PEP.
>
> The work we are preparing is to allow the interpreter to produce more
> fine-grained error messages, pointing to
> the source associated to the instructions that are failing. For example:
>
> Traceback (most recent call last):
>
>   File "test.py", line 14, in 
>
> lel3(x)
>
> ^^^
>
>   File "test.py", line 12, in lel3
>
> return lel2(x) / 23
>
>^^^
>
>   File "test.py", line 9, in lel2
>
> return 25 + lel(x) + lel(x)
>
> ^^
>
>   File "test.py", line 6, in lel
>
> return 1 + foo(a,b,c=x['z']['x']['y']['z']['y'], d=e)
>
>  ^
>
> TypeError: 'NoneType' object is not subscriptable
>
>
An additional cost to this is things that parse text tracebacks not knowing
how to handle it and things that log tracebacks generating additional
output.  We should provide a way for people to disable the feature on a
process as part of this while they address tooling and logging issues.
(via the usual set of command line flag + python env var + runtime API)

The cost of this is having the start column number and end column number
> information for every bytecode instruction
> and this is what we want to discuss (there is also some stack cost to
> re-raise exceptions but that's not a big problem in
> any case). Given that column numbers are not very big compared with line
> numbers, we plan to store these as unsigned chars
> or unsigned shorts. We ran some experiments over the standard library and
> we found that the overhead of all pyc files is:
>
> * If we use shorts, the total overhead is ~3% (total size 28MB and the
> extra size is 0.88 MB).
> * If we use chars. the total overhead is ~1.5% (total size 28 MB and the
> extra size is 0.44MB).
>
> One of the disadvantages of using chars is that we can only report columns
> from 1 to 255 so if an error happens in a column
> bigger than that then we would have to exclude it (and not show the
> highlighting) for that frame. Unsigned short will allow
> the values to go from 0 to 65535.
>

Neither of those is large. While I'd lean towards uint8_t instead of
uint16_t because not even humans can understand a 255 character line so why
bother being pretty about such a thing... Just document the caveat and move
on with the lower value. A future pyc format could change it if a
compelling argument were ever found.


> Unfortunately these numbers are not easily compressible, as every
> instruction would have very different offsets.
>
> There is also the possibility of not doing this based on some build flag
> on when using -O to allow users to opt out, but given the fact
> that these numbers can be quite useful to other tools like coverage
> measuring tools, tracers, profilers and the such adding conditional
> logic to many places would complicate the implementation considerably and
> will potentially reduce the usability of those tools so we prefer
> not to have the conditional logic. We believe this is extra cost is very
> much worth the better error reporting but we understand and respect
> other points of view.
>
> Does anyone see a better way to encode this information **without
> complicating a lot the implementation**? What are people thoughts on the
> feature?
>

A compromise if you want to handle longer lines: A single uint16_t.
Represent the start column in the 9 bits and width in the other 7 bits. (or
any variations thereof)  it's all a matter of what tradeoff you want to
make for space reasons.  encoding as start + width instead of start + end
is likely better anyways if you care about compression as the width byte
will usually be small and thus be friendlier to compression.  I'd
personally ignore compression entirely.

Overall doing this is going to be a big win for developer productivity!

-Greg
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ULNDFY5CWVDELNPE6S4HY5SDAODOT7DC/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Future PEP: Include Fine Grained Error Locations in Tracebacks

2021-05-07 Thread Gregory P. Smith
On Fri, May 7, 2021 at 3:01 PM Larry Hastings  wrote:

> On 5/7/21 2:45 PM, Pablo Galindo Salgado wrote:
>
> Given that column numbers are not very big compared with line numbers, we
> plan to store these as unsigned chars
> or unsigned shorts. We ran some experiments over the standard library and
> we found that the overhead of all pyc files is:
>
> * If we use shorts, the total overhead is ~3% (total size 28MB and the
> extra size is 0.88 MB).
> * If we use chars. the total overhead is ~1.5% (total size 28 MB and the
> extra size is 0.44MB).
>
> One of the disadvantages of using chars is that we can only report columns
> from 1 to 255 so if an error happens in a column
> bigger than that then we would have to exclude it (and not show the
> highlighting) for that frame. Unsigned short will allow
> the values to go from 0 to 65535.
>
> Are lnotab entries required to be a fixed size?  If not:
>
> if column < 255:
> lnotab.write_one_byte(column)
> else:
> lnotab.write_one_byte(255)
> lnotab.write_two_bytes(column)
>
> If non-fixed size is acceptable. use utf-8 to encode the column number as
a single codepoint number into bytes and you don't even need to write your
own encode/decode logic for a varint.

-gps
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QNWOZWTNFAVPD77KNG4LRYWCEDY3F6HX/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Keeping Python a Duck Typed Language.

2021-04-23 Thread Gregory P. Smith
When reading this, I wrote most of it early and left a draft to bake
Then deleted a ton of it after other people replied. I'm conscious that my
terminology might be all over the map.  Keep that in mind before hitting
reply.  It'll take me a while to digest and pedantically use Luciano's
terms, they appear to be a great start. :)

On Tue, Apr 20, 2021 at 10:09 AM Mark Shannon  wrote:

> Hi everyone,
>
> Once upon a time Python was a purely duck typed language.
>
> Then came along abstract based classes, and some nominal typing starting
> to creep into the language.
>
> If you guarded your code with `isinstance(foo, Sequence)` then I could
> not use it with my `Foo` even if my `Foo` quacked like a sequence. I was
> forced to use nominal typing; inheriting from Sequence, or explicitly
> registering as a Sequence.
>

True.  Though in practice I haven't run into this often *myself*.  Do you
have practical examples of where this has bitten users such that code they
would've written pre-abc is no longer possible?  This audience can come up
with plenty of theoretical examples, those aren't so interesting to me.
I'm more interested in observances of actual real world fallout due to
something "important" (as defined however each user wants) using isinstance
checks when it ideally wouldn't.

Practically speaking, one issue I have is how easy it is to write
isinstance or issubclass checks. It has historically been much more
difficult to write and maintain a check that something looks like a duck.

 `if hasattr(foo, 'close') and hasattr(foo, 'seek') and hasattr(foo,
'read'):`

Just does not roll off the figurative tongue and that is a relatively
simple example of what is required for a duck check.

To prevent isinstance use when a duck check would be better, we're missing
an easy builtin elevated to the isinstance() availability level behaving
as lookslikeaduck() that does matches against a (set of) declared
typing.Protocol shape(s). An implementation of this exists -
https://www.python.org/dev/peps/pep-0544/#runtime-checkable-decorator-and-narrowing-types-by-isinstance
- but it requires the protocols to declare runtime checkability and has
them work with isinstance similar to ABCs...  technically accurate *BUT via
isinstance*? Doh!  It promotes the use of isinstance when it really isn't
about class hierarchy at all...

Edit: Maybe that's okay, isinstance can be read leniently to mean "is an
instance of something that one of these things over here says it matches"
rather than meaning "a parent class type is..."?  From a past experience
user perspective I don't read "isinstance" as "looks like a duck" when I
read code.  I assume I'm not alone.

I'd prefer something not involving metaclasses and __instancecheck__ type
class methods.  Something direct so that the author and reader both
explicitly see that they're seeing a duck check rather than a type
hierarchy check.  I don't think this ship has sailed, it could be built up
on top of what already exists if we want it.  Was this already covered in
earlier 544 discussions perhaps?

As Nathaniel indicated, how deep do we want to go down this rabbit hole of
checking?  just names?  signatures and types on those?  What about
exceptions (something our type system has no way to declare at all)?  and
infinite side effects?  At the end of the day we're required to trust the
result of whatever check we use and any implementation may not conform to
our desires no matter how much checking we do. Unless we solve the halting
problem. :P

PEP 544 supports structural typing, but to declare a structural type you
> must inherit from Protocol.
> That smells a lot like nominal typing to me.
>

Not quite.  A Protocol is merely a way to describe a structural type.  You
do not *need* to have your *implementations* of anything inherit from
typing.Protocol.  I'd *personally* advise people *do not inherit* from
Protocol in their implementation. Leave that for a structural type
declaration for type description and annotation purposes only, even though
Protocol appears to support direct inheritance. I understand why some don't
like this separate shape declaration concept.

Luciano notes that it is preferred to define your protocols as narrow and
define them in places *where they're used*, to follow a golang interface
practice.  My thinking aligns with that.

That inheritance is used in the *declaration* of the protocol is an
implementation detail because our language has never had a syntax for
declaring an interface.  544 fit within our existing language syntax.

Then came PEP 563 and said that if you wanted to access the annotations
> of an object, you needed to call typing.get_type_hints() to get
> annotations in a meaningful form.
> This smells a bit like enforced static typing to me.
>

I think useful conversations are ongoing here.  Enforced is the wrong
word.  *[rest of comment deleted in light of Larry's work in progress
response]*

Nominal typing in a dynamically typed language makes 

[Python-Dev] Re: NOTE: Python 3.9.3 contains an unintentional ABI incompatibility leading to crashes on 32-bit systems

2021-04-03 Thread Gregory P. Smith
On Sat, Apr 3, 2021 at 7:49 PM Terry Reedy  wrote:

> On 4/3/2021 7:15 PM, Miro Hrončok wrote:
> > On 03. 04. 21 21:44, Łukasz Langa wrote:
> >> The memory layout of PyThreadState was unintentionally changed in the
> >> recent 3.9.3 bugfix release. This leads to crashes on 32-bit systems
> >> when importing binary extensions compiled for Python 3.9.0 - 3.9.2.
> >> This is a regression.
> >>
> >> We will be releasing a hotfix 3.9.4 around 24 hours from now to
> >> address this issue and restore ABI compatibility with C extensions
> >> built for Python 3.9.0 - 3.9.2.
> >
> > Thanks for the hotifx.
> >
> > However, I need to ask: Would this also happen if there was a rc version
> > of 3.9.3?
>
> Unless the mistake was just introduced, the mistake would have happened.
>   One this severe would likely have been caught within the week or two
> before a final.  But as Łukasz noted when announcing the change, .rcs
> are generally ignored.  (I suspect that most everyone assumes that
> someone else will test them.  And begging people to not do that does not
> work well enough to justify the release.) 3.8.5 (2020 July 20 was hotfix
> for 3.8.4 (2020 July 14, which did have a candidate, which did not get
> tested the way that 3.8.4 itself was.
>
> --
> Terry Jan Reedy
>

For 3.9.4 I suggest a strict revert of the offending change. I created such
a PR and attached it to the bpo-43710 issue. It is a holiday weekend for a
large swath of the world. The recursion based crasher issue the original
change was fixing can be saved for a future release and not made under time
pressure.

I filed https://bugs.python.org/issue43725 to track one suggested way to
help automate prevention of these from landing in a release branch and
slipping through the cracks in a release. (discuss that on the issue, not
here)

-Greg
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LOZUUVC22ZCTSQDJEVCYQACLHZ43FF73/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: New public C API functions must not steal references or return borrowed references

2021-03-26 Thread Gregory P. Smith
On Thu, Mar 25, 2021 at 11:58 AM Mark Shannon  wrote:

> Hi Victor,
>
> I'm with you 100% on not returning borrowed references, doing so is just
> plain dangerous.
>
> However, is a blanket ban on stealing references the right thing?
>
> Maybe the problem is the term "stealing".
> The caller is transferring the reference to the callee.
> In some circumstances it can make a lot of sense to do so, since the
> caller has probably finished with the reference and the callee needs a
> new one.
>
> Cheers,
> Mark.
>

When was the last time a non-internal API that transferred references added?

I suggest keeping the restriction on new APIs in place until we actually
find a situation where we think we "need" one outside of Include/internal/
to help force the discussion as to why that needs to be public.

-gps

On 25/03/2021 4:27 pm, Victor Stinner wrote:
> > Hi,
> >
> > A new Include/README.rst file was just added to document the 3 C API
> > provided by CPython:
> >
> > * Include/: Limited C API
> > * Include/cpython/: CPython implementation details
> > * Include/internal/: The internal API
> >
> > I would like to note that *new* public C API functions must no longer
> > steal references or return borrowed references.
> >
> > Don't worry, there is no plan to deprecate or remove existing
> > functions which do that, like PyModule_AddObject() (streal a
> > reference) or PyDict_GetItem() (return a borrowed reference). The
> > policy is only to *add* new functions.
> >
> > IMO for the *internal* C API, it's fine to continue doing that for
> > best performances.
> >
> > Moreover, the limited C API must not expose "implementation details".
> > For example, structure members must not be accessed directly, because
> > most structures are excluded from the limited C API. A function call
> > hiding implementation details is usually better.
> >
> > Here is a copy of the current Include/README.rst file:
> >
> > The Python C API
> > 
> >
> > The C API is divided into three sections:
> >
> > 1. ``Include/``
> > 2. ``Include/cpython/``
> > 3. ``Include/internal/``
> >
> >
> > Include: Limited API
> > 
> >
> > ``Include/``, excluding the ``cpython`` and ``internal`` subdirectories,
> > contains the public Limited API (Application Programming Interface).
> > The Limited API is a subset of the C API, designed to guarantee ABI
> > stability across Python 3 versions, and is defined in :pep:`384`.
> >
> > Guidelines for expanding the Limited API:
> >
> > - Functions *must not* steal references
> > - Functions *must not* return borrowed references
> > - Functions returning references *must* return a strong reference
> > - Macros should not expose implementation details
> > - Please start a public discussion before expanding the API
> > - Functions or macros with a ``_Py`` prefix do not belong in
> ``Include/``.
> >
> > It is possible to add a function or macro to the Limited API from a
> > given Python version.  For example, to add a function to the Limited API
> > from Python 3.10 and onwards, wrap it with
> > ``#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= 0x030A``.
> >
> >
> > Include/cpython: CPython implementation details
> > ===
> >
> > ``Include/cpython/`` contains the public API that is excluded from the
> > Limited API and the Stable ABI.
> >
> > Guidelines for expanding the public API:
> >
> > - Functions *must not* steal references
> > - Functions *must not* return borrowed references
> > - Functions returning references *must* return a strong reference
> >
> >
> > Include/internal: The internal API
> > ==
> >
> >
> > With PyAPI_FUNC or PyAPI_DATA
> > -
> >
> > Functions or structures in ``Include/internal/`` defined with
> > ``PyAPI_FUNC`` or ``PyAPI_DATA`` are internal functions which are
> > exposed only for specific use cases like debuggers and profilers.
> >
> >
> > With the extern keyword
> > ---
> >
> > Functions in ``Include/internal/`` defined with the ``extern`` keyword
> > *must not and can not* be used outside the CPython code base.  Only
> > built-in stdlib extensions (built with the ``Py_BUILD_CORE_BUILTIN``
> > macro defined) can use such functions.
> >
> > When in doubt, new internal C functions should be defined in
> > ``Include/internal`` using the ``extern`` keyword.
> >
> > Victor
> >
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/5UURMDSQUGSNZEUDUSQNHWRZIUKDIZJH/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to 

[Python-Dev] Re: Move support of legacy platforms/architectures outside Python

2021-02-21 Thread Gregory P. Smith
On Sun, Feb 21, 2021 at 12:03 PM Michał Górny  wrote:

> On Sun, 2021-02-21 at 13:35 +0100, Christian Heimes wrote:
> > On 21/02/2021 13.13, Victor Stinner wrote:
> > > Hi,
> > >
> > > I propose to actively remove support for *legacy* platforms and
> > > architectures which are not supported by Python according to PEP 11
> > > rules: hardware no longer sold and end-of-life operating systems. The
> > > removal should be discussed on a case by case basis, but I would like
> > > to get an agreement on the overall idea first. Hobbyists wanting to
> > > support these platforms/archs can continue to support them with
> > > patches maintained outside Python. For example, I consider that the
> > > 16-bit m68k architecture is legacy, whereas the OpenBSD platform is
> > > still actively maintained.
> > [...]
> > > Python has different kinds of platform and architecture supports. In
> > > practice, I would say that we have:
> > >
> > > * (1) Fully supported. Platform/architecture used by core developers
> > > and have at least one working buildbot worker: fully supported. Since
> > > core developers want to use Python on their machine, they fix issues
> > > as soon as they notice them. Examples: x86-64 on Linux, Windows and
> > > macOS.
> > >
> > > * (2) Best effort. Platform/architecture which has a buildbot worker
> > > usually not used by core developers. Regressions (buildbot failures)
> > > are reported to bugs.python.org, if someone investigates and provides
> > > a fix, the fix is merged. But there is usually no "proactive" work to
> > > ensure that Python works "perfectly" on these platforms. Example:
> > > FreeBSD/x86-64.
> > >
> > > * (3) Not (officially) supported. We enter the blurry grey area. There
> > > is no buildbot worker, no core dev use it, but Python contains code
> > > specific to these platforms/architectures. Example: 16-bit m68k and
> > > 31-bit s390 architectures, OpenBSD.
> > >
> > > The Rust programming language has 3 categories of Platform Support,
> > > the last one is :
> >
> > Thanks Victor!
> >
> > (short reply, I'm heading out)
> >
> > I'm +1 in general for your proposal. I also like the idea to adopt
> > Rust's platform support definition.
> >
> > For 3.10 I propose to add a configure option to guard builds on
> > unsupported / unstable platforms. My draft PR
> >
> https://github.com/python/cpython/pull/24610/commits/f8d2d56757a9cec7ae4dc721047336eaba097125
> > implements a checker for unsupported platforms and adds a
> > --enable-unstable-platforms flag. Configuration on unsupported platforms
> > fails unless users explicitly opt-in.
> >
> > The checker serves two purposes:
> >
> > 1) It gives users an opportunity to provide full PEP 11 support
> > (buildbot, engineering time) for a platform.
>
> Does that mean that if someone offers to run the build bot for a minor
> platform and do the necessary maintenance to keep it working, they will
> be able to stay?  How much maintenance is actually expected, i.e. is it
> sufficient to maintain CPython in a 'good enough' working state to
> resolve major bugs blocking real usage on these platforms?
>
>
Definitely start with this.  This level of effort to maintain the minor
platform support in-tree may be able to keep an otherwise neglected minor
platform in second tier non-release-blocker best effort status.  Having a
buildbot, at least provides visibility, and sometimes PR authors will be
sympathetic to easy edits that keep something working that way (no
guarantee).

The main thing from a project maintenance perspective is for platforms to
not become a burden to other code maintainers.  PRs need to be reviewed.
Every #if/#endif in code is a cognitive burden.  So being a minor platform
can come with unexpected breakages that need fixing due to other changes
made in the codebase that did not pay attention to the platform.  As we
cannot expect everyone working on code to care about anything beyond the
tier-1 fully supported platforms, buildbot or not.

Example: I consider many of the BSDs and the Solaris derivatives to be in
this state. (non specific here, i don't even know which ones we claim are
supported or not without going and reading whatever policy docs we might or
might not have today - Victor alludes to this state of the world).  We tend
to accept patches when someone offers them.  Occasionally we have a core
dev who actually runs one of them.  But most of us don't go out of the way
ourselves to try and keep changes we make working there.  We expect
interested parties to jump in when something isn't working right.  And will
generally do related PR reviews/merges if they're not burdensome.

An example of the above happening recently is VxWorks support via
https://bugs.python.org/issue31904.

-gps


> --
> Best regards,
> Michał Górny
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> 

[Python-Dev] Re: Move support of legacy platforms/architectures outside Python

2021-02-21 Thread Gregory P. Smith
On Sun, Feb 21, 2021 at 10:15 AM Christian Heimes 
wrote:

> On 21/02/2021 13.47, glaub...@debian.org wrote:
> > Rust doesn't keep any user from building Rust for Tier 2 or Tier 3
> platforms. There is no separate configure guard. All platforms that Rust
> can build for, are always enabled by default. No one in Rust keeps anyone
> from cross-compiling code for sparc64 or powerpcspe, for example.
> >
> > So if you want to copy Rust's mechanism, you should just leave it as is
> and not claim that users are being confused because "m68k" shows up in
> configure.ac.
>
> A --enable-unstable-platforms configure flag is my peace offer to meet
> you half way. You get a simple way to enable builds on untested
> platforms and we can clearly communicate that some OS and hardware
> platforms are not supported.
>

I personally wouldn't want to maintain such a check in autoconf, but it'll
be an isolated thing on its own, that if you or someone else creates, will
do its job and not bother the rest of us.

I think just publishing our list of (1) supported, (2) best-effort
non-release-blocker quasi-supported, and (3) explicitly unsupported in a
policy doc is sufficient.  But it's not like any of us are going to stop
someone from codifying that in configure.ac to require a flag.

-gps



> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/W3L2RISXFKHRWPYQB232XH7PDIOPKNDY/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TX6UDWZZJX5IMJIQCGUM7C3QDJ6ESACR/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: PEP 647 (type guards) -- final call for comments

2021-02-14 Thread Gregory P. Smith
(there's a small TL;DR towards the end of my reply if you want to read in
reverse to follow my thought process from possible conclusions to how i got
there - please don't reply without reading the whole thing first)

*TL;DR of my TL;DR* - Not conveying bool-ness directly in the return
annotation is my only complaint.  A BoolTypeGuard spelling would alleviate
that.  I'm +0.3 now.  Otherwise I elaborate on other guarding options and
note a few additional Rejected/Postponed/Deferred Ideas sections that the
PEP should mention as currently out of scope: Unconditional guards,
multiple guarded parameters, and type mutators.  Along the way I work my
way towards suggestions for those, but I think they don't belong in _this_
PEP and could serve as input for future ones if/when desired.

On Sun, Feb 14, 2021 at 8:53 AM Paul Bryan  wrote:

> I'm a +1 on using Annotated in this manner. Guido mentioned that it was
> intended for only third-parties though. I'd like to know more about why
> this isn't a good pattern for use by Python libraries.
>
> On Sun, 2021-02-14 at 16:29 +0100, Adrian Freund wrote:
>
> Here's another suggestion:
>
> PEP 593 introduced the `Annotated` type annotation. This could be used to
> annotate a TypeGuard like this:
>
> `def is_str_list(val: List[object]) -> Annotated[bool,
> TypeGuard(List[str])]`
>
>
I like Annotated better than not having it for the sake of not losing the
return type.  BUT I still feel like this limits things too much and
disconnects the information about what parameter(s) are transformed. It
also doesn't solve the problem of why the guard _must_ be tied to the
return value. Clearly sometimes it is desirable to do that. But on many
other scenarios the act of not raising an exception is the narrowing
action: ie - it should be declared as always happening.  Nothing in the
above annotation reads explicitly to me as saying that the return value
determines the type outcome.


>
> Note that I used ( ) instead of [ ] for the TypeGuard, as it is no longer
> a type.
>
> This should fulfill all four requirements, but is a lot more verbose and
> therefore also longer.
> It would also be extensible for other annotations.
>
> For the most extensible approach both `-> TypeGuard(...)` and `->
> Annotated[bool, TypeGuard(...)]` could be allowed, which would open the
> path for future non-type-annotations, which could be used regardless of
> whether the code is type-annotated.
>
>
> --
> Adrian
>
> On February 14, 2021 2:20:14 PM GMT+01:00, Steven D'Aprano <
> st...@pearwood.info> wrote:
>
> On Sat, Feb 13, 2021 at 07:48:10PM -, Eric Traut wrote:
>
> I think it's a reasonable criticism that it's not obvious that a
> function annotated with a return type of `TypeGuard[x]` should return
> a bool.
>
>
> [...]
>
> As Guido said, it's something that a developer can easily
> look up if they are confused about what it means.
>
>
>
> Yes, developers can use Bing and Google :-)
>
> But it's not the fact that people have to look it up. It's the fact that
> they need to know that this return annotation is not what it seems, but
> a special magic value that needs to be looked up.
>
> That's my objection: we're overloading the return annotation to be
> something other than the return annotation, but only for this one
> special value. (So far.) If you don't already know that it is special,
> you won't know that you need to look it up to learn that its special.
>
>
>  I'm open to alternative formulations that meet the following requirements:
>
>  1. It must be possible to express the type guard within the function
>  signature. In other words, the implementation should not need to be
>  present. This is important for compatibility with type stubs and to
>  guarantee consistent behaviors between type checkers.
>
>
>
> When you say "implementation", do you mean the body of the function?
>
> Why is this a hard requirement? Stub files can contain function
> bodies, usually `...` by convention, but alternatives are often useful,
> such as docstrings, `raise NotImplementedError()` etc.
>
> https://mypy.readthedocs.io/en/stable/stubs.html
>
> I don't think that the need to support stub files implies that the type
> guard must be in the function signature. Have I missed something?
>
>
> 2. It must be possible to annotate the input parameter types _and_ the
> resulting (narrowed) type. It's not sufficient to annotate just one or
> the other.
>
>
>
> Naturally :-)
>
> That's the whole point of a type guard, I agree that this is a truly
> hard requirement.
>
>
> 3. It must be possible for a type checker to determine when narrowing
> can be applied and when it cannot. This implies the need for a bool
> response.
>
>
>
> Do you mean a bool return type? Sorry Eric, sometimes the terminology
> you use is not familiar to me and I have to guess what you mean.
>
>
> 4. It should not require changes to the grammar because that would
> prevent this from being adopted in most code bases for many years.
>
>
>
> Fair 

[Python-Dev] Re: PEP 647 (type guards) -- final call for comments

2021-02-12 Thread Gregory P. Smith
My primary reaction seems similar to Mark Shannon's.

When I see this code:

def is_str_list(val: List[object]) -> TypeGuard[List[str]]:
...

I cannot tell what it returns.  There is no readable indication in that
this returns a boolean so the reader cannot immediately see how to use the
function.  In fact my first reaction is to assume it returns some form of
List[str].  Which it most definitely does not do.

Additionally, it seems like restricting this concept of a type guard to
only a function that returns a bool is wrong.
It is also a natural idiom to encounter functions that raise an exception
if their type conditions aren't met.
Those should also narrow types within a non-exception caught codepath after
their return.  Be simple and assume that catching any exception from their
call ends their narrowing?

A narrowing is meta-information about a typing side effect, it isn't a type
itself.  It isn't a value.  So replacing a return type with it seems like
too much.  But if we do wind up going that route, at least make the name
indicate that the return value is a bool.  i.e.: def parse_num(thing: Any)
-> NarrowingBool[float]

The term TypeGuard is too theoretical for anyone reading code.  I don't
care if TypeScript uses it in their language... looking that up at a quick
glance -
https://www.typescriptlang.org/docs/handbook/advanced-types.html#user-defined-type-guards
- it doesn't look like they ever use the term TypeGuard in the language
syntax itself? good.

Perhaps this would be better expressed as an annotation on the argument(s)
that the function narrows?

def is_str_list(val: Constrains[List[object]:List[str]) -> bool:
...

I'm using Constrains as an alternative to Narrows here... still open to
suggestions.  But I do not think we should get hung up on TypeScript and
assume that TypeGuard is somehow a good name.  This particular proposed
example uses : slice notation rather than a comma as a proposal that may be
more readable.  I'd also consider using the 'as' keyword instead of , or :
but that wouldn't even parse today.  *Really* what might be nice for this
is our -> arrow.

def assert_str_list(var:  Narrows[List[object] -> List[str]]) -> None:
...

as the arrow carries the implication of something effective "after" the
call. I suspect adding new parsing syntax isn't what anyone had in mind for
now though.  : seems like a viable stopgap, whether syntax happens or not.
(i'm intentionally avoiding the comma as an exploration in readability)

The key point I'm trying to represent is that declaring it per argument
allows for more expressive functions and doesn't interfere with their
return value (if they even return anything). Clearly there's a use case
that _does_ want to make the narrowing conditional on the return value
boolness.  I still suggest expressing that on the arguments themselves.
More specific flavors of the guarding parameter such as ConstrainsWhenTrue
or NarrowsWhenFalse perhaps.


def is_set_of(val: Constrains[Set[Any]:Set[_T]], type: Type[_T]) -> bool:
   ...

which might also be written more concisely as:

def is_set_of(val: Set[Constrains[Any:_T]], type: Type[_T]) -> bool:

If we allow the full concept.

hopefully food for thought,  i'm -1 right now.
-gps


On Fri, Feb 12, 2021 at 2:34 AM Mark Shannon  wrote:

> Hi,
>
> First of all, sorry for not commenting on this earlier.
> I only became aware of this PEP yesterday.
>
>
> I like the general idea of adding a marker to show that a boolean
> function narrows the type of one (or more?) of its arguments.
> However, the suggested approach seems clunky and impairs readability.
>
> It impairs readability, because it muddles the return type.
> The function in the example returns a bool.
> The annotation is also misleading as the annotation is on the return
> type, not on the parameter that is narrowed.
>
> At a glance, most programmers should be able to work out what
>
> def is_str_list(val: List[object]) -> bool:
>
> returns.
>
> But,
>
> def is_str_list(val: List[object]) -> TypeGuard[List[str]]:
>
> is likely to confuse and require careful reading.
> Type hints are for humans as well as type checkers.
>
>
>
> Technical review.
> -
>
> For an annotation of this kind to be useful to a checker, that checker
> must perform both flow-sensitive and call-graph analysis. Therefore it
> is theoretically possible to remove the annotation altogether, using the
> following approach:
>
> 1. Scan the code looking for functions that return boolean and
> potentially narrow the type of their arguments.
> 2. Inline those functions in the analysis
> 3. Rely on pre-existing flow-senstive analysis to determine the correct
> types.
>
> However, explicit is better and implicit. So some sort of annotation
> seems sensible.
>
> I would contend that the minimal:
>
> @narrows
> def is_str_list(val: List[object]) -> bool:
>
> is sufficient for a checker, as the checker can inline anything marked
> @narrows.
> Plus, it does not mislead 

[Python-Dev] Re: Security releases of CPython

2021-02-12 Thread Gregory P. Smith
On Thu, Feb 11, 2021 at 8:29 PM Terry Reedy  wrote:

> On 2/11/2021 3:23 PM, Michał Górny wrote:
> > Hello,
> >
> > I'm the primary maintainer of CPython packages in Gentoo. I would like
> > to discuss possible improvement to the release process in order to
> > accelerate releasing security fixes to users.
> >
> > I feel that vulnerability fixes do not make it to end users fast enough.
>
> When we introduce a bad regression in a release, including a
> severe-enough security vulnerability, and discover it soon enough, we
> have sometimes made immediately releases.
>

Indeed. What Terry said. There is risk in doing that. It means the solution
is more rushed and hasn't been vetted nearly as well. So without a
compelling severe reason we're unlikely to.

I would expect distros shipping their own python runtime packages to track
security issues and their fixes and backport those (generally easy) to
their own packages before an official release is ready anyways. The big
distros definitely do this. How to deal with this is up to the individual
policies of any given OS distro's package owners.

-gps

Beyond that, your proposal to add several releases per Python version,
> perhaps double what we do now, runs up against the limits of volunteer
> time and effort.  As it is now, becoming a release manager is a 7 year
> commitment, with at least one release about every other month for the
> first 4.  There are also the 2 people who produce the Windows and macOS
> installers.  I believe each has done it for at least 7 or 8 years
> already.  Releases are not just a push of a button.  Make the release
> job too onerous, and there might not be any more volunteers.
>
> > For example, according to the current release schedules for 3.9 and 3.8,
> > the bugfix releases are planned two months apart. While the release is
> > expected to happen in the next few days, both versions are known to be
> > vulnerable for almost a month!
> >
> > Ironically, things look even worse for security-supported versions.
> > Please correct me if I'm wrong but both 3.7 and 3.6 seem to be behind
> > schedule (planned for Jan 15th), and they are known to be vulnerable
> > since mid-October.
> >
> > In my opinion, this causes three issues:
> >
> > 1. Users using official releases are exposed to security vulnerabilities
> > for prolonged periods of time.
> >
> > 2. When releases happen, security fixes are often combined with many
> > other changes. This causes problems for distribution maintainers who, on
> > one hand, would like to deploy the security fixes to production versions
> > ASAP, and on the other, would prefer that the new version remained in
> > testing for some time due to the other changes.
> >
> > 3. Effectively, it means that distribution developers need to track
> > and backport security fixes themselves. In the end, this means a lot of
> > duplicate work.
>
> Perhaps people doing duplicate work could get together to eliminate some of
> the duplication.  It should be possible to filter security commits from the
> python-checkins list by looking at the news entries and if need be, the bpo
> issues.
>
> > I think that security fixes are important enough to justify not sticking
> > to a strict release schedule. Therefore, I would like to propose that if
> > vulnerability fixes are committed, new releases are made
> > as frequently as necessary and as soon as possible (i.e. possibly giving
> > some time for testing) rather than according to a strict schedule.
> >
> > Furthermore, I think that at least for branches that are in higher level
> > of maintenance than security, it could make sense to actually make
> > security releases (e.g. 3.9.1.x) that would include only security fixes
> > without other changes.
>
> --
> Terry Jan Reedy
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/4S5LOTVJZUA5PQ5TRGQFCWHTGA4BOXBO/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/Y6KYIFEMECIMWM7BYYSCJ7AMW2FVNUWK/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Why aren't we allowing the use of C11?

2021-01-28 Thread Gregory P. Smith
On Thu, Jan 28, 2021 at 10:52 AM Charalampos Stratakis 
wrote:

>
>
> - Original Message -
> > From: "Mark Shannon" 
> > To: "Python Dev" 
> > Sent: Thursday, January 28, 2021 5:26:37 PM
> > Subject: [Python-Dev] Why aren't we allowing the use of C11?
> >
> > Hi everyone,
> >
> > PEP 7 says that C code should conform to C89 with a subset of C99
> allowed.
> > It's 2021 and all the major compilers support C11 (ignoring the optional
> > parts).
> >
> > C11 has support for thread locals, static asserts, and anonymous structs
> > and unions. All useful features.
> >
> > Is there a good reason not to start using C11 now?
> >
> > Cheers,
> > Mark.
> >
> >
> > ___
> > Python-Dev mailing list -- python-dev@python.org
> > To unsubscribe send an email to python-dev-le...@python.org
> > https://mail.python.org/mailman3/lists/python-dev.python.org/
> > Message archived at
> >
> https://mail.python.org/archives/list/python-dev@python.org/message/PLXETSQE7PRFXBXN2QY6VNPKUTM6I7OD/
> > Code of Conduct: http://python.org/psf/codeofconduct/
> >
> >
>
> Depends what platforms the python core developers are willing to support.
>
> Currently downstream on e.g. RHEL7 we compile versions of CPython under
> gcc 4.8.2 which does not support C11.
>
> In addition the manylinux2014 base image is also based on CentOS 7, which
> wouldn't support C11 as well.
>

I *suspect* this is the primary technical reason not to adopt C11 left.

But aren't things like manylinux2014 defined by the contents of a centrally
maintained docker container?
If so (I'm not one who knows how wrong my guess likely is...), can we get
those updated to include a more modern compiler so we can move on sooner
than the deprecation of manylinux2014?

-gps


>
> --
> Regards,
>
> Charalampos Stratakis
> Software Engineer
> Python Maintenance Team, Red Hat
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/PCOZN5NHNZ7HIANOKQQ7GQQMV3A3JF72/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/MNLR45HD4X6EKYO2ARXLOF7OGTKODOJG/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Bumping minimum Sphinx version to 3.2 for cpython 3.10?

2021-01-13 Thread Gregory P. Smith
My take on this is to keep it simple for CPython: Require the newer version
of Sphinx on all branches we backport docs changes to.

We, the CPython project, determine the exact version(s) of Sphinx a
documentation build for a given version of CPython requires.  If Sphinx has
unfortunately made a breaking change (as appears to be the case?), we
should update our docs for all versions we backport doc changes to to use
the most modern version (along with the docs themselves).

We should explicitly not care at all about people using external sphinx
installs that are not fetched at the correct version via our Doc tree's
Doc/requirements.txt.

That some distros want to use their own incompatible-version packaged
version of sphinx to build our docs is on them to handle on their own.  Not
on the CPython project.  That was never a guaranteed thing that would
work.  Until now, they merely got lucky.  This is a docs build-time
dependency with no impact on anything at runtime so we should be free to
change versions as we see fit.

-gps


On Wed, Jan 13, 2021 at 8:25 AM Carol Willing  wrote:

> Hi Julien,
>
> I think that there are two items to consider:
> - `needs_sphinx` in `conf.py`
> - setting for Sphinx in `cpython/Doc/requirements.txt`
>
> I believe that the `needs_sphinx` field identifies the minimal version of
> Sphinx that can be used to build the docs. The actual version that is used
> to build the docs is in the `requirements.txt` file.
>
>
> On Wed, Jan 13, 2021 at 7:29 AM Serhiy Storchaka 
> wrote:
>
>> 12.01.21 22:38, Julien Palard via Python-Dev пише:
>> > During the development of cpython 3.10, Sphinx was bumped to 3.2.1.
>> >
>> > Problem is Sphinx 3 have some incompatibilities with Sphinx 2, some
>> that
>> > we could work around, some are bit harder, so we may need to bump
>> > `needs_sphinx = '3.2'` (currently it is 1.8).
>>
>> Sphinx version in the current Ubuntu LTS (20.04) is 1.8.5. Would not it
>> cause problems with builting documentation on Ubuntu?
>> ___
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-le...@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-dev@python.org/message/EJWTFPHZUX522RNCTIGAAOHWD23VD7NQ/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/S53K4P272EE73EF2NLZWS5DVNR6VJG3R/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/Y2ZCBJAF7NL5CP7GDGYVAICBYBIVDTC2/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: unittest of sequence equality

2020-12-22 Thread Gregory P. Smith
Numpy chose to violate the principal of equality by having __eq__ not
return a bool. So a numpy type can't be used reliably outside of the numpy
DSL.

-gps

On Tue, Dec 22, 2020, 11:51 AM Alan G. Isaac  wrote:

> Here, `seq1 == seq2` produces a boolean array (i.e., an array of boolean
> values).
> hth, Alan Isaac
>
>
> On 12/22/2020 2:28 PM, Ivan Pozdeev via Python-Dev wrote:
> >
> > You sure about that? For me, bool(np.array) raises an exception:
> >
> > In [12]: np.__version__ Out[12]: '1.19.4'
> >
> > In [11]: if [False, False]==np.array([False, False]): print("foo") <...>
> ValueError: The truth value of an array with more than one element is
> ambiguous.
> > Use a.any() or a.all()
> >
> >
>
>
>
> > On 22.12.2020 21:52, Alan G. Isaac wrote:
> >> The following test fails because because `seq1 == seq2` returns a
> (boolean) NumPy array whenever either seq is a NumPy array.
> >>
> >> import unittest import numpy as np
> unittest.TestCase().assertSequenceEqual([1.,2.,3.], np.array([1.,2.,3.]))
> >>
> >> I expected `unittest` to rely only on features of a
> `collections.abc.Sequence`, which based on
> https://docs.python.org/3/glossary.html#term-sequence, I
> >> believe are satisfied by a NumPy array. Specifically, I see no
> requirement that a sequence implement __eq__ at all much less in any
> particular way.
> >>
> >> In short: a test named `assertSequenceEqual` should, I would think,
> work for any sequence and therefore (based on the available documentation)
> should not
> >> depend on the class-specific implementation of __eq__.
> >>
> >> Is that wrong?
> >>
> >> Thank you, Alan Isaac ___
> Python-Dev mailing list -- python-dev@python.org To unsubscribe send an
> email to
> >> python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/ Message
> archived at
> >>
> https://mail.python.org/archives/list/python-dev@python.org/message/6Z43SU2RPIHTRABYAXBOGRKWGTLIFJYK/
> Code of Conduct:
> >> http://python.org/psf/codeofconduct/
> >
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/74CUML37OCQXXKMVUFZORMSCOYP2W5GH/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/FNMNQVMKCN3F3XHLESZLV2NT6XGV2DIM/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: macOS issues with 3.8.7rc1

2020-12-09 Thread Gregory P. Smith
As a meta question: Is there a good reason to support binaries running on
macOS earlier than ~ $latest_version-1?

Aren't systems running those old releases rather than upgrading unsupported
by Apple, never to be patched, and thus not wise to even have on a network?

Yes, that means some very old hardware becomes useless as Apple drops
support. But that is what people signed up for when they bought it. Why
should that be our problem?

(It sounds like y'all will make it work, that's great! I'm really just
wondering where the motivation comes from)

-gps

On Wed, Dec 9, 2020, 9:25 AM Gregory Szorc  wrote:

> On Wed, Dec 9, 2020 at 4:13 AM Ronald Oussoren 
> wrote:
>
>>
>>
>> On 8 Dec 2020, at 19:59, Gregory Szorc  wrote:
>>
>> Regarding the 3.8.7rc1 release, I wanted to raise some issues regarding
>> macOS.
>>
>> Without the changes from https://github.com/python/cpython/pull/22855
>> backported, attempting to build a portable binary on macOS 11 (e.g. by
>> setting `MACOSX_DEPLOYMENT_TARGET=10.9`) results in a myriad of `warning:
>> 'XXX' is only available on macOS 10.13 or newer
>> [-Wunguarded-availability-new]` warnings during the build. This warning
>> could be innocuous if there is run-time probing in place (the symbols in
>> question are weakly linked, which is good). But if I'm reading the code
>> correctly, run-time probing was introduced by commits like eee543722 and
>> isn't present in 3.8.7rc1.
>>
>> I don't have a machine with older macOS sitting around to test, but I'm
>> fairly certain the lack of these patches means binaries built on macOS 11
>> will blow up at run-time when run on older macOS versions.
>>
>> These same patches also taught CPython to build and run properly on Apple
>> ARM hardware. I suspect some people will care about these being backported
>> to 3.8.
>>
>> We know. Backporting the relevant changes to 3.8 is taking more time than
>> I had hoped. It doesn’t help that I’ve been busy at work and don’t have as
>> much energy during the weekend as I’d like.
>>
>> The backport to 3.9 was fairly easy because there were few changes
>> between master and the 3.9 branch at the time. Sadly there have been
>> conflicting changes since 3.8 was forked (in particular in posixmodule.c).
>>
>> The current best practice for building binaries that work on macOS 10.9
>> is to build on that release (or rather, with that SDK).  That doesn’t help
>> if you want to build Universal 2 binaries though.
>>
>
> Thank you for your hard work devising the patches and working to backport
> them.
>
> I personally care a lot about these patches and I have the technical
> competency to perform the backport. If you need help, I could potentially
> find time to hack on it. Just email me privately (or ping @indygreg on
> GitHub) and let me know. Even if they don't get into 3.8.7, I'll likely
> cherry pick the patches for
> https://github.com/indygreg/python-build-standalone. And I'm sure other
> downstream packagers will want them as well. So having them in an
> unreleased 3.8 branch is better than not having them at all.
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/5AWFX2POTPNVW72VUPCPTJIOA6AOSVWY/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/Q45VNQYOYXTRRTA26Q4M2WNXPFKL4S2O/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: __future__ annotations loses closure scope

2020-12-08 Thread Gregory P. Smith
What is the utility of a type annotation when the thing it refers to cannot
exist?

Deferred annotation lookups are intended to be something that analysis time
can make sense of but can always have no useful meaning at runtime.

No nesting required:

```
from __future__ import annotations
Class X:
  ...

def foo(hi: X):
  ...

del X
```

Now try analyzing foo at runtime...  I assume "Boom" with that NameError
again?  (*On a phone, can't try it now)*

I believe this isn't a problem get_type_hints() can ever solve.

Code that does this isn't what I'd call "reasonably" structured for use
with type hints.

If anything, type checkers should try to warn about it?

-gps

On Tue, Dec 8, 2020, 7:03 PM Paul Bryan  wrote:

> Let's try an example that static type checkers should have no problem with:
>
> Python 3.9.0 (default, Oct  7 2020, 23:09:01)
>
> [GCC 10.2.0] on linux
>
> Type "help", "copyright", "credits" or "license" for more information.
>
> >>> from __future__ import annotations
>
> >>>
>
> >>> def make_a_class():
>
> ... class A:
>
> ... def get_b(self) -> B:
>
> ... return B()
>
> ... class B:
>
> ... def get_a(self) -> A:
>
> ... return A()
>
> ... return A
>
> ...
>
> >>> A = make_a_class()
>
> >>> a = A()
>
> >>>
>
> >>> import typing
>
> >>> typing.get_type_hints(a.get_b)
>
> Traceback (most recent call last):
>
>   File "", line 1, in 
>
>   File "/usr/lib/python3.9/typing.py", line 1386, in get_type_hints
>
> value = _eval_type(value, globalns, localns)
>
>   File "/usr/lib/python3.9/typing.py", line 254, in _eval_type
>
> return t._evaluate(globalns, localns, recursive_guard)
>
>   File "/usr/lib/python3.9/typing.py", line 493, in _evaluate
>
> eval(self.__forward_code__, globalns, localns),
>
>   File "", line 1, in 
>
> NameError: name 'B' is not defined
>
> >>>
>
>
>
>
> On Tue, 2020-12-08 at 18:48 -0800, Guido van Rossum wrote:
>
> Yeah, static type checkers won't like it regardless.
>
> On Tue, Dec 8, 2020 at 6:39 PM Paul Bryan  wrote:
>
> It appears that when from future import __annotations__, a type hint
> annotation derived from a closure loses scope.
>
> Simplistic example:
>
> Python 3.9.0 (default, Oct  7 2020, 23:09:01)
>
> [GCC 10.2.0] on linux
>
> Type "help", "copyright", "credits" or "license" for more information.
>
> >>> def make_a_class(data_type):
>
> ... class Foo:
>
> ... def put_data(self, data: data_type):
>
> ... self.data = data
>
> ... return Foo
>
> ...
>
> >>> import typing
>
> >>> foo = make_a_class(str)()
>
> >>> typing.get_type_hints(foo.put_data)
>
> {'data': }
>
> >>>
>
>
> If I add a single import to the top, it breaks:
>
> Python 3.9.0 (default, Oct  7 2020, 23:09:01)
>
> [GCC 10.2.0] on linux
>
> Type "help", "copyright", "credits" or "license" for more information.
>
> *>>> from __future__ import annotations  # added this line*
>
> >>> def make_a_class(data_type):
>
> ... class Foo:
>
> ... def put_data(self, data: data_type):
>
> ... self.data = data
>
> ... return Foo
>
> ...
>
> >>> import typing
>
> >>> foo = make_a_class(str)()
>
> >>> typing.get_type_hints(foo.put_data)
>
> Traceback (most recent call last):
>
>   File "", line 1, in 
>
>   File "/usr/lib/python3.9/typing.py", line 1386, in get_type_hints
>
> value = _eval_type(value, globalns, localns)
>
>   File "/usr/lib/python3.9/typing.py", line 254, in _eval_type
>
> return t._evaluate(globalns, localns, recursive_guard)
>
>   File "/usr/lib/python3.9/typing.py", line 493, in _evaluate
>
> eval(self.__forward_code__, globalns, localns),
>
>   File "", line 1, in 
>
> NameError: name 'data_type' is not defined
>
> >>>
>
>
> I don't see how I can supply the closure scope as localns to
> get_type_hints. Any suggestions? Is constructing a
> (dynamically-type-annotated) class in a function like this an anti-pattern?
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/5RK6VXF263F5I4CU7FUMOGOYN2UQG73Q/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/NRH4HBD36WDIP4WR2L4TLTOYMQL2NUFV/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived 

[Python-Dev] Re: nanosecond stat fields, but not os.path methods ?

2020-12-07 Thread Gregory P. Smith
If the precision is available via OS APIs, this is mostly an issue+PR away
from being implemented by someone who cares.

FAT32 has a two billion nanosecond resolution IIRC. :P

-gps

On Mon, Dec 7, 2020 at 8:22 AM David Mertz  wrote:

> Are there any filesystems that can actually record a meaningful ns
> modification time?  I find discussions claiming this:
>
>
>- XFS and EXT3: second precision
>- EXT4: millisecond precision
>- NTFS: 100ns precision
>- APFS: 1 ns precision
>
>
> But also notes that the precision is likely to exceed the accuracy by many
> times on real systems.
>
> On Mon, Dec 7, 2020 at 2:34 PM Mats Wichmann  wrote:
>
>>
>> there are stat fields now for ns precision, e.g. st_mtime now has an
>> analogue st_mtime_ns.  But os.path didn't grow corresponding methods -
>> there's an os.path.getmtime but not _ms. Was that intentional?  The
>> wrappers in genericpath.py are trivial and arguably aren't particularly
>> needed, but still curious...
>> ___
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-le...@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-dev@python.org/message/CK3S2DYI3PKZ7VGRFEO4PKVLZCPUTR6N/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
>
> --
> The dead increasingly dominate and strangle both the living and the
> not-yet born.  Vampiric capital and undead corporate persons abuse
> the lives and control the thoughts of homo faber. Ideas, once born,
> become abortifacients against new conceptions.
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/ZAPQSEQHL3KO7AALG7NQTIA2BPG753AN/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/WJU7EMOTVCYA36CFBBEPR3DOIX4KV77R/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Drop Solaris, OpenSolaris, Illumos and OpenIndiana support in Python

2020-10-30 Thread Gregory P. Smith
On Fri, Oct 30, 2020 at 2:30 PM Garrett D'Amore via Python-Dev <
python-dev@python.org> wrote:

> I’m not on this list.  But I have offered to help - if there are tasks
> that need to be done to help this I can help put the weight of a commercial
> entity behind it whether that involves assigning our developers to work on
> this, helping pay for external developers to do so, or assisting with
> access to machine resources.
>
> For the record there are multiple  illumos distributions and most are both
> free and run reasonably well in virtual machines. Claiming that developers
> don’t have access as a reason to discontinue the port is a bit
> disingenuous. Anyone can get access if they want and if they can figure out
> how to login and use Linux then this should be pretty close to trivial for
> them.
>

Thanks!  It usually isn't just about access.  This email thread and related
tweets appear to have served their purpose: To drum up volunteers+resources
from the otherwise potentially impacted communities.  The valuable thing is
developer time.

Access: I took it upon myself to spin up some Solaris-derivative VMs for
Python dev things in the (now distant) past.  It wasn't a positive
experience, I won't do it again.  Bonus on top of that: Oracle, the owner
of Solaris, is *still* actively attempting to destroy the entire software
industry .
Working on anything attempting to directly benefit them is a major ethical
violation for me.  Others make their own choices.

I won't even spin up major BSD VMs anymore for a similar reason.  It isn't
a good positive use of my time, even despite having an enjoyable past with
those OSes and knowing several past and present Free/Net/OpenBSD core devs.

I look forward to new *Solaris buildbot(s) joining the fleet,
-gps
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/7Y7BXPMRPYJE6CUCMIYBY3EXOKQLMIVW/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Drop Solaris, OpenSolaris, Illumos and OpenIndiana support in Python

2020-10-30 Thread Gregory P. Smith
On Fri, Oct 30, 2020 at 1:14 PM Raymond Hettinger <
raymond.hettin...@gmail.com> wrote:

> FWIW, when the tracker issue landed with a PR, I became concerned that it
> would be applied without further discussion and without consulting users.


An issue and a PR doesn't simply mean "it is happening".  It is an
effective way to communicate and demonstrate a possible change.  It appears
to have served its purpose.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BSN5BIHQWBXZ2G5EXVWKEIEBC4X6KJYO/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Drop Solaris, OpenSolaris, Illumos and OpenIndiana support in Python

2020-10-30 Thread Gregory P. Smith
On Fri, Oct 30, 2020 at 11:03 AM Brett Cannon  wrote:

>
> On Fri, Oct 30, 2020 at 6:37 AM Pablo Galindo Salgado 
> wrote:
>
>> >Two volunteer core developers and at least one buildbot would help a
>> > lot to ensure that Python is working on Solaris for real, and reduce
>> > the number of open Solaris issues. If it happens, I'm perfectly fine
>> > with keeping Solaris support.
>> > I also hope that more people will contribute to maintain the code, not
>> > only Ronald and Jesús. Many people wrote to me that Python is a key
>> > component of Illumos (the package manager is written in Python). So
>> > maybe Python on Illumos deserves some love?
>>
>> If we decide to take the proposal seriously, I think it may be beneficial
>> to establish a deadline for having the buildbot green and the issues
>> fixed,
>> so the promise of having "real" support at some point in the future does
>> actually manifest itself or it does not block following Victor's proposal
>> if it
>> actually does not happen.
>>
>
> I agree, and so maybe it's time to more formally establish what platforms
> we do support, such that any platform not listed is not considered
> supported. We could reorient PEP 11 towards that and for each platform we
> list:
>
> 1. Which buildbot must be green for that platform to be considered
> supported
> 2. Who is in charge of submitting patches to keep that buildbot green (and
> what that minimum number of people is)
> 3. We have an agreed upon timeline that if a buildbot stays red for longer
> than the specified time then the platform is considered abandoned
> 4. We all agree to prioritize patches from the people listed for a
> platform to fix their buildbots if they are not core devs
> 5. We have a clear definition of "platform" (e.g. is "Linux" a platform,
> or does it break down to individual distros?)
>
> Would there be anything else we would want for a platform for us to be
> willing to consider it supported?
>

If we're going to explicitly list Platforms, that gets messy to maintain.
It becomes important to draw a distinction between types of support.
Support is not a boolean.  We're autoconf based on the posix side (like it
or not) which leads to a lot of things that for the most part just mostly
work, regardless of support.  That is working as intended.

Generally, recent enough of each of: Linux distros (all architectures),
macOS, Windows, and AIX.  But that alone does not constitute a platform.

But to define things explicitly you need a definition of what a Platform is:

is it an OS kernel? what version(s)? compiler toolchain? what versions? Is
it also the C library? what version(s) of which libc variants (linux has at
least three)? other system libraries? what versions? specific CPU
architectures? what versions.  The matrix gets huge.

It can be defined.

But then you need to clarify the different levels of not being in that
matrix. "on the list, all clear" vs "likely works, no guarantees" vs
"expect lots of test failures" vs "expect extension module build failures"
vs "expect the core interpreter build to fail".

We've not done this in the past.  Would it even add value?

It is much easier to look at the list of buildbots tagged "stable".  and
consider that "on the list, all clear" and everything else is in an
unspecified one of the other four+ categories without arguments over which
sub-support list any given thing is in.

-gps


> -Brett
>
>
>>
>> On Fri, 30 Oct 2020 at 12:54, Victor Stinner  wrote:
>>
>>> Hi Ronald,
>>>
>>> Le ven. 30 oct. 2020 à 12:59, Ronald Oussoren 
>>> a écrit :
>>> > I agree. That’s what I tried to write, its not just providing a
>>> buildbot but also making sure that it keeps working and stays green.
>>>
>>> This is really great!
>>>
>>> Jesús Cea Avión is also a volunteer to maintain the Solaris (see the
>>> bpo).
>>>
>>> Moreover, it seems like some people would like to provide servers to
>>> run a Solaris buildbot. Example:
>>> https://bugs.python.org/issue42173#msg379895
>>>
>>> Two volunteer core developers and at least one buildbot would help a
>>> lot to ensure that Python is working on Solaris for real, and reduce
>>> the number of open Solaris issues. If it happens, I'm perfectly fine
>>> with keeping Solaris support.
>>>
>>> I also hope that more people will contribute to maintain the code, not
>>> only Ronald and Jesús. Many people wrote to me that Python is a key
>>> component of Illumos (the package manager is written in Python). So
>>> maybe Python on Illumos deserves some love?
>>>
>>> There are many ways to contribute to the Solaris support of Python:
>>>
>>> * Comment Solaris issues (bugs.python.org, search for "Solaris" in the
>>> title)
>>> * Propose PRs to fix issues or implement Solaris specific features
>>> * Review Solaris PRs
>>> * Provide Solaris servers accessible to Python core developers (SSH
>>> access)
>>> * Donate to the CPython project:
>>>
>>>   * https://www.python.org/psf/donations/python-dev/
>>>   * 

[Python-Dev] Re: Drop Solaris, OpenSolaris, Illumos and OpenIndiana support in Python

2020-10-30 Thread Gregory P. Smith
On Fri, Oct 30, 2020 at 8:26 AM Sebastian Wiedenroth 
wrote:

> Hi,
>
> I've already commented on the issue, but want to make a few more points
> here as well.
>
> Removing Solaris support would not only impact Oracle Solaris but the open
> source illumos community as well.
> Both systems share the "SunOS" uname for historical reasons. Removing
> support would be a disaster for us.
>
> We are a small community compared to Linux, but there are illumos
> distributions (OpenIndiana, OmniOS, SmartOS, Tribblix, ...) that have many
> python users.
> It's also an essential part of our tooling and package management.
>
> I've offered to host an illumos buildbot before but it was not accepted
> because not all tests passed at that time.
> There is active work going on to get this debugged and fixed.
> If it is acceptable to skip some tests we can have the buildbot online
> tomorrow.
>

FWIW making a PR that adds platform specific test skips or expected failure
decorators is a good way to start bringing up new buildbots.  It serves as
effective documentation of what does and doesn't work that lives directly
in the codebase, in a way that can be evolved over time as more is made to
work.

-gps

On the ticket many users and developers have offered to step up, myself
> included.
> In our IRC channel we also had some discussions yesterday and we're hoping
> to bring more patches upstream soon.
>
> If there is interest in ssh access to illumos systems that is also
> something I can offer.
>
> Please let us know if there is more we need to do to keep python on
> illumos supported.
>
> Best regards,
> Sebastian
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/UL4MQAKQZOIEEA2DHNJNB4BB4J3QR7FY/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/S3MDFH6PHZR2TN4NYLM4PBNQPHQZZMWW/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Drop Solaris, OpenSolaris, Illumos and OpenIndiana support in Python

2020-10-29 Thread Gregory P. Smith
I agree, remove Solaris support. Nobody willing to contribute seems
interested.

-gps

--
blame half the typos on my phone.

On Thu, Oct 29, 2020, 2:50 PM Victor Stinner  wrote:

> Hi,
>
> I propose to drop the Solaris support in Python to reduce the Python
> maintenance burden:
>
>https://bugs.python.org/issue42173
>
> I wrote a draft PR to show how much code could be removed (around 700
> lines in 65 files):
>
>https://github.com/python/cpython/pull/23002/files
>
> In 2016, I asked if we still wanted to maintain the Solaris support in
> Python, because Solaris buildbots were failing for longer than 6
> months and nobody was able to fix them. It was requested to find a
> core developer volunteer to fix Solaris issues and to set up a Solaris
> buildbot.
>
>
> https://mail.python.org/archives/list/python-dev@python.org/thread/NOT2RORSNX72ZLUHK2UUGBD4GTPNKBUS/#NOT2RORSNX72ZLUHK2UUGBD4GTPNKBUS
>
> Four years later, nothing has happened. Moreover, in 2018, Oracle laid
> off the Solaris development engineering staff. There are around 25
> open Python bugs specific to Solaris.
>
> I see 3 options:
>
> * Current best effort support (no change): changes only happen if a
> core dev volunteers to review and merge a change written by a
> contributor.
>
> * Schedule the removal in 2 Python releases (Python 3.12) and start to
> announce that Solaris support is going to be removed
>
> * Remove the Solaris code right now (my proposition): Solaris code
> will have to be maintained outside the official Python code base, as
> "downstream patches"
>
>
> Solaris has a few specific features visible at the Python level:
> select.devpoll, os.stat().st_fstype and stat.S_ISDOOR().
>
> While it's unclear to me if Oracle still actively maintains Solaris
> (latest release in 2018, no major update since 2018), Illumos and
> OpenSolaris (variants or "forks") still seem to be active.
>
> In 2019, a Solaris blog post explains that Solaris 11.4 still uses
> Python 2.7 but plans to migrate to Python 3, and Python 3.4 is also
> available. These two Python versions are no longer supported.
>
> https://blogs.oracle.com/solaris/future-of-python-on-solaris
>
> The question is if the Python project has to maintain the Solaris
> specific code or if this code should now be maintained outside Python.
>
> What do you think? Should we wait 5 more years? Should we expect a
> company will offer to maintain the Solaris support? Is there a
> motivated core developer to fix Solaris issue? As I wrote, nothing has
> happened in the last 4 years...
>
> Victor
> --
> Night gathers, and now my watch begins. It shall not end until my death.
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/VDD7NMEDFXMOP4S74GEYJUHJRJPK2UR3/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/HHP6DVPMDNUHEZZPKJDLZTMOI7O7LWPO/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: os.scandir bug in Windows?

2020-10-26 Thread Gregory P. Smith
On Mon, Oct 26, 2020, 4:06 PM Chris Angelico  wrote:

> On Tue, Oct 27, 2020 at 10:00 AM Greg Ewing 
> wrote:
> >
> > On 27/10/20 8:24 am, Victor Stinner wrote:
> > > I would
> > > rather want to kill the whole concept of "access" time in operating
> > > systems (or just configure the OS to not update it anymore). I guess
> > > that it's really hard to make it efficient and accurate at the same
> > > time...
> >
> > Also it's kind of weird that just looking at data on the
> > disk can change something about it. Sometimes it's an
> > advantage to *not* have quantum computing!
> >
>
> And yet, it's of incredible value to be able to ask "now, where was
> that file... the one that I was looking at last week, called something
> about calendars, and it had a cat picture in it". Being able to answer
> that kinda depends on recording accesses one way or another, so the
> weirdnesses are bound to happen.
>

scandir is never going to answer that. Neither is a simple blind "access"
time stored in filesystem metadata.

ChrisA
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/ZMNVRGZ7ZEC5EAKLUOX64R4WKHOLPF4O/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YW5NMIE2SC3RQWDMJX2DVIS3FRHNPEQM/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Speeding up CPython

2020-10-21 Thread Gregory P. Smith
On Wed, Oct 21, 2020 at 4:04 PM Greg Ewing 
wrote:

> A concern I have about this is what effect it will have on the
> complexity of CPython's implementation.
>
> CPython is currently very simple and straightforward. Some parts
> are not quite as simple as they used to be, but on the whole it's
> fairly easy to understand, and I consider this to be one of its
> strengths.
>
> I worry that adding four layers of clever speedup tricks will
> completely destroy this simplicity, leaving us with something
> that can no longer be maintained or contributed to by
> ordinary mortals.
>

I have never considered ceval.c to be simple and straightforward.  Nor our
parser(s).  Or our optimizers.  Or the regular expression implementation.
Or the subprocess internals.  (we may differ on lists of what isn't simple
and straightforward, but i guarantee you we've all got something in mind)

Making this not a major concern for me.

We'd decide if there is something we find to be dark magic that seems
challenging to maintain during the review phases and decide what if
anything needs to be done about that to ameliorate such issues.

-gps


> --
> Greg
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/6XBJ2OA46KNMJ6FFI3B6RFYRTTD4HYOI/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QMIKJ3EE73U3J6TM4RGHMTHGB6N4WZGV/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Speeding up CPython

2020-10-20 Thread Gregory P. Smith
On Tue, Oct 20, 2020 at 5:59 AM Mark Shannon  wrote:

> Hi everyone,
>
> CPython is slow. We all know that, yet little is done to fix it.
>
> I'd like to change that.
> I have a plan to speed up CPython by a factor of five over the next few
> years. But it needs funding.
>
> I am aware that there have been several promised speed ups in the past
> that have failed. You might wonder why this is different.
>
> Here are three reasons:
> 1. I already have working code for the first stage.
> 2. I'm not promising a silver bullet. I recognize that this is a
> substantial amount of work and needs funding.
> 3. I have extensive experience in VM implementation, not to mention a
> PhD in the subject.
>
> My ideas for possible funding, as well as the actual plan of
> development, can be found here:
>
> https://github.com/markshannon/faster-cpython
>
> I'd love to hear your thoughts on this.
>
> Cheers,
> Mark.
>

+1

Overall I think you are making quite a reasonable proposal.  It sounds
like effectively bringing your hotpy2 concepts into the CPython interpreter
with an intent to help maintain them over the long term.

People worried that you are doing this out of self interest may not know
who you are.  Sure, you want to be paid to do work that you appear to love
and have been mulling over for a decade+.  There is nothing wrong with
that.  Payment is proposed as on delivery per phase.  I like the sound of
that, nice!

Challenges I expect we'll face, that seem tractable to me, are mostly
around what potential roadblocks we, us existing python-committers and our
ultimate decider steering council might introduce intentionally or not that
prevents landing such work.  Payment on delivery helps that a lot, if we
opt out of some work, it is both our losses.  One potential outcome is that
you'd burn out and go away if we didn't accept something meaning payment
wasn't going to happen.  That already happens amongst all core devs today
so I don't have a problem with this even though it isn't what we'd
rightfully want to happen.  Middle grounds are quite reasonable
renegotiations.  The deciders on this would be the PSF (because money) and
the PSF would presumably involve the Steering Council in decisions of terms
and judgements.

Some background materials for those who don't already know Mark's past work
along these lines that is where this proposal comes from:
  https://sites.google.com/site/makingcpythonfast/ (hotpy)
  and the associated presentation in 2012:
https://www.youtube.com/watch?v=c6PYnZUMF7o

The amount of money seems entirely reasonable to me. Were it to be taken
on, part of the PSF's job is to drum up the money. This would be a contract
with outcomes that could effectively be sold to funders in order to do so.
There are many companies who use CPython a lot that we could solicit
funding from, many of whom have employees among our core devs already. Will
they bite? It doesn't even have to be from a single source or be all
proposed phases up front, that is what the PSF exists to decide and
coordinate on.

We've been discussing on and off in the past many years how to pay people
for focused work on CPython and the ecosystem and balance that with being
an open source community project.  We've got some people employed along
these lines already, this would become more of that and in many ways just
makes sense to me.

Summarizing some responses to points others have brought up:

Performance estimates:
 * Don't fret about claimed speedups of each phase.  We're all going to
doubt different things or expect others to be better.  The proof is
ultimately in the future pudding.

JIT topics:
 * JITs rarely stand alone. The phase 1+2 improved interpreter will always
exist. It is normal to start with an interpreter for fast startup and
initial tracing before performing JIT compilations, and as a fallback
mechanism when the JIT isn't appropriate or available. (my background:
Transmeta. We had an Interpreter and at least two levels of Translators
behind our x86 compatible CPUs, all were necessary)
 * Sometimes you simply want to turn tracing and jit stuff off to save
memory. That knob always winds up existing. If nothing else it is normal to
run our test suite with such a knob in multiple positions for proper
coverage.
 * It is safe to assume any later phase actual JIT would target at least
one important platform (ex: amd64 or aarch64) and if successful should
easily elicit contributions supporting others either as work or as funding
to create it.

"*Why this, why not fund XyZ?*" whataboutism:
 * This conversation is separate from other projects. The way attracting
funding for a project works can involve spelling out what it is for. It
isn't my decision, but I'd be amazed if anything beyond maybe phase 1 came
solely out of a PSF general no obligation fund. CPython is the most used
Python VM in the world. A small amount of funding is not going to get
maintainers and users to switch to PyPy.  There is unlikely to be a major
this or 

[Python-Dev] Re: os.scandir bug in Windows?

2020-10-19 Thread Gregory P. Smith
On Mon, Oct 19, 2020 at 6:28 AM Ivan Pozdeev via Python-Dev <
python-dev@python.org> wrote:

>
> On 19.10.2020 14:47, Steve Dower wrote:
> > On 19Oct2020 1242, Steve Dower wrote:
> >> On 15Oct2020 2239, Rob Cliffe via Python-Dev wrote:
> >>> TLDR: In os.scandir directory entries, atime is always a copy of mtime
> rather than the actual access time.
> >>
> >> Correction - os.stat() updates the access time to _now_, while
> os.scandir() returns the last access time without updating it.
> >
> > Let me correct myself first :)
> >
> > *Windows* has decided not to update file access time metadata *in
> directory entries* on reads. os.stat() always[1] looks at the file entry
> > metadata, while os.scandir() always looks at the directory entry
> metadata.
>
> Is this behavior documented somewhere?
>
> Such weirdness certaintly something that needs to be documented but I
> really don't like describing such quirks that are out of our control
> and may be subject to change in Python documentation. So we should only
> consider doing so if there are no other options.
>

I'm sure this is covered in MSDN.  Linking to that if it has it in a
concise explanation would make sense from a note in our docs.

If I'm understanding Steve correctly this is due to Windows/NTFS storing
the access time potentially redundantly in two different places. One within
the directory entry itself and one with the file's own metadata.  Those of
us with a traditional posix filesystem background may raise eyeballs at
this duplication, seeing a directory as a place that merely maps names to
inodes with the inode structure (equiv: file entry metadata) being the sole
source of truth.  Which ones get updated when and by what actions is up to
the OS.

So yes, just document the "quirk" as an intended OS behavior.  This is one
reason scandir() can return additional information on windows vs what it
can return on posix.  The entire point of scandir() is to return as much as
possible from the directory without triggering reads of the
inodes/file-entry-metadata. :)

-gps


>
> >
> > My suggested approach still applies, other than the bit where we might
> fix os.stat(). The best we can do is regress os.scandir() to have
> > similarly poor performance, but the best *you* can do is use os.stat()
> for accurate timings when files might be being modified while your
> > program is running, and don't do it when you just need names/kinds (and
> I'm okay adding that note to the docs).
> >
> > Cheers,
> > Steve
> >
> > [1]: With some fallback to directory entries in exceptional cases that
> don't apply here.
> > ___
> > Python-Dev mailing list -- python-dev@python.org
> > To unsubscribe send an email to python-dev-le...@python.org
> > https://mail.python.org/mailman3/lists/python-dev.python.org/
> > Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/QHHJFYEDBANW7EC3JOUFE7BQRT5ILL4O/
> > Code of Conduct: http://python.org/psf/codeofconduct/
> > --
> > Regards,
> > Ivan
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/VFXDBURSZ4QKA6EQBZLU6K4FKMGZPSF5/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/IZ6KSRTJLORCB33OMVUPFYQYLMBM26EJ/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: os.scandir bug in Windows?

2020-10-17 Thread Gregory P. Smith
Could you please file this as an issue on bugs.python.org?

Thanks!
-Greg


On Sat, Oct 17, 2020 at 7:25 PM Rob Cliffe via Python-Dev <
python-dev@python.org> wrote:

>
> TLDR: In os.scandir directory entries, atime is always a copy of mtime
> rather than the actual access time.
>
> Demo program: Windows 10, Python 3.8.3:
>
> # osscandirtest.py
> import time, os
> with open('Test', 'w') as f: f.write('Anything\n') # Write to a file
> time.sleep(10)
> with open('Test', 'r') as f: f.readline() # Read the file
> print(os.stat('Test'))
> for DirEntry in os.scandir('.'):
>  if DirEntry.name == 'Test':
>  stat = DirEntry.stat()
>  print(f'scandir DirEntry {stat.st_ctime=} {stat.st_mtime=}
> {stat.st_atime=}')
>
> Sample output:
>
> os.stat_result(st_mode=33206, st_ino=8162774324687317,
> st_dev=2230120362, st_nlink=1, st_uid=0,
> st_gid=0, st_size=10, st_atime=1600631381, st_mtime=1600631371,
> st_ctime=1600631262)
> scandir DirEntry stat.st_ctime=1600631262.951019
> stat.st_mtime=1600631371.7062848 stat.st_atime=1600631371.7062848
>
> For os.stat, atime is 10 seconds more than mtime, as would be expected.
> But for os.scandir, atime is a copy of mtime.
> ISTM that this is a bug, and in fact recently it stopped me from using
> os.scandir in a program where I needed the access timestamp. No big
> deal, but ...
> If it is a feature for some reason, presumably it should be documented.
>
> Best wishes
> Rob Cliffe
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/RIKQAXZVUAQBLECFMNN2PUOH322B2BYD/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/INJBNXRKOBYFGFJ7CLHNJKVQQKU6X6NM/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Hygenic macros PEP.

2020-09-15 Thread Gregory P. Smith
On Tue, Sep 15, 2020 at 1:27 PM  wrote:

> September 15, 2020 4:02 PM, "Daniel Butler"  >
> wrote:
>
> > Users would be encouraged to try it but
>
> > NOT to publish code using it.
>
> Thinking out loud, macros could be enabled with a command line flag.
> Advanced users would know how to use it but others would not. If the macro
> flag is not enabled it raises a syntax error. Thoughts?
> --
> Thank you!
>
> Daniel Butler
>
>
> A command line flag would be slightly better. All the documentation
> warnings in the world will not be enough to prevent all the cargo culters
> from creating some of the hardest to read code you ever saw.
>

If you're talking about a command line flag, I suggest you read the
pre-PEP.  The proposal requires explicit import-like syntax to bring the
macro in for parsing of code in the duration of the scope of that import.
Which is actually what you'd want for this kind of thing: explicitly
declaring which additional / alternate syntax features you use in the code
using it. It is similar from the existing `from __future__ import behavior`
system.

-gps
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6Z7NFMSP32NYW23J3EEH2R7QZF2EHZ5O/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: PEP 626: Precise line numbers for debugging and other tools.

2020-07-29 Thread Gregory P. Smith
On Tue, Jul 28, 2020 at 2:12 PM Jim J. Jewett  wrote:

> ah... we may have been talking past each other.
>
> Steve Dower wrote:
> > On 25Jul2020 2014, Jim J. Jewett wrote:
> > > But it sounds as though you are saying the benefit
>
> [of storing the line numbers in an external table, I thought,
> but perhaps Pablo Galindo Salgado and yourself were
> talking only of the switch from an lnotab string to an opaque
> co_linetable?]
>
> > > is irrelevant; it is just inherently too expensive to ask programs
> that are already dealing
> > > with internals and trying to optimize performance to make a mechanical
> change from:
> > >  code.magic_attrname
> > > to:
> > >  magicdict[code]
> > > What have I missed?
>
> > You've missed that debugging and profiling tools that operate purely on
> > native memory can't execute Python code, so the "magic" has to be easily
> > representable in C such that it can be copied into whichever language is
> > being used (whether it's C, C++, C#, Rust, or something else).
>
> Unless you really were talking only of the switch to co_linetable, I'm
> still
> missing the problem.  To me, it still looks like a call to:
>
> PyAPI_FUNC(PyObject *) PyObject_GetAttrString(PyObject *, const char
> *);
>
> with the code object being stepped through and "co_lnotab"
> would be replaced by:
>
> PyAPI_FUNC(PyObject *) PyDict_GetItem(PyObject *mp, PyObject *key);
>
> using that same code object as the key, but getting the dict from
> some well-known (yet-to-be-defined) location, such as sys.code_to_lnotab.
>
> Mark Shannon and Carl Shapiro had seemed to object to the PEP because
> the new structure would make the code object longer, and making it smaller
> by a string does seem likely to be good.  But if your real objections are
> to
> just to replacing the lnotab format with something that needs to be
> executed, then I apologize for misunderstanding.
>

Introspection of the running CPython process is happening from outside of
the CPython interpreter itself.  Either from a signal handler or C/C++
managed thread within the process, or (as Pablo suggested) from outside the
process entirely.  Calling CPython APIs is a non-option in all of those
situations.

That is why I suggested that the "undocumented" new co_linetable will be
used instead of the disappeared co_lnotab regardless of documentation or
claimed stability guarantees.  It sounds like an equivalent read only data
source for this purpose.  It doesn't matter to anyone with such a profiler
if it is claimed to be unspecified.

The data is needed, and the format shouldn't change within a stable python
major.minor release (we'd be unlikely to anyways even without that
guarantee).  Given this, I suggest at least specifying valuable properties
of it such as "read only, never mutated" even if the exact format is
intentionally left as implementation defined, subject to change between
minor releases structure.

-gps


>
> -jJ
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/WUEFHFTPVTOPA3EFHACDECT3ZPLGGTFJ/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/MQHSIXLAUL2ZSRORRDJEF3I3T73XP772/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: PEP 626: Precise line numbers for debugging and other tools.

2020-07-22 Thread Gregory P. Smith
On Wed, Jul 22, 2020 at 5:19 AM Mark Shannon  wrote:

>
>
> On 21/07/2020 9:46 pm, Gregory P. Smith wrote:
> >
> >
> > On Fri, Jul 17, 2020 at 8:41 AM Ned Batchelder  > <mailto:n...@nedbatchelder.com>> wrote:
> >
> > https://www.python.org/dev/peps/pep-0626/ :)
> >
> > --Ned.
> >
> > On 7/17/20 10:48 AM, Mark Shannon wrote:
> >  > Hi all,
> >  >
> >  > I'd like to announce a new PEP.
> >  >
> >  > It is mainly codifying that Python should do what you probably
> > already
> >  > thought it did :)
> >  >
> >  > Should be uncontroversial, but all comments are welcome.
> >  >
> >  > Cheers,
> >  > Mark.
> >
> >
> > """When a frame object is created, the f_lineno will be set to the line
> > at which the function or class is defined. For modules it will be set to
> > zero."""
> >
> > Within this PEP it'd be good for us to be very pedantic.  f_lineno is a
> > single number.  So which number is it given many class and function
> > definition statements can span multiple lines.
> >
> > Is it the line containing the class or def keyword?  Or is it the line
> > containing the trailing :?
>
> The line of the `def`/`class`. It wouldn't change for the current
> behavior. I'll add that to the PEP.
>
> >
> > Q: Why can't we have the information about the entire span of lines
> > rather than consider a definition to be a "line"?
>
> Pretty much every profiler, coverage tool, and debugger ever expects
> lines to be natural numbers, not ranges of numbers.
> A lot of tooling would need to be changed.
>
> >
> > I think that question applies to later sections as well.  Anywhere we
> > refer to a "line", it could actually mean a span of lines. (especially
> > when you consider \ continuation in situations you might not otherwise
> > think could span lines)
>
> Let's take an example:
> ```
> x = (
>  a,
>  b,
> )
> ```
>
> You would want the BUILD_TUPLE instruction to have a of span lines 1 to
> 4 (inclusive), rather just line 1?
> If you wanted to break on the BUILD_TUPLE where you tell pdb to break?
>
> I don't see that it would add much value, but it would add a lot of
> complexity.
>

We should have the data about the range at bytecode compilation time,
correct?  So why not keep it?  sure, most existing tooling would just use
the start of the range as the line number as it always has.  but some
tooling could find the range useful (ex: semantic code indexing for use in
display, search, editors, IDEs. Rendering lint errors more accurately
instead of just claiming a single line or resorting to parsing hacks to
come up with a range, etc.).  The downside is that we'd be storing a second
number in bytecode making it slightly larger.  Though it could be stored
efficiently as a prefixed delta so it'd likely average out as less than 2
bytes per line number stored.  (i don't have a feeling for our current
format to know if that is significant or not - if it is, maybe this idea
just gets nixed)

The reason the range concept was on my mind is due to something not quite
related but involving a changed idea of a line number in our current system
that we recently ran into with pytype during a Python upgrade.

"""in 3.7, if a function body is a plain docstring, the line number of the
RETURN_VALUE opcode corresponds to the docstring, whereas in 3.6 it
corresponds to the function definition.""" (Thanks, Martin & Rebecca!)

```python
def no_op():
  """docstring instead of pass."""
```

so the location of what *was* originally an end of line `# pytype:
disable=bad-return-type` comment (to work around an issue not relevant
here) turned awkward and version dependent.  pytype is bytecode based, thus
that is where its line numbers come from.  metadata comments in source can
only be tied to bytecode via line numbers.  making end of line directives
occasionally hard to match up.

When there is no return statement, this opcode still exists.  what line
number does it belong to?  3.6's answer made sense to me.  3.7's seems
wrong - a docstring isn't responsible for a return opcode.  I didn't check
what 3.8 and 3.9 do.  An alternate answer after this PEP is that it
wouldn't have a line number when there is no return statement (pedantically
correct, I approve! #win).

-gps


>
> Cheers,
> Mark.
>
> >
> > -gps
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/H3YBK275SUSCR5EHWHYBTJBF655UK7JG/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: PEP 626: Precise line numbers for debugging and other tools.

2020-07-21 Thread Gregory P. Smith
On Tue, Jul 21, 2020 at 1:39 PM Gregory P. Smith  wrote:

>
> On Fri, Jul 17, 2020 at 10:41 AM Pablo Galindo Salgado <
> pablog...@gmail.com> wrote:
>
>> I like the proposal in general but I am against removing lnotab. The
>> reason is that many tools rely on reading this attribute to figure out the
>> Python call stack information. For instance, many sampler profilers read
>> this memory by using ptrace or process_vm_readv and they cannot execute any
>> code on the process under tracing as that would be a security issue. If we
>> remove a 'static' view of that information, it will impact negatively the
>> current set of remote process analysis tools. The proposed new way of
>> retrieving the line number will rely (if we deprecate and remove lnotab) on
>> executing code, making it much more difficult for the ecosystem of
>> profilers and remote process analysis tools to do their job.
>>
>
> +1 agreed.
>
> """Some care must be taken not to break existing tooling. To minimize
> breakage, the co_lnotab attribute will be retained, but lazily generated on
> demand.""" - https://www.python.org/dev/peps/pep-0626/#id4
>
> This breaks existing tooling.
>

"The co_linetable attribute will hold the line number information. The
format is opaque, unspecified and may be changed without notice."
...
"Tools that parse the co_lnotab table should move to using the new
co_lines() method as soon as is practical."

Given it is impossible for tools doing passive inspection of Python VM
instances to execute code, co_linetable's exact format will be depended on
just as co_lnotab was.  co_lnotab was only quasi-"officially" documented in
the Python docs, it's spec lives in
https://github.com/python/cpython/blob/master/Objects/lnotab_notes.txt (pointed
to by a couple module's docs). The lnotab format "changed" once, in 3.6, an
unsigned delta was changed to signed (but I don't believe anything beyond
some experiments ever actually used negatives?).

How about creating something defined and always present for once given the
need has been demonstrated.  Even if we don't, it will be used, and we will
be unable to change it within a release.

-gps


> -gps
>
>
>> --
>>
>> Pablo
>>
>> On Fri, 17 Jul 2020, 15:55 Mark Shannon,  wrote:
>>
>>> Hi all,
>>>
>>> I'd like to announce a new PEP.
>>>
>>> It is mainly codifying that Python should do what you probably already
>>> thought it did :)
>>>
>>> Should be uncontroversial, but all comments are welcome.
>>>
>>> Cheers,
>>> Mark.
>>> ___
>>> Python-Dev mailing list -- python-dev@python.org
>>> To unsubscribe send an email to python-dev-le...@python.org
>>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>>> Message archived at
>>> https://mail.python.org/archives/list/python-dev@python.org/message/BMX32UARJFY3PZZYKRANS6RCMR2XBVVM/
>>> Code of Conduct: http://python.org/psf/codeofconduct/
>>>
>> ___
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-le...@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-dev@python.org/message/57OXMUBV5FAEFXULRBCRAHEF7Q5GP6QT/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/HFQKDM4TVJPNHHHIJN3BGU2N3CRRXNQY/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: PEP 626: Precise line numbers for debugging and other tools.

2020-07-21 Thread Gregory P. Smith
On Fri, Jul 17, 2020 at 8:41 AM Ned Batchelder 
wrote:

> https://www.python.org/dev/peps/pep-0626/ :)
>
> --Ned.
>
> On 7/17/20 10:48 AM, Mark Shannon wrote:
> > Hi all,
> >
> > I'd like to announce a new PEP.
> >
> > It is mainly codifying that Python should do what you probably already
> > thought it did :)
> >
> > Should be uncontroversial, but all comments are welcome.
> >
> > Cheers,
> > Mark.
>
>
"""When a frame object is created, the f_lineno will be set to the line at
which the function or class is defined. For modules it will be set to
zero."""

Within this PEP it'd be good for us to be very pedantic.  f_lineno is a
single number.  So which number is it given many class and function
definition statements can span multiple lines.

Is it the line containing the class or def keyword?  Or is it the line
containing the trailing :?

Q: Why can't we have the information about the entire span of lines rather
than consider a definition to be a "line"?

I think that question applies to later sections as well.  Anywhere we refer
to a "line", it could actually mean a span of lines. (especially when you
consider \ continuation in situations you might not otherwise think could
span lines)

-gps
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/F6TQIXELOJWBUDSO5MDW3X5RTQXO6EI3/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: PEP 626: Precise line numbers for debugging and other tools.

2020-07-21 Thread Gregory P. Smith
On Fri, Jul 17, 2020 at 10:41 AM Pablo Galindo Salgado 
wrote:

> I like the proposal in general but I am against removing lnotab. The
> reason is that many tools rely on reading this attribute to figure out the
> Python call stack information. For instance, many sampler profilers read
> this memory by using ptrace or process_vm_readv and they cannot execute any
> code on the process under tracing as that would be a security issue. If we
> remove a 'static' view of that information, it will impact negatively the
> current set of remote process analysis tools. The proposed new way of
> retrieving the line number will rely (if we deprecate and remove lnotab) on
> executing code, making it much more difficult for the ecosystem of
> profilers and remote process analysis tools to do their job.
>

+1 agreed.

"""Some care must be taken not to break existing tooling. To minimize
breakage, the co_lnotab attribute will be retained, but lazily generated on
demand.""" - https://www.python.org/dev/peps/pep-0626/#id4

This breaks existing tooling.

-gps


> --
>
> Pablo
>
> On Fri, 17 Jul 2020, 15:55 Mark Shannon,  wrote:
>
>> Hi all,
>>
>> I'd like to announce a new PEP.
>>
>> It is mainly codifying that Python should do what you probably already
>> thought it did :)
>>
>> Should be uncontroversial, but all comments are welcome.
>>
>> Cheers,
>> Mark.
>> ___
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-le...@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-dev@python.org/message/BMX32UARJFY3PZZYKRANS6RCMR2XBVVM/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/57OXMUBV5FAEFXULRBCRAHEF7Q5GP6QT/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/NCW4PCOINV7HYUHND7EQ2GUWR22OZDXF/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: PEP 622: Structural Pattern Matching

2020-06-26 Thread Gregory P. Smith
On Fri, Jun 26, 2020 at 6:42 AM Mark Shannon  wrote:

>
> > Let us start from some anecdotal evidence: isinstance() is one of the
> most called functions in large scale Python code-bases (by static call
> count). In particular, when analyzing some multi-million line production
> code base, it was discovered that isinstance() is the second most called
> builtin function (after len()). Even taking into account builtin classes,
> it is still in the top ten. Most of such calls are followed by specific
> attribute access.
>
> Why use anecdotal evidence? I don't doubt the numbers, but it would be
> better to use the standard library, or the top N most popular packages
> from GitHub.
>

Agreed.  This anecdote felt off to me and made for a bad introductory
feeling.  I know enough of who is involved to read it as likely "within the
internal Dropbox code base we found isinstance() to be the second most
called built-in by static call counts".  It'd be better worded as such
instead of left opaque if you are going to use this example at all.  [but
read on below, i'm not sure the anecdotal evidence is even relevant to
state]

Also if using this, please include text explaining what "static call count
means".  Was that "number of grep 'isinstance[(]' matches in all .py files
which we reasonably assume are calls"?  Or was that "measuring a running
application and counting cumulative calls of every built-in for the
lifetime of the large application"?  Include a footnote of if you have you
removed all use of six and py2->py3-isms?  Both six and manual py2->3
porting often wound up adding isinstance in places where they'll rightfully
be refactored out when cleaning up the py2 dead code legacy becomes anyones
priority.

A very rough grep of our much larger Python codebase within Google shows
isinstance *call site counts* to likely be lower than int or len and
similar to print.  With a notable percentage of isinstance usage clearly
related to py2 -> py3 compatibility, suggesting many can now go away.  I'm
not going to spend much time looking further as I don't think actual
numbers matter:  *Confirmed, isinstance gets used a lot.*  We can simply
state that as a truth and move on without needing a lot of justification.

>
> > There are two possible conclusions that can be drawn from this
> information:
> >
> > Handling of heterogeneous data (i.e. situations where a variable can
> take values of multiple types) is common in real world code.
> > Python doesn't have expressive ways of destructuring object data
> (i.e. separating the content of an object into multiple variables).
>
> I don't see how the second conclusion can be drawn.
> How does the prevalence of `isinstance()` suggest that Python doesn't
> have expressive ways of destructuring object data?

...

> >
> > We believe this will improve both readability and reliability of
> relevant code. To illustrate the readability improvement, let us consider
> an actual example from the Python standard library:
> >
> > def is_tuple(node):
> > if isinstance(node, Node) and node.children == [LParen(), RParen()]:
> > return True
> > return (isinstance(node, Node)
> > and len(node.children) == 3
> > and isinstance(node.children[0], Leaf)
> > and isinstance(node.children[1], Node)
> > and isinstance(node.children[2], Leaf)
> > and node.children[0].value == "("
> > and node.children[2].value == ")")
> >
>
> Just one example?
> The PEP needs to show that this sort of pattern is widespread.
>

Agreed.  I don't find application code following this pattern to be
common.  Yes it exists, but I would not expect to encounter it frequently
if I were doing random people's Python code reviews.

The supplied "stdlib" code example is lib2to3.fixer_util.is_tuple.  Using
that as an example of code "in the standard library" is *technically*
correct. But lib2to3 is an undocumented deprecated library that we have
slated for removal.  That makes it a bit weak to cite.

Better practical examples don't have to be within the stdlib.

Randomly perusing some projects I know that I expect to have such
constructs, here's a possible example:
https://github.com/PyCQA/pylint/blob/master/pylint/checkers/logging.py#L231.

There are also code patterns in pytype such as
https://github.com/google/pytype/blob/master/pytype/vm.py#L480  and
https://github.com/google/pytype/blob/master/pytype/vm.py#L1088 that might
make sense.

Though I realize you were probably in search of a simple one for the PEP in
order to write a before and after example.

-gps
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/X7XWT4YZFRFJJYJFHZY6X5LYLBZ7LH52/
Code of Conduct: 

[Python-Dev] Re: PEP 622: Structural Pattern Matching

2020-06-25 Thread Gregory P. Smith
Litmus test: Give someone who does not know Python this code example from
the PEP and ask them what it does and why it does what it does:

match get_shape():
case Line(start := Point(x, y), end) if start == end:
print(f"Zero length line at {x}, {y}")

I expect confusion to be the result.  If they don't blindly assume the
variables come from somewhere not shown to stop their anguish.

With Python experience, my own reading is:
 * I see start actually being assigned.
 * I see nothing giving values to end, x, or y.
 * Line and Point are things being called, probably class constructions due
to being Capitalized.
 * But where did the parameter values come from and why and how can end be
referred to in a conditional when it doesn't exist yet?
   They appear to be magic!

Did get_shape() return these? (i think not).  Something magic and *implicit
rather than explicit* happens in later lines.  The opposite of what Python
is known for.

Where's the pseudo-code describing *exactly* what the above looks like
logically speaking? (there's a TODO in the PEP for the __match__ protocol
code so I assume it will come, thanks!).  I can guess _only_ after reading
a bunch of these discussions and bits of the PEP.  Is it this?  I can't
tell.

shape = get_shape()
values_or_none = Line.__match__(shape)
if values_or_none:
  start, end = values_or_none
  if start == end:
if x, y := Point.__match__(shape):
  print(...)
  del x, y
  else:
print(...)
  del start, end
else:
  # ... onto the next case: ?

Someone unfamiliar with Python wouldn't even have a chance of seeing that.
I had to rewrite the above many times, I'm probably still wrong.

That sample is very confusing code.  It makes me lean -1 on the PEP overall
today.

This syntax does not lead to readable logically understandable code.  I
wouldn't encourage anyone to write code that way because it is not
understandable to others.  We must never assume others are experts in the
language they are working on code in if we want it to be maintainable.  I
wouldn't approve a code review containing that example.

It would help *in part* if ()s were not used to invoke the __match__
protocol.  I think a couple others also mentioned this earlier.  Don't make
it look like a call.  Use different tokens than ().  Point{x, y} for
example.  Or some way to use another token unused in that context in our
toolbook such as @ to signify "get a matcher for this class" instead of
"construct this class".  for example ClassName@() as our match protocol
indicator, shown here with explicit assignments for clarity:

match get_shape() as shape:
  case start, end := Line@(shape):

no implicit assignments, it is clear where everything comes from.  it is
clear it isn't a constructor call.

downside?  possibly a painful bug when someone forgets to type the @.  but
the point of it not being construction needs to be made.  not using ()s but
instead using ClassName@{} or just ClassName{} would prevent that.

The more nested things get with sub-patterns, the worse the confusion
becomes.  The nesting sounds powerful but is frankly something I'd want to
forbid anyone from using when the assignment consequences are implicit.  So
why implement sub-patterns at all?  All I see right now is pain.

-gps
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/B4OXI6CZTSNC6LHWRAKT4XVFFDIUR4K3/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: The Anti-PEP

2020-06-25 Thread Gregory P. Smith
On Thu, Jun 25, 2020 at 6:49 PM Raymond Hettinger <
raymond.hettin...@gmail.com> wrote:

> > it is hard to make a decision between the pros and cons,
> > when the pros are in a single formal document and the
> > cons are scattered across the internet.
>
> Mark, I support your idea.  It is natural for PEP authors to not fully
> articulate the voices of opposition or counter-proposals.
> The current process doesn't make it likely that a balanced document is
> created for decision making purposes.
>
>
On some PEPs in the past I seem to recall we've had the PEP author, or at
least editor after the initial draft kicked things off _not_ be among those
invested in seeing the PEP be approved.

Or maybe I'm conflating the old role of the PEP delegate with the editor?

Regardless i don't see how an anti-pep would work much better, but I also
don't see anything stopping anyone from trying one.  I worry that it'll
fragment conversation even more and separate discussions so that everyone
is even more confused about overall opinion tallies?  one way to find out...

-gps
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JYJ2GB2LKX7ELWKURAQMOA7Z52DHE3B6/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: PEP 622: Structural Pattern Matching

2020-06-25 Thread Gregory P. Smith
On Wed, Jun 24, 2020 at 7:58 PM Stephen J. Turnbull <
turnbull.stephen...@u.tsukuba.ac.jp> wrote:

> Ethan Furman writes:
>
>  > _ does not bind to anything, but of what practical importance is that?
>
> *sigh* English speakers ... mutter ... mutter ... *long sigh*
>
> It's absolutely essential to the use of the identifier "_", otherwise
> the I18N community would riot in the streets of Pittsburgh.  Not good
> TV for Python (and if Python isn't the best TV, what good is it? ;-)
>
>
Can I use an i18n'd _("string") within a case without jumping through hoops
to assign it to a name before the match:?


> Steve
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/ENUPVNZEW7DFJKVCKK5XQ4NLV3KOW36C/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/WPPCVM6YPAYSGA7AJ5YOKGLQQNGXFLF4/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: PEP 622: Structural Pattern Matching

2020-06-25 Thread Gregory P. Smith
On Wed, Jun 24, 2020 at 7:34 PM Taine Zhao  wrote:

> > e.g., "or", and then I wonder "what does short-circuiting have to do
> > with it?". All reuse of symbols carries baggage.
>
> "or" brings an intuition of the execution order of pattern matching, just
> like how people already know about "short-circuiting".
>
> "or" 's operator precedence also suggests the syntax of OR patterns.
>
> As we have "|"  as an existing operator, it seems that there might be
> cases that the precedence of "|" is not consistent with it in an
> expression. This will mislead users.
>

I prefer "or" to "|" as a combining token as there is nothing bitwise going
on here.  "or" reads better.   Which is why Python used it for logic
operations in the first place.  It is simple English.  "|" does not read
like or to anyone but a C based language programmer.  Something Python
users should never need to know.

There is no existing pythonic way to write "evaluate all of these things at
once in no specific order".

And in reality, there will be an order. It'll be sequential, and if it
isn't the left to right order that things are written with the "|" between
them, it will break someones assumptions and make optimization harder.
Some too-clever-for-the-worlds-own-good author is going to implement
__match__ classmethods that have side effects and make state changes that
impact the behavior of later matcher calls (no sympathy for them). Someone
else is going to order them most likely to least likely for performance (we
should have sympathy for that).

Given we only propose to allow a single trailing guard if per case, using
"or" instead of "|" won't be confused with an if's guard condition.

-gps
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/2C2S4EDIFGYI6AW3K6VVU2QKEPLASZJT/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: PEP 622: Structural Pattern Matching

2020-06-25 Thread Gregory P. Smith
On Wed, Jun 24, 2020 at 1:38 PM Luciano Ramalho  wrote:

> Thank you Guido, Brandt Bucher, Tobias Kohn, Ivan Levkivskyi and Talin
> for this fun and very useful new feature.
>
> I do enjoy pattern matching a lot in Elixir—my favorite language these
> days, after Python.
>
> I don't want to start a discussion, but I just want to say that as an
> instructor I fear this core language addition may make the language
> less approachable to the non-IT professionals, researchers etc. who
> have saved Python from the decline that we can observe happening in
> Ruby—a language of similar age, with similar strengths and weaknesses,
> but never widely adopted outside of the IT profession.
>
> After I wrap up Fluent Python 2e (which is aimed at professional
> developers) I hope I can find the time to tackle the challenge of
> creating introductory Python content that manages to explain pattern
> matching and other recent developments in a way that is accessible to
> all.
>

Hold on a while.  This feature does not exist.  This PEP has not been
accepted.  Don't count your chickens.py before they hatch.

The last P in PEP stands for Proposal for a reason.  Rejection is a
perfectly valid option.  As is a significant reworking of the proposal so
that it doesn't turn into a nightmare.  This one needs a lot of work at a
minimum.  As proposed, it is extremely non-approachable and non-trivial.

-gps
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/GUQLG3H5REWRB5ZINAISX7ZCS3DJQOED/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: New optional dependency

2020-06-23 Thread Gregory P. Smith
On Tue, Jun 23, 2020 at 7:41 AM Elisha Paine  wrote:

> Hi all,
>
> I am looking at getting TkDND support into tkinter, and have created
> issue40893. However, we are currently considering the practicalities of
> adding a new optional dependency to Python and I was hoping someone
> could help answer the question of: is there a procedure for this?
>
> The problem is that third-party distributors need to be notified of the
> change and edit the package details accordingly. The current idea is
> just to make it very clear on the “what’s new” page, however this would
> not guarantee it would be seen, so I am very open to ideas/opinions.
>
> Thanks,
> Elisha
>

What's New documentation combined with a configure.ac check to
conditionally compile it only when available is sufficient.  (once that
works, also ensuring that the dependency is added to the windows and mac
build setups so our binary builds include it)

It is on the third party distributors to pay attention.  If that worries
you, file your own issues in the debian and fedora trackers and that should
do it.

-gps
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/DE7NIJATCRMQEQOUNWGSEIAL3T5GSWO5/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Can we stop adding to the C API, please?

2020-06-03 Thread Gregory P. Smith
On Wed, Jun 3, 2020 at 2:13 PM Victor Stinner  wrote:

> Le mer. 3 juin 2020 à 19:17, Mark Shannon  a écrit :
> > > I also *added* a bunch of *new* "getter" or "setter" functions to the
> > > public C API for my project of hiding implementation details, like
> > > making structures opaque:
> > > https://docs.python.org/dev/whatsnew/3.9.html#id1
> >
> > Adding "setters" is generally a bad idea.
> > "getters" can be computed if the underlying field disappears, but the
> > same may not be true for setters if the relation is not one-to-one.
> > I don't think there are any new setters in 3.9, so it's not an immediate
> > problem.
>
> You're making the assumption that the member can be set directly. But
> my plan is to make the structure opaque. In that case, you need
> getters and setters for all fields you would like to access. No member
> would be accessible directly anymore.
>
> > `PyThreadState_GetInterpreter()` can't replace `tstate->interp` for two
> > reasons.
> > 1. There is no way to stop third party C code accessing the internals of
> > data structures. We can warn them not to, but that's all.
> > 2. The internal layout of C structures has never been part of the API,
> > with arguably two exceptions; the PyTypeObject struct and the
> > `ob_refcnt` field of PyObject.
>
> My long term plan is to make all structures opaque :-) So far,
> PyInterpreterState structure was made opaque in Python 3.7. It helped
> *a lot* the development of Python 3.8 and 3.9, especially for
> subinterpreters. And I made PyGC_Head opaque in Python 3.9.
>
> Examples of issues to make structures opaque:
>
> PyGC_Head: https://bugs.python.org/issue40241 (done in Python 3.9)
> PyObject: https://bugs.python.org/issue39573
> PyTypeObject: https://bugs.python.org/issue40170
> PyThreadState: https://bugs.python.org/issue39947
> PyInterpreterState: https://bugs.python.org/issue35886 (done in Python
> 3.8)
>
> For the short term, my plan is to make structure opaque in the limited
> C API, before breaking more stuff in the public C API :-)
>

Indeed, your plan and the work you've been doing and discussing with other
core devs about this (including at multiple sprints and summits) over the
past 4+ years is the right one.  Our reliance on structs and related cpp
macros unfortunately exposed as public is a burden that freezes reasonable
CPython VM implementation evolution options.  This work moves us away from
that into a better place one step at a time without mass disruption.

More prior references related to this work are critical reading and should
not be overlooked:

[2017 "Keeping Python Competitive"] https://lwn.net/Articles/723949/
[2018 "Lets change the C API" thread]
https://mail.python.org/archives/list/python-dev@python.org/thread/B67MYCAO4H4AJNMLSWVT3UVFTHSDGQRB/#B67MYCAO4H4AJNMLSWVT3UVFTHSDGQRB
[2019 "The C API"]
https://pyfound.blogspot.com/2019/06/python-language-summit-lightning-talks-part-2.html
[2020-04 "PEP: Modify the C API to hide implementation details" thread -
with a lot of links to much earlier 2017 and such references]
https://mail.python.org/archives/list/python-dev@python.org/thread/HKM774XKU7DPJNLUTYHUB5U6VR6EQMJF/#HKM774XKU7DPJNLUTYHUB5U6VR6EQMJF
and Victors overall https://pythoncapi.readthedocs.io/roadmap.html as
referenced a few places in those.

It is also worth paying attention to the
https://mail.python.org/archives/list/capi-...@python.org/latest mailing
list for anyone with a CPython C API interest.

-gps


>
> > PyObject_CallNoArgs() seems harmless.
> > Rationalizing the call API has merit, but PyObject_CallNoArgs()
> > leads to PyObject_CallOneArg(), PyObject_CallTwoArgs(), etc. and an even
> > larger API.
>
> PyObject_CallOneArg() also exists:
> https://docs.python.org/dev/c-api/call.html#c.PyObject_CallOneArg
>
> It was added as a private function https://bugs.python.org/issue37483
> add made public in commit 3f563cea567fbfed9db539ecbbacfee2d86f7735
> "bpo-39245: Make Vectorcall C API public (GH-17893)".
>
> But it's missing in What's New in Python 3.9.
>
> There is no plan for two or more arguments.
>
>
> > PyObject_GC_IsTracked(). I don't like this.
> > Shouldn't GC track *all* objects?
> > Even if it were named PyObject_Cycle_GC_IsTracked() it would be exposing
> > internal implementation details for no good reason. A cycle GC that
> > doesn't "track" individual objects, but treats all objects the same
> > could be more efficient. In which case, what would this mean?
> >
> > What is the purpose of PyObject_GC_IsFinalized()?
> > Third party objects can easily tell if they have been finalized.
> > Why they would ever need this information is a mystery to me.
>
> Did you read the issues which added these functions to see the
> rationale? https://bugs.python.org/issue40241
>
> I like the "(Contributed by xxx in bpo-xxx.)" in What's New in Python
> 3.9: it became trivial to find such rationale.
>
> Victor
> --
> Night gathers, and now my watch begins. It shall not end until my death.
> 

[Python-Dev] Re: Map errno==ETIME to TimeoutError

2020-05-24 Thread Gregory P. Smith
Sounds like a natural fit, I'd just do it for 3.10.

On Sun, May 24, 2020, 9:45 AM Eric V. Smith  wrote:

> Does anyone have an opinion on https://bugs.python.org/issue39673? It
> maps ETIME to TimeoutError, in addition to the already existing ETIMEDOUT.
>
> http://man7.org/linux/man-pages/man3/errno.3.html says:
>
>*ETIME   *Timer expired (POSIX.1 (XSI STREAMS option)).
>
>(POSIX.1 says "STREAM ioctl(2) 
>  timeout".)
>
>*ETIMEDOUT   *Connection timed out (POSIX.1-2001).
>
>
> It seems like a reasonable change to me, but I'm not a subject matter
> expert on STREAMS, or what other affect this might have.
>
> And if added to 3.10, should it be backported? I'd tend to say "no",
> because of unknown impacts on existing code.
>
> Eric
>
>  ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/C7KK6VSGPQKPA5IUCZ2MHH7QNLP2Q5QX/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/KF66ZEH55RXJSMTNP5B37NS2O2WY6AGZ/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: PEP 618: Add Optional Length-Checking To zip

2020-05-20 Thread Gregory P. Smith
On Wed, May 20, 2020 at 11:09 AM Jim J. Jewett  wrote:

> David Mertz wrote:
>
> > Fwiw, I don't think it changes my order, but 'strict' is a better word
> than
> > 'equal' in all those places. I'd subtract 0.1 from each of those votes if
> > they used "equal".
>
> I would say that 'equal' is worse than 'strict'. but 'strict' is also
> wrong.
>
> Zipping to a potentially infinite sequence -- like a manual enumerate --
> isn't wrong.  It may be the less common case, but it isn't wrong.  Using
> 'strict' implies that there is something sloppy about the data in, for
> example, cases like Stephen J. Turnbull's lagged time series.
>
> Unfortunately, the best I can come up with is 'same_length', or possibly
> 'equal_len' or 'equal_length'.  While those are better semantically, they
> are also slightly too long or awkward.  I would personally still consider
> 'same_length' the least bad option.
>
> -jJ
>

As we've come down to naming things... if you want it to read more like
English,

`zip(vorpal_rabbits, holy_hand_grenades, lengths_must_match=True)`

or another chosen variation of that such as `len_must_match=` or
`length_must_match=` reads nicely and is pretty self explanatory that an
error can be expected if the condition implied by the "must" is found
untrue without really feeling a need to look it up in documentation.

It is also harder to type or fit on a line.  Which is one advantage to a
short thing like `strict=`.

I don't care so much about the particular spelling here to argue among any
of those, I primarily want the feature to exist.

I expect we're entering steering council territory for a decision soon...

-gps

___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/G5JTZI6SARFG5AA3H2XJUIUNXR2TBNQ4/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TGLTTQ7PKYSWYBYVKE6PHUH5RLJSDUH5/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: PEP 618: Add Optional Length-Checking To zip

2020-05-15 Thread Gregory P. Smith
On Fri, May 15, 2020 at 11:55 AM Henk-Jaap Wagenaar <
wagenaarhenkj...@gmail.com> wrote:

> I'll join in with the fun...
>
> zip(strict=True) +1
> itertools.zip_strict() +0
>

Agreed.  The best way to reduce accidental incorrect use of the builtin is
to make the builtin capable of doing what a people want directly without
having to go discover something in a module somewhere.

OTOH so long as zip's docstring that shows up to people from interactive
help, pydoc, and hover/alt/meta text in fancier IDEs mentions the caveat
and the way to get the strict behavior, either of these two should be
sufficient.  I'm pushing forward with a pure documentation update in
https://github.com/python/cpython/pull/20118 suitable for 3.8 - it doesn't
mention the way to get the other behavior as that isn't settled or short
yet, just makes it more obvious what the actual behavior is.

-gps
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZWQJJXHQTHAVIQXU4ZJWO2QUFAWFA6ZV/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: PoC: Subinterpreters 4x faster than sequential execution or threads on CPU-bound workaround

2020-05-07 Thread Gregory P. Smith
On Wed, May 6, 2020 at 1:14 PM Serhiy Storchaka  wrote:

> 06.05.20 00:46, Victor Stinner пише:
> > Subinterpreters and multiprocessing have basically the same speed on
> > this benchmark.
>
> It does not look like there are some advantages of subinterpreters
> against multiprocessing.
>

There is not an implementation worthy of comparison at this point, no.  I
don't believe meaningful conclusions of that comparative nature can be
drawn from the current work.  We shouldn't be blocking any decision on
reducing our existing tech debt around subinterpreters on a viable
multi-core solution existing.  There are benchmarks I could propose that I
predict would show a different result even today but I'm refraining because
I believe such things to be a distraction.

I am wondering how much 3.9 will be slower than 3.8 in single-thread
> single-interpreter mode after getting rid of all process-wide singletons
> and caches (Py_None, Py_True, Py_NonImplemented. small integers,
> strings, tuples, _Py_IDENTIFIER, _PyArg_Parser, etc). Not mentioning
> breaking binary compatibility.
>

I'm not worried, because it won't happen in 3.9.  :)  Nobody is seriously
proposing that that be done in that manner.

The existing example work Victor did here (thanks!) was a rapid prototype
where the easiest approach to getting _something_ running parallel as a
demo was just to disable a bunch of shared global things instead of also
doing much larger work to make those per-interpreter.

That isn't how we'd likely ever actually land this kind of change.

Longer term we need to aim to get rid of process global state by moving
that into per-interpreter state.  No matter what.  This isn't something
only needed by subinterpreters.  Corralling everything into a
per-interpreter state with proper initialization and finalization
everywhere allows other nice things like multiple independent interpreters
in a process.  Even sequentially (spin up, tear down, spin up, tear down,
repeat...).  We cannot reliably do that today without side effects such as
duplicate initializations and resulting resource leaks or worse.  Even if
such per-interpreter state instead of per-process state isolation is never
used for parallel execution, I still want to see it happen.

Python already loses out to Lua because of this.  Lua is easily embedded in
a self-contained fashion.  CPython has never been.  This kind of work helps
open up that world instead of relegating us to only
single life-of-the-process long lived language VM uses that we can serve
today.

-gps
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BUIRTWX3FTXHFQUDAJZ5VHFG6ND3QT4U/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Remove ctypes from uuid

2020-05-04 Thread Gregory P. Smith
On Mon, May 4, 2020 at 9:45 AM Steve Dower  wrote:

> For those who haven't looked in a while, the uuid module uses ctypes to
> look up libuuid for uuid_generate_time_safe() and uuid_generate_time()
> functions.
>
> I've run into scenarios where I need to remove this from our own builds,
> but it seems like it's probably unnecessary anyway? It's certainly a
> security risk, though in most cases the _uuid module should provide them
> anyway.
>
> I'm proposing to remove the ctypes fallbacks:
> https://bugs.python.org/issue40501
>
> If anyone knows that they are reliant on not having libuuid at compile
> time, but being able to load it later via ctypes, it would be great if
> you could drop by the issue and explain the scenario.
>
> Thanks,
> Steve
>

That seems like a good idea.  The ctypes code has been in there since the
module was added in 2006.  The _uuid extension module only landed in 2017
for Python 3.7.

I can't imagine a normal scenario where the _uuid extension module would
not exist unless we're talking maybe special needs for PyPy.  (I doubt they
want ctypes either these days)

-gps
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LW5CZQZ3RW5EDNKWRBZVPJHWPPZ6WLFU/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: PEP 554 for 3.9 or 3.10?

2020-04-29 Thread Gregory P. Smith
On Wed, Apr 29, 2020 at 5:40 AM Julien Salort  wrote:

> Le 29/04/2020 à 03:18, Eric Snow a écrit :
>
> > My (honest) question is, how many folks using subinterpreters are
> > going to want to use numpy (or module X) enough to get mad about it
> > before the extension supports subinterpreters?  What will user
> > expectations be when it comes to subinterpreters?
> >
> > We will make the docs as clear as we can, but there are plenty of
> > users out there that will not pay enough attention to know that most
> > extension modules will not support subinterpreters at first.  Is there
> > anything we can do to mitigate this impact?  How much would it help if
> > the ImportError for incompatible modules give a clear (though
> > lengthier) explanation of the situation?
>
> For what it's worth, I can give you the feedback of a simple user. It
> happens that I tried some time ago to use Numpy in a Flask project which
> was deployed with mod_wsgi on an Apache server. Basically, the page was
> dynamically generating some plots. And I got weird unreliable behaviour,
> which took me some time to debug.
>
> I had to look it up on the internet to figure out the problem was that
> Numpy cannot reliably work with mod_wsgi. I originally thought that I
> had made a mistake somewhere in my code instead. So, I rewrote the code
> to remove the dependency on Numpy. I had used Numpy in the first place,
> because, as a physicist, this is what I am used to, but it was clearly
> very possible to rewrite this particular code without Numpy.
>
> If your proposal leads to an intelligible actual error, and a clear
> warning in the documentation, instead of a silent crash, this sounds
> like progress, even for those packages which won't work on
> subinterpreters anytime soon...
>
>
+10 to this, the mysterious failures of today are way worse than a clear
"this module doesn't support this execution environment" ImportError.

I'm not worried at all about someone going and filing an issue in a project
saying "please support this execution environment".  Someone may eventually
come along and decide they want to be the one to make that happen and do
the work because they have a reason.

-gps
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YMHYG7B6U3ACMOSB3A3A7VJXQA5AMJL5/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: PEP 617: New PEG parser for CPython

2020-04-21 Thread Gregory P. Smith
On Tue, Apr 21, 2020 at 9:35 PM Gregory P. Smith  wrote:

> Could we go ahead and mark lib2to3 as Pending Deprecation in 3.9 so we can
> get it out of the stdlib by 3.11 or 3.12?
>

I'm going ahead and tracking the idea in https://bugs.python.org/issue40360.


>
> lib2to3 is the basis of all sorts of general source code manipulation
> tooling.  Its name and original reason d'etre have moved on.  It is
> actively used to parse and rewrite Python 3 code all the time.  yapf uses
> it, black uses a fork of it.  Other Python code manipulation tooling uses
> it.  Modernize like fixers are useful for all sorts of cleanups.
>
> IMNSHO it would be better if lib2to3 were *not* in the stdlib anymore -
> Black already chose to fork lib2to3
> <https://github.com/psf/black/tree/master/blib2to3>.  So given that it is
> eventually not going to be able to parse future syntax, the better answer
> seems like deprecation, putting the final version up on PyPI and letting
> any descendants of it live on PyPI where they can get more active care than
> a stdlib module ever does.
>
> -gps
>
>
> On Tue, Apr 21, 2020 at 6:58 PM Guido van Rossum  wrote:
>
>> Great! Please submit a PR to update the [lib]2to3 docs and CC me
>> (@gvanrossum).
>>
>> While perhaps it wouldn't hurt if the PEP mentioned lib2to3, it was just
>> accepted by the Steering Council without such language, and I wouldn't want
>> to imply that the SC agrees with everything I said. So I still think we
>> ought to deal with lib2to3 independently (and no, it won't need its own PEP
>> :-). A reasonable option would be to just deprecate it and recommend people
>> use parso, LibCST or something else (I wouldn't recommend pegen in its
>> current form yet).
>>
>> On Tue, Apr 21, 2020 at 6:21 PM Carl Meyer  wrote:
>>
>>> On Sat, Apr 18, 2020 at 10:38 PM Guido van Rossum 
>>> wrote:
>>> >
>>> > Note that, while there is indeed a docs page about 2to3, the only docs
>>> for lib2to3 in the standard library reference are a link to the source code
>>> and a single "Note: The lib2to3 API should be considered unstable and may
>>> change drastically in the future."
>>> >
>>> > Fortunately,  in order to support the 2to3 application, lib2to3
>>> doesn't need to change, because the syntax of Python 2 is no longer
>>> changing. :-) Choosing to remove 2to3 is an independent decision. And
>>> lib2to3 does not depend in any way on the old parser module. (It doesn't
>>> even use the standard tokenize module, but incorporates its own version
>>> that is slightly tweaked to support Python 2.)
>>>
>>> Indeed! Thanks for clarifying, I now recall that I already knew it
>>> doesn't, but forgot.
>>>
>>> The docs page for 2to3 does currently say "lib2to3 could also be
>>> adapted to custom applications in which Python code needs to be edited
>>> automatically." Perhaps at least this sentence should be removed, and
>>> maybe also replaced with a clearer note that lib2to3 not only has an
>>> unstable API, but also should not necessarily be expected to continue
>>> to parse future Python versions, and thus building tools on top of it
>>> should be discouraged rather than recommended. (Maybe even use the
>>> word "deprecated.") Happy to submit a PR for this if you agree it's
>>> warranted.
>>>
>>> It still seems to me that it wouldn't hurt for PEP 617 itself to also
>>> mention this shift in lib2to3's effective status (from "available but
>>> no API stability guarantee" to "probably will not parse future Python
>>> versions") as one of its indirect effects.
>>>
>>> > You've mentioned a few different tools that already use different
>>> technologies: LibCST depends on parso which has a fork of pgen2, lib2to3
>>> which has the original pgen2. I wonder if this would be an opportunity to
>>> move such parsing support out of the standard library completely. There are
>>> already two versions of pegen, but neither is in the standard library:
>>> there is the original pegen repo which is where things started, and there
>>> is a fork of that code in the CPython Tools directory (not yet in the
>>> upstream repo, but see PR 19503).
>>> >
>>> > The pegen tool has two generators, one generating C code and one
>>> generating Python code. I think that the C generator is really only
>>> relevant for CPython itself: it relies on the builtin tokenizer (the one
>>> written in C, not th

[Python-Dev] Re: PEP 617: New PEG parser for CPython

2020-04-21 Thread Gregory P. Smith
Could we go ahead and mark lib2to3 as Pending Deprecation in 3.9 so we can
get it out of the stdlib by 3.11 or 3.12?

lib2to3 is the basis of all sorts of general source code manipulation
tooling.  Its name and original reason d'etre have moved on.  It is
actively used to parse and rewrite Python 3 code all the time.  yapf uses
it, black uses a fork of it.  Other Python code manipulation tooling uses
it.  Modernize like fixers are useful for all sorts of cleanups.

IMNSHO it would be better if lib2to3 were *not* in the stdlib anymore -
Black already chose to fork lib2to3
.  So given that it is
eventually not going to be able to parse future syntax, the better answer
seems like deprecation, putting the final version up on PyPI and letting
any descendants of it live on PyPI where they can get more active care than
a stdlib module ever does.

-gps


On Tue, Apr 21, 2020 at 6:58 PM Guido van Rossum  wrote:

> Great! Please submit a PR to update the [lib]2to3 docs and CC me
> (@gvanrossum).
>
> While perhaps it wouldn't hurt if the PEP mentioned lib2to3, it was just
> accepted by the Steering Council without such language, and I wouldn't want
> to imply that the SC agrees with everything I said. So I still think we
> ought to deal with lib2to3 independently (and no, it won't need its own PEP
> :-). A reasonable option would be to just deprecate it and recommend people
> use parso, LibCST or something else (I wouldn't recommend pegen in its
> current form yet).
>
> On Tue, Apr 21, 2020 at 6:21 PM Carl Meyer  wrote:
>
>> On Sat, Apr 18, 2020 at 10:38 PM Guido van Rossum 
>> wrote:
>> >
>> > Note that, while there is indeed a docs page about 2to3, the only docs
>> for lib2to3 in the standard library reference are a link to the source code
>> and a single "Note: The lib2to3 API should be considered unstable and may
>> change drastically in the future."
>> >
>> > Fortunately,  in order to support the 2to3 application, lib2to3 doesn't
>> need to change, because the syntax of Python 2 is no longer changing. :-)
>> Choosing to remove 2to3 is an independent decision. And lib2to3 does not
>> depend in any way on the old parser module. (It doesn't even use the
>> standard tokenize module, but incorporates its own version that is slightly
>> tweaked to support Python 2.)
>>
>> Indeed! Thanks for clarifying, I now recall that I already knew it
>> doesn't, but forgot.
>>
>> The docs page for 2to3 does currently say "lib2to3 could also be
>> adapted to custom applications in which Python code needs to be edited
>> automatically." Perhaps at least this sentence should be removed, and
>> maybe also replaced with a clearer note that lib2to3 not only has an
>> unstable API, but also should not necessarily be expected to continue
>> to parse future Python versions, and thus building tools on top of it
>> should be discouraged rather than recommended. (Maybe even use the
>> word "deprecated.") Happy to submit a PR for this if you agree it's
>> warranted.
>>
>> It still seems to me that it wouldn't hurt for PEP 617 itself to also
>> mention this shift in lib2to3's effective status (from "available but
>> no API stability guarantee" to "probably will not parse future Python
>> versions") as one of its indirect effects.
>>
>> > You've mentioned a few different tools that already use different
>> technologies: LibCST depends on parso which has a fork of pgen2, lib2to3
>> which has the original pgen2. I wonder if this would be an opportunity to
>> move such parsing support out of the standard library completely. There are
>> already two versions of pegen, but neither is in the standard library:
>> there is the original pegen repo which is where things started, and there
>> is a fork of that code in the CPython Tools directory (not yet in the
>> upstream repo, but see PR 19503).
>> >
>> > The pegen tool has two generators, one generating C code and one
>> generating Python code. I think that the C generator is really only
>> relevant for CPython itself: it relies on the builtin tokenizer (the one
>> written in C, not the stdlib tokenize.py) and the generated C code depends
>> on many internal APIs. In fact the C generator in the original pegen repo
>> doesn't work with Python 3.9 because those internal APIs are no longer
>> exported. (It also doesn't work with Python 3.7 or older because it makes
>> critical use of the walrus operator. :-) Also, once we started getting
>> serious about replacing the old parser, we worked exclusively on the C
>> generator in the CPython Tools directory, so the version in the original
>> pegen repo is lagging quite a bit behind (is is the Python grammar in that
>> repo). But as I said you're not gonna need it.
>> >
>> > On the other hand, the Python generator is designed to be flexible, and
>> while it defaults to using the stdlib tokenize.py tokenizer, you can easily
>> hook up your own. Putting this version in the stdlib would be a mistake,
>> because 

[Python-Dev] Re: PEP 554 for 3.9 or 3.10?

2020-04-21 Thread Gregory P. Smith
On Tue, Apr 21, 2020 at 10:49 AM Antoine Pitrou  wrote:

> On Tue, 21 Apr 2020 18:46:04 +0200
> Petr Viktorin  wrote:
> > On 2020-04-21 11:01, Antoine Pitrou wrote:
> > > On Mon, 20 Apr 2020 19:21:21 -0600
> > > Eric Snow  wrote:
> > >> Honest question: how many C extensions have process-global state that
> > >> will cause problems under subinterpreters?  In other words, how many
> > >> already break in mod_wsgi?
> > >
> > > A slightly tricky question is what happens if a PyObject survives
> > > longer than the subinterpreter that created it.
> > >
> > > For example, in PyArrow, we allow passing a Python buffer-like object
> > > as a C++ buffer to C++ APIs.  The C++ buffer could conceivably be kept
> > > around by C++ code for some time.  When the C++ buffer is destroyed,
> > > Py_DECREF() is called on the Python object (I figure that we would have
> > > to switch to the future interpreter-aware PyGILState API -- when will
> > > it be available?).
>

In this scenario given that the PyObject owning the buffer passed to C++ by
PyArrow was Py_INCREF'd, an option during Py_Finalize is to not release
anything who's refcount never made it to 0.  Which also implies leaving
enough of the interpreter state around so that Py_DECREF could be called
and trigger the free().

Basically (sub)interpreter finalization should have the right to return
failure.

Everything ever allocated by an interpreter holds a reference (implied
today) to its associated interpreter state.

-gps


> >
> >
> > > But what happens if the interpreter which created
> > > the Python object is already destroyed at this point?
> >
> > That's a good question. What happens today if someone calls Py_Finalize
> > while the buffer is still around?
> >
> > I don't think you need to treat *sub*interpreters specially.
>
> The difference is that subinterpreters may have a shorter lifetime than
> the main interpreter.  Also, we're currently using a global memory
> allocator, so simple operations may work until the process dies.


> Regards
>
> Antoine.
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/UWWO3RS3UOYCZXVFORGTJAEXXOCUQQXD/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LJJVHHK7OG6ZRISCT6G2NWBBIYDGFHVB/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Improvement to SimpleNamespace

2020-04-15 Thread Gregory P. Smith
On Wed, Apr 15, 2020 at 7:37 AM Victor Stinner  wrote:

> Le mer. 15 avr. 2020 à 05:02, Raymond Hettinger
>  a écrit :
> > I would like to make that functionality available to the JSON module (or
> just about anything else that accepts a custom dict) by adding the magic
> methods for mappings so that this works:
> >
> >  catalog = json.load(f, object_hook=SimpleNamespace)
> >
> print(catalog['clothing']['mens']['shoes']['extra_wide']['quantity'])
> # currently possible with dict()
> >  print(catalog.clothing.mens.shoes.extra_wide.quantity])
>  # proposed with SimpleNamespace()
> >  print(catalog.clothing.boys['3t'].tops.quantity
># would also be supported
>
> It seems like there are many projects on PyPI which provide such
> features. A few examples:
>
> * https://pypi.org/project/attrdict/ "mapping objects that allow their
> elements to be accessed both as keys and as attributes"
> * https://pypi.org/project/dictlib/ "Dictionary keys as object attributes"
> * https://pypi.org/project/objdict/ "An ObjDict supports all normal
> dict operations, but adds support for accessing and setting entries as
> attributes"
> * https://pypi.org/project/mydict/ "dict subclass which tries to act
> like JavaScript objects, so you can use the dot notation (d.foo) to
> access members of the object"
> * https://pypi.org/project/dpath/ "accessing and searching
> dictionaries via /slashed/paths ala xpath". Example:
> dpath.util.search(x, "a/b/[cd]")
> * etc.
>
> There are many variants:
>
> * Support nested dictionaries or not
> * "Path" as a string ala XPath using "/" or "." separator -versus-
> don't give a direct access into nested dictionaries
> * Optional default value when a key/attribute is missing
> * etc.
>
> Is there a project on PyPI which already already been battle-tested
> and gained enough popularity to consider that it's now mature enough
> to enter the stdlib?


+1  that was my first reaction as well.  We don't need this battery.

If you wanted to make it a more general thing, introduce a special operator
for it instead of overloading ".".  Ex: copy Perl and adopt -> as the
"lookup the name in a dictionary without typing quotes around the constant
value" operator.  (please don't!)

by the time we're saying "JSON should decode into a structured format" we
can also say very true things like "JSON data should be validated against a
schema" at which point we're ultimately going to wind up reinventing
https://googleapis.dev/python/protobuf/latest/google/protobuf/json_format.html
and equivalent other things.

-gps

The problem is that so far, I see no clear "winner" of a "standard"
> API. So maybe leaving such feature on PyPI is better for now. It seems
> like these projects are opinionated about what should be the "right"
> behavior :-)
>
> And yeah, there are even more advanced tools like
> https://glom.readthedocs.io/
>
> As I wrote in the issue, I would prefer to leave SimpleNamespace
> unchanged. If such feature lands into the stdlib, I would prefer it to
> be a new class.
>
> Victor
> --
> Night gathers, and now my watch begins. It shall not end until my death.
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/XDFTMWK5727ETLVNEANWYN6A3HQN6STT/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/C5AVB5Z3SQ2LLUYKU65YLSVTLCL2XFOM/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Improvement to SimpleNamespace

2020-04-15 Thread Gregory P. Smith
On Wed, Apr 15, 2020 at 12:49 PM Ivan Pozdeev via Python-Dev <
python-dev@python.org> wrote:

> First of all, be aware of the limitations of this approach (which will
> need to be clearly documented if we go this way):
>
>- It only allows valid Python identifiers -- while JSON accepts any
>sequence of Unicode characters for keys (
>http://www.ecma-international.org/publications/files/ECMA-ST/ECMA-404.pdf,
>p.5)
>- It will clash with any regular attributes of whatever type
>implements this senantic
>
> Agreed, these are all problems that lead to long term code maintenance
headaches (add a new method and a bunch of existing code mysteriously
breaks).  The dict ['key'] lookup / get('key') syntax must always remain
for access to those as a result.  this is mere syntactic sugar convenience.

Unfortunately it's a slippery slope; do this just for JSON and people will
ask why it can't be done for any dict and why it isn't the default
available everywhere.

Similar to -> in Perl.

>
>-
>- It's not very scalable and will upset static code checkers and IDEs
>as it's impossible to determine if the attribute name is valid or provide
>typing hints
>
>
Not necessarily.  If you've given the JSON a schema, you can use that to
teach tools attr names and types.  otherwise anything doing such checking
should see that this is an instance of a type with a __getattr__
implemented and thus all attributes must be deemed valid.


>-
>
> --
> Now, I think it's possible to make a class that would *wrap* an existing
> mapping.
> This would completely decouple the convenience syntax from the storage
> semantics and any specific modules and types.
>
> When writing a proof-of-concept implementation, however, I bumped into the
> need to distinguish which of the child objects are containers (thus need to
> be wrapped as well) and which are the leaves (thus need to be returned as
> is). I guess this is going to be application-specific. And the same problem
> will arise with any implementation, not only a wrapper.
> On 15.04.2020 5:59, Raymond Hettinger wrote:
>
> SimpleNamespace() is really good at giving attribute style-access. I would 
> like to make that functionality available to the JSON module (or just about 
> anything else that accepts a custom dict) by adding the magic methods for 
> mappings so that this works:
>
>  catalog = json.load(f, object_hook=SimpleNamespace)
>  print(catalog['clothing']['mens']['shoes']['extra_wide']['quantity'])
>   # currently possible with dict()
>  print(catalog.clothing.mens.shoes.extra_wide.quantity]) 
> # proposed with SimpleNamespace()
>  print(catalog.clothing.boys['3t'].tops.quantity  
>  # would also be supported
>
> I've already seen something like this in production; however, people are 
> having to write custom subclasses to do it.  This is kind of bummer because 
> the custom subclasses are a pain to write, are non-standard, and are 
> generally somewhat slow.  I would like to see a high-quality version this 
> made more broadly available.
>
> The core idea is keep the simple attribute access but make it easier to load 
> data programmatically:
>
> >>> ns = SimpleNamespace(roses='red', violets='blue')
> >>> thing = input()
> sugar
> >>> quality = input()
> sweet
> >>> setattr(ns, thing, quality)# current
> >>> ns['sugar'] = 'sweet'   # proposed
>
> If the PEP 584 __ior__ method were supported, updating a SimpleNamespace 
> would be much cleaner:
>
>   ns |= some_dict
>
> I posted an issue on the tracker: https://bugs.python.org/issue40284 .  There 
> was a suggestion to create a different type for this, but I don't see the 
> point in substantially duplicating everything SimpleNamespace already does 
> just so we can add some supporting dunder methods.   Please add more 
> commentary so we can figure-out the best way to offer this powerful 
> functionality.
>
>
> Raymond
>
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to 
> python-dev-leave@python.orghttps://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/JOMND56PJGRN7FQQLLCWONE5Z7R2EKXW/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
> --
> Regards,
> Ivan
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/IEKXDY7HUUCGF4BI327URFAWV67WQJZJ/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- 

[Python-Dev] Re: PEP 616 -- String methods to remove prefixes and suffixes

2020-03-24 Thread Gregory P. Smith
On Tue, Mar 24, 2020 at 11:55 AM Brett Cannon  wrote:

> -1 on "cut*" because my brain keeps reading it as "cute".
> +1 on "trim*" as it is clear what's going on and no confusion with
> preexisting methods.
> +1 on "remove*" for the same reasons as "trim*".
>
> And if no consensus is reached in this thread for a name I would assume
> the SC is going to ultimately decide on the name if the PEP is accepted as
> the burden of being known as "the person who chose _those_ method names on
> str" is more than any one person should have bear. ;)
>

"raymondLuxuryYacht*" pronounced Throatwobbler Mangrove it is!

Never fear, the entire stdlib is full of naming inconsistencies and
questionable choices accumulated over time.  Whatever is chosen will be
lost in the noise and people will happily use it.

The original PEP mentioned that trim had a different use in PHP which is
why I suggest avoiding that one.  I don't know how much crossover there
actually is between PHP and Python programmers these days outside of FB.

-gps

* https://montypython.fandom.com/wiki/Raymond_Luxury-Yacht

___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/Z7TK4C5PPECBRTCTPKCJEABFM62TDYWW/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/XC5GLSUXP6RNKEDFDGWLBSKT4TPSP5GU/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: PEP 616 -- String methods to remove prefixes and suffixes

2020-03-21 Thread Gregory P. Smith
Nice PEP! That this discussion wound up in the NP-complete "naming things"
territory as the main topic right from the start/prefix/beginning speaks
highly of it. :)

The only things left I have to add are (a) agreed on don't specify if it is
a copy or not for str and bytes.. BUT (b) do specify that for bytearray.

Being the only mutable type, it matters. Consistency with other bytearray
methods based on https://docs.python.org/3/library/stdtypes.html#bytearray
suggests copy.

(Someone always wants inplace versions of bytearray methods, that is a
separate topic not for this pep)

Fwiw I *like* your cutprefix/suffix names. Avoiding the terms strip and
trim is wise to avoid confusion and having the name read as nice English is
Pythonic.  I'm not going to vote on other suggestions.

-gps

On Sat, Mar 21, 2020, 9:32 PM Kyle Stanley  wrote:

> > In this case, being in line with the existing string API method names
> take priority over PEP 8, e.g. splitlines, startswith, endswith,
> splitlines, etc.
>
> Oops, I just realized that I wrote "splitlines" twice there. I guess that
> goes to show how much I use that specific method in comparison to the
> others, but the point still stands. Here's a more comprehensive set of
> existing string methods to better demonstrate it (Python 3.8.2):
>
> >>> [m for m in dir(str) if not m.startswith('_')]
> ['capitalize', 'casefold', 'center', 'count', 'encode', 'endswith',
> 'expandtabs', 'find', 'format', 'format_map', 'index', 'isalnum',
> 'isalpha', 'isascii', 'isdecimal', 'isdigit', 'isidentifier', 'islower',
> 'isnumeric', 'isprintable', 'isspace', 'istitle', 'isupper', 'join',
> 'ljust', 'lower', 'lstrip', 'maketrans', 'partition', 'replace', 'rfind',
> 'rindex', 'rjust', 'rpartition', 'rsplit', 'rstrip', 'split', 'splitlines',
> 'startswith', 'strip', 'swapcase', 'title', 'translate', 'upper', 'zfill']
>
> On Sun, Mar 22, 2020 at 12:17 AM Kyle Stanley  wrote:
>
>> Ivan Pozdeez wrote:
>> > I must note that names conforming to
>> https://www.python.org/dev/peps/pep-0008/#function-and-variable-names
>> would be "strip_prefix" and "strip_suffix".
>>
>> In this case, being in line with the existing string API method names
>> take priority over PEP 8, e.g. splitlines, startswith, endswith,
>> splitlines, etc. Although I agree that an underscore would probably be a
>> bit easier to read here, it would be rather confusing to randomly swap
>> between the naming convention for the same API. The benefit gained in 
>> *slightly
>> *easier readability wouldn't make up for the headache IMO.
>>
>> On Sun, Mar 22, 2020 at 12:13 AM Ivan Pozdeev via Python-Dev <
>> python-dev@python.org> wrote:
>>
>>> On 22.03.2020 6:38, Guido van Rossum wrote:
>>>
>>> On Sat, Mar 21, 2020 at 6:46 PM Nick Coghlan  wrote:
>>>
 On Sat., 21 Mar. 2020, 11:19 am Nathaniel Smith,  wrote:

> On Fri, Mar 20, 2020 at 11:54 AM Dennis Sweeney
>  wrote:
> > This is a proposal to add two new methods, ``cutprefix`` and
> > ``cutsuffix``, to the APIs of Python's various string objects.
>
> The names should use "start" and "end" instead of "prefix" and
> "suffix", to reduce the jargon factor and for consistency with
> startswith/endswith.
>

 This would also be more consistent with startswith() & endswith(). (For
 folks querying this: the relevant domain here is "str builtin method
 names", and we already use startswith/endswith there, not
 hasprefix/hassuffix. The most challenging relevant audience for new str
 builtin method *names* is also 10 year olds learning to program in school,
 not adults reading the documentation)

>>>
>>> To my language sense, hasprefix/hassuffix are horrible compared to
>>> startswith/endswith. If you were to talk about this kind of condition using
>>> English instead of Python, you wouldn't say "if x has prefix y", you'd say
>>> "if x starts with y". (I doubt any programming language uses hasPrefix or
>>> has_prefix for this, making it a strawman.)
>>>
>>> *But*, what would you say if you wanted to express the idea or removing
>>> something from the start or end? It's pretty verbose to say "remove y from
>>> the end of x", and it's not easy to translate that into a method name.
>>> x.removefromend(y)? Blech! And x.removeend(y) has the double 'e', which
>>> confuses the reader.
>>>
>>> The thing is that it's hard to translate "starts" (a verb) into a noun
>>> -- the "start" of something is its very beginning (i.e., in Python,
>>> position zero), while a "prefix" is a noun that specifically describes an
>>> initial substring (and I'm glad we don't have to use *that* :-).
>>>
>>>
 I think the concern about stripstart() & stripend() working with
 substrings, while strip/lstrip/rstrip work with character sets, is valid,
 but I also share the concern about introducing "cut" as yet another verb to
 learn in the already wide string API.

>>>
>>> It's not great, and I actually think that "stripprefix" and

[Python-Dev] Re: Remove formatter module from Python 3.9, deprecated for 7 years (Python 3.4)

2020-01-17 Thread Gregory P. Smith
As another datapoint, in an internal codebase with a high XX million lines
of Python code, I see a mere 7 uses.  Most of which are in legacy things
that would be trivial to use f-strings instead if the code doesn't just get
deleted when the owners notice. :)

(and my team is responsible for upgrading us to 3.9 so we'd simply make
that happen at that time)

-gps

On Fri, Jan 17, 2020 at 11:09 AM Gregory P. Smith  wrote:

> +1 remove it to match that policy.  I find it unlikely someone will ask
> for it to be restored, but our first response before doing so should be to
> see if they can instead use it as a dependency via PyPI: Someone who wants
> to maintain it for future use can take 3.8 formatter module and put it
> there.
>
> -gps
>
> On Fri, Jan 17, 2020 at 10:43 AM Andrew Svetlov 
> wrote:
>
>> I have no objections, the module raises the explicit deprecation
>> warning since 2013.
>>
>> On Fri, Jan 17, 2020 at 7:31 PM Victor Stinner 
>> wrote:
>> >
>> > Hi,
>> >
>> > I proposed https://bugs.python.org/issue39352 to remove the formatter
>> > module. It's deprecated since Python 3.4. The main reason why it's
>> > still around is the PEP 4 rule:
>> >
>> > "In order to facilitate writing code that works in both Python 2 & 3
>> > simultaneously, any module that exists in both Python 3.5 and Python
>> > 2.7 will not be removed from the standard library until Python 2.7 is
>> > no longer supported as specified by PEP 373."
>> >
>> >
>> https://www.python.org/dev/peps/pep-0004/#for-modules-existing-in-both-python-2-7-and-python-3-5
>> >
>> > Python 2.7 is not longer supported. So can we now remove the formatter
>> module?
>> >
>> > The module has no test, I failed to find any user in a dummy GitHub
>> > code search and I didn't know that this module even existed.
>> >
>> > We can still revert the module before 3.9.0 final release if someone
>> > shows up and asks to keep it for one more cycle.
>> >
>> > My intent here is to reduce the size of the Python standard library to
>> > reduce the maintenance burden. Python became quite large and it's more
>> > and more expensive to maintain it. Wait... I'm not asking to remove
>> > all modules of the standard library :-) The decision must be taken on
>> > a case by case basis, for each module. Here I'm only taking about the
>> > formatter module.
>> >
>> > Victor
>> > --
>> > Night gathers, and now my watch begins. It shall not end until my death.
>> > ___
>> > Python-Dev mailing list -- python-dev@python.org
>> > To unsubscribe send an email to python-dev-le...@python.org
>> > https://mail.python.org/mailman3/lists/python-dev.python.org/
>> > Message archived at
>> https://mail.python.org/archives/list/python-dev@python.org/message/ESSRZXEJ7MWMGNZQKNDURGRWIZ5FQHKP/
>> > Code of Conduct: http://python.org/psf/codeofconduct/
>>
>>
>>
>> --
>> Thanks,
>> Andrew Svetlov
>> ___
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-le...@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-dev@python.org/message/BHCWXBTELE354DCELWADMOX6O6CJOJ2A/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/IAYJ5HXIMRU7YMZUZDOPW54PQOBPPFDQ/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Remove formatter module from Python 3.9, deprecated for 7 years (Python 3.4)

2020-01-17 Thread Gregory P. Smith
+1 remove it to match that policy.  I find it unlikely someone will ask for
it to be restored, but our first response before doing so should be to see
if they can instead use it as a dependency via PyPI: Someone who wants to
maintain it for future use can take 3.8 formatter module and put it there.

-gps

On Fri, Jan 17, 2020 at 10:43 AM Andrew Svetlov 
wrote:

> I have no objections, the module raises the explicit deprecation
> warning since 2013.
>
> On Fri, Jan 17, 2020 at 7:31 PM Victor Stinner 
> wrote:
> >
> > Hi,
> >
> > I proposed https://bugs.python.org/issue39352 to remove the formatter
> > module. It's deprecated since Python 3.4. The main reason why it's
> > still around is the PEP 4 rule:
> >
> > "In order to facilitate writing code that works in both Python 2 & 3
> > simultaneously, any module that exists in both Python 3.5 and Python
> > 2.7 will not be removed from the standard library until Python 2.7 is
> > no longer supported as specified by PEP 373."
> >
> >
> https://www.python.org/dev/peps/pep-0004/#for-modules-existing-in-both-python-2-7-and-python-3-5
> >
> > Python 2.7 is not longer supported. So can we now remove the formatter
> module?
> >
> > The module has no test, I failed to find any user in a dummy GitHub
> > code search and I didn't know that this module even existed.
> >
> > We can still revert the module before 3.9.0 final release if someone
> > shows up and asks to keep it for one more cycle.
> >
> > My intent here is to reduce the size of the Python standard library to
> > reduce the maintenance burden. Python became quite large and it's more
> > and more expensive to maintain it. Wait... I'm not asking to remove
> > all modules of the standard library :-) The decision must be taken on
> > a case by case basis, for each module. Here I'm only taking about the
> > formatter module.
> >
> > Victor
> > --
> > Night gathers, and now my watch begins. It shall not end until my death.
> > ___
> > Python-Dev mailing list -- python-dev@python.org
> > To unsubscribe send an email to python-dev-le...@python.org
> > https://mail.python.org/mailman3/lists/python-dev.python.org/
> > Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/ESSRZXEJ7MWMGNZQKNDURGRWIZ5FQHKP/
> > Code of Conduct: http://python.org/psf/codeofconduct/
>
>
>
> --
> Thanks,
> Andrew Svetlov
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/BHCWXBTELE354DCELWADMOX6O6CJOJ2A/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/EGKD2KC4ZG2B7ZPA3NCBCWM3GRCJOP5M/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Adding a scarier warning to object.__del__?

2020-01-01 Thread Gregory P. Smith
On Wed, Jan 1, 2020 at 6:40 AM Andrew Svetlov 
wrote:

> __del__ is very useful not for interpreter shutdown but as a regular
> destructor in object's lifecycle.
>

The reason we should warn people against ever implementing __del__ is that
people rarely actually understand object lifecycle.  Its presence
guarantees delayed garbage collection and that its code will code execute
in a context way outside of normal control flow that all other methods are
invoked from.

-gps


>
> Action on the shutdown is another beast.
> Personally, I prefer to do all finalization works by explicit calls
> instead.
>
> On Wed, Jan 1, 2020 at 2:39 AM Yonatan Zunger  wrote:
> >
> > Hey everyone,
> >
> > I just encountered yet another reason to beware of __del__: when it's
> called during interpreter shutdown, for reasons which are kind of obvious
> in retrospect, if it calls notify() on a threading.Condition, the waiting
> thread may or may not ever actually receive it, and so if it does that and
> then tries to join() the thread the interpreter may hang in a hard-to-debug
> way.
> >
> > This isn't something that can reasonably be fixed, and (like in most
> cases involving __del__) there's a very simple fix of using
> weakref.finalize instead. My question for the dev list: How would people
> feel about changing the documentation for the method to more bluntly warn
> people against using it, and refer them to weakref.finalize and/or
> atexit.register as an alternative? The text already has an undertone of
> "lasciate ogni speranza, voi ch'entrate" but it may be helpful to be more
> explicit to avoid people getting caught in unexpected pitfalls.
> >
> > Yonatan
> > ___
> > Python-Dev mailing list -- python-dev@python.org
> > To unsubscribe send an email to python-dev-le...@python.org
> > https://mail.python.org/mailman3/lists/python-dev.python.org/
> > Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/AAZQBWD6PHC4PVNCCPX4A2745SS7B3LS/
> > Code of Conduct: http://python.org/psf/codeofconduct/
>
>
>
> --
> Thanks,
> Andrew Svetlov
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/57CDW4NIYEQ3JEVX2JVCJDA5TXTC5MBR/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TCLYO5W2WL57WV573GMEEVJSA7K6GBMH/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: sqlite3 module and thread-safe SQLite library

2019-12-17 Thread Gregory P. Smith
On Tue, Dec 17, 2019 at 12:30 PM Kacvinsky, Tom 
wrote:

> We ran into an issue where having the SQLite library built with
> -DSQLITE_THREADSAFE=0,
> but then the sqlite3 module (really, the _sqlite3.so0 crashing in
> threading code.  So I  have
> to ask if it is intended that the sqlite3 Python module always  be built
> with a thread safe
> SQLite library.
>
> Thanks,
>
> Tom
>

Given your experience I think you've answered the question: Yes.  But it
would be good if we could detect this at build time (is there a way to do
that?) and prevent the module from being built against the sqlite3 library
compiled in this unusual mode.  Threading support is required in order to
be a valid CPython platform, so all libraries used should be aware of that.

-gps


> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/PKHFTUQSLXWYOZNX4QMSNCPAF5Y3TL2H/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/KJV2KH7PZ55TX6XQKQ63WZ5WE7ILO26E/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: GitHub Actions enabled (was: Travis CI for backports not working)

2019-12-16 Thread Gregory P. Smith
On Mon, Dec 16, 2019 at 11:11 AM Steve Dower  wrote:

> On 13Dec2019 0959, Brett Cannon wrote:
> > Steve Dower wrote:
> >> If people are generally happy to move PR builds/checks to GitHub
> >> Actions, I'm happy to merge https://github.com/zooba/cpython/pull/7
> >> into
> >> our active branches (with probably Brett's help) and disable Azure
> >> Pipelines?
> >
> > I'm personally up for trying this out on master, making sure everything
> runs fine, and then push down into the other active branches.
>
> This is now running on master (and likely 3.8 and 3.7, at a guess) - you
> can see it on my PR at https://github.com/python/cpython/pull/17628
> (adding badges and making a tweak to when the builds run)
>
> The checks are not required yet - that requires admin powers.
>
> Please just shout out, either here on at
> https://bugs.python.org/issue39041 if you see anything not working.
>

neat, thanks Steve!


>
> (Apart from post-merge coverage checks. We know they're not working -
> see https://github.com/python/cpython/runs/351136928 - and if you'd like
> to help fix it you're more than welcome to jump in!)
>
> Cheers,
> Steve
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/R6VOXOL2HARK6ZHD7OWE4UP7PWTT5A4N/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QH5EA45GGDJVVZA44TZDP3OPJECP6AKX/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Handling cross-distro variations when running autoreconf?

2019-12-15 Thread Gregory P. Smith
I personally think we shouldn't worry about it. I *try* to use an autoconf
version at least as recent as what was used for the current file, but
beyond that it is unlikely anything depends on autoconf distroisms within
our own in tree configure.

Forcing humans to do weird tricks (always guaranteed to fail) to keep this
in some I'll defined state or another seems odd given anyone can rerun
autoconf (as the F package apparently does?) themselves if an issue with it
ever comes up. We only check it in and ship it in source tarballs as a
convenience.

--
blame half the typos on my phone.

On Sun, Dec 15, 2019, 5:33 AM Nick Coghlan  wrote:

> On Sun, 8 Dec 2019 at 23:58, Nick Coghlan  wrote:
> > Does anyone have any recommendations for dealing with this? My current
> > plan is to revert back to the configure script from master, run
> > autoreconf, and then use `git add -p` to only add in the desired
> > changes, leaving everything else as it was on master.
>
> I didn't end up using this plan. Instead, I copied the configure
> script *output* from master, and then used "git add -p" to revert the
> changes that arose solely from running autoreconf on a non-Debian
> system, while keeping the intentional changes from the PR.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/5T4UY6WON6ZTLAG36ELAYX4K6UDTVXGE/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5HK7I6RIIWNZIH7TJTRNFVXAUFMMFT2N/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Please be more precise when commenting on PEP 611.

2019-12-09 Thread Gregory P. Smith
My overall problem with the PEP and a reason I'd reject it by default it
that it is asking to pre-emptively impose limits on things, some of which
we believe would cause problems to existing long running applications
(limiting the total number of types for example), without having actually
demonstrated practical benefits from such a change.  Having an
implementation that takes limits and runs with them to practical effect
would provide motivation to consider adopting them.

Otherwise it doesn't feel like it solves a real problem, and could cause
some existing users pain.  So what's our real motivation?

picking on some nits within the PEP 611 text:

> Motivation: "poorly generated code could cause values to exceed 2^32"

Malicious or poorly generated code can always do anything so this isn't a
strong motivation.  If you want a nice failure when we come near actual
existing physical limits, that could be done today as a PR without a PEP.

If correctness is the motivation, we could be correct without changing the
existing unchecked limits.

> : "there are two ways to use a 32 bit refcount on a 64 bit machine. One
is to limit each sub-interpreter to 32Gb of memory. The other is to use a
saturating reference count, which would be a little bit slower, but allow
unlimited memory allocation."

Please do not arbitrarily cap sub-interpreter memory usage to a small
value.  32GiB is very small today.

Also, at least one existing eternal refcount implementation I've had
experience with demonstrated a notable hit to interpreter cpu performance
as it required an additional test+branch within the extremely widely used
Py_INCREF and Py_DECREF macro code.

-gps

On Mon, Dec 9, 2019 at 6:10 AM Mark Shannon  wrote:

> Hi everyone,
>
> Thanks again for all your comments on PEP 611.
>
> I would like to ask a favour; please be more specific in your comments.
>
> Ideally state which part of the PEP you are disagreeing with and why you
> disagree with the relevant part of the rationale/motivation.
>
> Also, when asking for limits to be raised or removed entirely, could you
> state what you perceive to be the costs and benefits of larger limits.
> What do you believe is an acceptable cost in memory or runtime for
> larger limits?
>
> For example, you might say that the limit of one million lines of code
> per module is too small, and that it is worth a small, say 1%, impact on
> speed to allow a larger of limit of 100 million.
>
> If you believe a limit would have no cost, then please give a
> explanation of why that is so.
>
> Merely saying that you would like a larger limit is pointless.
> If there were no cost to arbitrarily large limits, then I wouldn't have
> proposed the PEP in the first place.
>
> Bear in mind that the costs of higher limits are paid by everyone, but
> the benefits are gained by few.
>
> Cheers,
> Mark.
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/ZXDAJKRVSF6UUE5UEPE5PMXYXOLJ5A4V/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/EYXC7CDHVC4W2MHY5BTKAVH4IFSFV6LO/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: PEP 611: The one million limit.

2019-12-06 Thread Gregory P. Smith
I'd prefer it if we stayed on topic here...

On Fri, Dec 6, 2019 at 3:15 PM Chris Angelico  wrote:

> On Sat, Dec 7, 2019 at 9:58 AM Greg Ewing 
> wrote:
> >
> > On 7/12/19 2:54 am, Rhodri James wrote:
> >
> > > You've talked some about not making the 640k mistake
> >
> > I think it's a bit unfair to call it a "mistake". They had a 1MB
> > address space limit to work with, and that was a reasonable place
> > to put the division between RAM and display/IO space. If anything
> > is to be criticised, it's Intel's decision to only add 4 more
> > address bits when going from an 8-bit to a 16-bit architecture.
> >
>
> And to construct a bizarre segmented system that means that 16 + 16 =
> 20, thus making it very hard to improve on it later. If it hadn't been
> for the overlapping segment idea, it would have been easy to go to 24
> address lines later, and eventually 32. But since the 16:16 segmented
> system was built the way it was, every CPU afterwards had to remain
> compatible with it.
>
> Do you know when support for the A20 gate was finally dropped? 2013.
> Yes. THIS DECADE. If they'd decided to go for 32-bit addressing (even
> with 20 address lines), it would have been far easier to improve on it
> later.
>
> I'm sure there were good reasons for what they did (and hey, it did
> mean TSRs could be fairly granular in their memory requirements), but
> it's still a lesson to be learned from in not unnecessarily restrict
> something that follows Moore's Law.
>
> ChrisA
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/SW5HDQ27KBNYFMWAIMPVCGLVLNGQVNOK/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/2WHKCV6JZDTEWENX2LQTPH4JWU6PYGRI/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: PEP proposal to limit various aspects of a Python program to one million.

2019-12-03 Thread Gregory P. Smith
On Tue, Dec 3, 2019 at 8:21 AM Mark Shannon  wrote:

> Hi Everyone,
>
> I am proposing a new PEP, still in draft form, to impose a limit of one
> million on various aspects of Python programs, such as the lines of code
> per module.
>
> Any thoughts or feedback?
>
> The PEP:
> https://github.com/markshannon/peps/blob/one-million/pep-100.rst
>
> Cheers,
> Mark.
>
>
> Full text
> *
>
> PEP: 100
> Title: The one million limit
> Author: Mark Shannon 
> Status: Active
> Type: Enhancement
> Content-Type: text/x-rst
> Created: 03-Dec-2019
> Post-History:
>
>
>
> Abstract
> 
> This PR proposes a limit of one million (1 000 000) for various aspects
> of Python code and its implementation.
>
> The Python language does not specify limits for many of its features.
> Not having any limit to these values seems to enhance programmer freedom,
> at least superficially, but in practice the CPython VM and other Python
> virtual
> machines have implicit limits or are forced to assume that the limits are
> astronomical, which is expensive.
>
> This PR lists a number of features which are to have a limit of one
> million.
> If a language feature is not listed but appears unlimited and must be
> finite,
> for physical reasons if no other, then a limit of one million should be
> assumed.
>
>
> Motivation
> ==
>
> There are many values that need to be represented in a virtual machine.
> If no limit is specified for these values, then the representation must
> either be inefficient or vulnerable to overflow.
> The CPython virtual machine represents values like line numbers,
> stack offsets and instruction offsets by 32 bit values. This is
> inefficient, and potentially unsafe.
>
> It is inefficient as actual values rarely need more than a dozen or so
> bits to represent them.
>
> It is unsafe as malicious or poorly generated code could cause values to
> exceed 2\ :sup:`32`.
>
> For example, line numbers are represented by 32 bit values internally.
> This is inefficient, given that modules almost never exceed a few
> thousand lines.
> Despite being inefficent, is is still vulnerable to overflow as
> it is easy for an attacker to created a module with billions of newline
> characters.
>
> Memory access is usually a limiting factor in the performance of modern
> CPUs.
> Better packing of data structures enhances locality and reduces memory
> bandwith,
> at a modest increase in ALU usage (for shifting and masking).
> Being able to safely store important values in 20 bits would allow
> memory savings
> in several data structures including, but not limited to:
>
> * Frame objects
> * Object headers
> * Code objects
>
> There is also the potential for a more efficient instruction format,
> speeding up interpreter dispatch.
>
> Rationale
> =
>
> Imposing a limit on values such as lines of code in a module, and the
> number of local variables,
> has significant advantages for ease of implementation and efficiency of
> virtual machines.
> If the limit is sufficiently large, there is no adverse effect on users
> of the language.
>
> By selecting a fixed but large limit for these values,
> it is possible to have both safety and efficiency whilst causing no
> inconvience to human programmers
> and only very rare problems for code generators.
>
> One million
> ---
>
> The Java Virtual Machine (JVM) [1]_ specifies a limit of 2\ :sup:`16`-1
> (65535) for many program
> elements similar to those covered here.
> This limit enables limited values to fit in 16 bits, which is a very
> efficient machine representation.
> However, this limit is quite easily exceeded in practice by code
> generators and
> the author is aware of existing Python code that already exceeds 2\
> :sup:`16` lines of code.
>
> A limit of one million fits into 20 bits which, although not as
> convenient for machine representation,
> is still reasonably compact. Three signed valuses in the range -1000_000
> to +1000_000 can fit into a 64 bit word.
> A limit of one million is small enough for efficiency advantages (only
> 20 bits),
> but large enough not to impact users (no one has ever written a module
> of one million lines).
>
> The value "one million" is very easy to remember.
>
> Isn't this "640K ought to be enough for anybody" again?
> ---
>
> The infamous 640K memory limit was a limit on machine usable resources.
> The proposed one million limit is a limit on human generated code.
>
> While it is possible that generated code could exceed the limit,
> it is easy for a code generator to modify its output to conform.
> The author has hit the 64K limit in the JVM on at least two occasions
> when generating Java code.
> The workarounds were relatively straightforward and
> probably wouldn't have been necessary with a limit of one million
> bytecodes or lines of code.
>
>
> Specification
> =
>
> This PR proposes that the following language features and runtime values
> 

[Python-Dev] Re: Are we ready to start cleaning up deprecated stuff?

2019-11-27 Thread Gregory P. Smith
On Wed, Nov 27, 2019 at 10:15 AM Brett Cannon  wrote:

> Gregory P. Smith wrote:
> > On Tue, Nov 26, 2019 at 12:00 PM Brett Cannon br...@python.org wrote:
> > > Python 3.9 is going to be the first release which
> > > will exist without any
> > > Python 2.7 overlap. Does this mean we are ready to start removing
> things
> > > that have been deprecated since at least Python 3.7? PEP 4 says we
> > > are in
> > > the clear for modules,
> > > but I figured I would double-check as questions of cleaning up
> individual
> > > functions that have been deprecated for a very long time are now
> starting
> > > to come up (e.g. https://bugs.python.org/issue38916).
> > > If it has been through a usual deprecation cycle (in the past that was
> two
> > releases... with 3.9's now accelerated schedule does it count as a full
> > release for that purpose?  if not, three releases is always good) it
> seems
> > fair to consider removal.
> > The only thing that would make me say "hold off" on a specific removal is
> > if removing it will cause pain for people still dealing with a mixed 2.7
> > and 3.x codebase.  ie: If it is an old API from the 2.x era and there is
> no
> > easy way to accomplish the equivalent of that deprecation in 2.7 and 3.9+
> > without contortions I'd hold it just a little longer, until 3.10 or 3.11,
>
> But what's an acceptable contortion? Some might say something that can't
> be done with a search-and-replace is too much while others wouldn't.
>

Anything a lib2to3 fixer could be written for.  it probably isn't worth
trying to define this without a list of practical examples.  has anyone
collected a list of things we deprecated but have yet to remove?  I
anticipate we may wind up in "oh yeah, just remove it all already"
territory and this discussion will be moot. :)

> unless the existence of the deprecated thing is a large maintenance burden
> > rather than just an annoyance.
>
> Unfortunately that's hard to measure. For instance, the array.fromstring()
> deprecation that triggered this is probably fine to just leave, but if
> someone submits a PR to tweak the docs, the burden of that code suddenly
> went up. There's also the cost to users who import array, do a
> `dir(array)`, see fromstring(), and then start coding with it to find out
> later it's deprecated when they run their code (we all know people _should_
> read docs first, but I'm sure we are all guilty of having not done it as
> well ). Once again, potentially small, but it also adds up across all
> Python developers (which is probably is past 10,000,000 people).
>
> The fact that all code is a shared resource/burden and everything has a
> cognitive cost to it even if it's just to choose to ignore a PR that
> touches deprecated code is why I'm asking about this. I think I will start
> a separate thread on this that's not tied to Python 2.7.
>

By "large maintenance burden" I was specifically thinking of "the code
existing in the CPython codebase is preventing other nice refactorings and
redesigns from being done".  Anytime a long deprecated thing gets in the
way of such work, we should feel free to go ahead and remove it.

I guess I'm advocating for not going on a deprecation rampage and removing
all things we said would be gone by date $X unless there is a need.
Changing our policy to always do them by date $X would also be reasonable.

We could even come up with some post-release automation to auto-file bugs
reminding us remove deprecated things after deprecation-release-1 is out.
(we could require those get filed at deprecation time, but humans are good
at humaning and may forget to do that, plus we don't have N+3 release tags
for future release blockers to be filed in our bug tracker)

-gps


> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/KDDIHOJIHSAKNKJIYCOJM5ZELVFAQGFH/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/DRH3PXRQITWDTX73RJGSNNNMGFKJAOKC/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Are we ready to start cleaning up deprecated stuff?

2019-11-26 Thread Gregory P. Smith
On Tue, Nov 26, 2019 at 12:00 PM Brett Cannon  wrote:

> Python 3.9 is going to be the first release which will exist without any
> Python 2.7 overlap. Does this mean we are ready to start removing things
> that have been deprecated since at least Python 3.7? PEP 4 says [we are in
> the clear for modules](
> https://www.python.org/dev/peps/pep-0004/#for-modules-existing-in-both-python-2-7-and-python-3-5),
> but I figured I would double-check as questions of cleaning up individual
> functions that have been deprecated for a very long time are now starting
> to come up (e.g. https://bugs.python.org/issue38916).
>

If it has been through a usual deprecation cycle (in the past that was two
releases... with 3.9's now accelerated schedule does it count as a full
release for that purpose?  if not, three releases is always good) it seems
fair to consider removal.

The only thing that would make me say "hold off" on a specific removal is
if removing it will cause pain for people still dealing with a mixed 2.7
and 3.x codebase.  ie: If it is an old API from the 2.x era and there is no
easy way to accomplish the equivalent of that deprecation in 2.7 and 3.9+
without contortions I'd hold it just a little longer, until 3.10 or 3.11,
unless the existence of the deprecated thing is a large maintenance burden
rather than just an annoyance.

-gps
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/IVWSS7C26MXUTPLOT3E6ASDNZQQ2YBD2/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Stalemate on bringing back PEP 523 support into Python 3.8

2019-11-22 Thread Gregory P. Smith
On Fri, Nov 22, 2019 at 10:10 AM Mark Shannon  wrote:

> Hi,
>
> On 21/11/2019 8:04 pm, Brett Cannon wrote:
> > An unfortunate side-effect of making PyInterpreterState in Python 3.8
> opaque is it removed [PEP 523](https://www.python.org/dev/peps/pep-0523/)
> support. https://www.python.org/dev/peps/pep-0523/ was opened to try and
> fix this, but there seems to be a stalemate in the issue.
> >
> > A key question is at what API level should setting the frame evaluation
> function live at? No one is suggesting the stable ABI, but there isn't
> agreement between CPython or the internal API (there's also seems to be a
> suggestion to potentially remove PEP 523 support entirely).
> >
> > And regardless of either, there also seems to be a disagreement about
> providing getters and setters to continue to try and hide
> PyInterpreterState regardless of which API level support is provided at (if
> any).
> >
> > If you have an opinion please weight in on the issue.
>
> I have opinions :)
>
> I am opposed to adding any new API to modify the frame evaluation
> function (the interpreter). Although I would like to remove PEP 523, for
> now I propose that we do absolutely nothing.
> The pydev/pycharm debugger can be easily modified to use the internal
> data structure, so there is no urgency to do anything.
>
>
>
> Longer term I would like to remove PEP 523. I will write a PEP for this,
> but in the meantime let me explain why.
>
> First of all, PEP 523 doesn't explain who gets priority if two modules
> want to change the interpreter. Who wins? Or does it just crash?
> In case that seems unlikely, consider that PyCharm and PyDev already
> (mis)use PEP 523 for debugging and that cProfile could be sped up by
> using PEP 523 (please don't). What happens if you want to debug the
> profiled code?
>

It is a process wide global for use in special circumstances.  Those two
examples are debuggers.  Others are not.  But as a global it should be
expected that it isn't for general application level use.  Only one user
per process.  Or else the old manual chaining rule applies.  If you find it
set when you are setting your own, you need to call the previously set
one.  and when unsetting, if it isn't set to yours, you should leave it set
and turn your own into a no-op.  Realistically these aren't problems and
most people things aren't going to use the API.  I don't think anyone has
ever even encountered a combination of uses at once situation yet given the
complexity and obscurity.

At Google we use it to provide C/C++ level sampling profilers access to the
Python level program state by plumbing all Python calls through a unique C
function per Python callable such that a C level stack trace symbolizes to
information directly related to which Python function was called.  So that
light weight sampling profilers can get an accurate picture at the function
level of where time is being spent in a process executing Python code.
Without heavy handed live running scary signal based interpreter state
internal introspections (ie: vmprof).  It's a cool hack to get Python
functions to show up on the native C stack by proxy.  No, I don't expect
others to have implemented that (it requires a build step that creates C
stubs for all python callables, links those in and loads it all up - not
something most people have a way to do)   [yes I call this a hack, so no I
don't block anything on it - i'm just pointing out that there are many
creative uses of being able to inject oneself as a C function call within a
Python call stack whether we ultimately want that to be a thing or not]

Another use was made that recorded live types of functions at runtime to be
fed into type analyzers and ultimately feed back into adding practical
annotations on existing code.  your event callback approach would work for
that case.


>
> PEP 523 is not a good fit for debugging.
> ---
> Because using sys.settrace is so slow and there is no API specifically
> for debugging, the PyDev debugger has little choice but to use PEP 523.
> Debuggers using this approach need to directly modify bytecode, which is
> fragile.
> We need a good API for debuggers and other tools.
>
>
> PEP 523 isn't great for "JIT"s either.
> -
> (I use "JIT" as it is the term used in PEP 523. "Adaptive optimizer"
> would be a better term, IMO)
> A frame is a poor region for optimization, any effective optimizer would
> only use PEP 523 as a starting hook to allow it to monitor execution and
> select better regions for optimization.
> It is however, fine for experimental purposes. I have no problem keeping
> until we have evidence of working "JIT".
>

Does evidence require an open source release?  An existing JIT
implementation *was* mentioned in the PEP...

As mentioned above, any such optimizer would also interact very poorly
> with a PEP 523 based debugger.
>
>
> PEP 523 prevents some optimizations
> 

[Python-Dev] Re: What to do about invalid escape sequences

2019-08-09 Thread Gregory P. Smith
On Fri, Aug 9, 2019 at 11:37 AM Eric V. Smith  wrote:

> On 8/9/2019 2:28 PM, Jonathan Goble wrote:
> > On Fri, Aug 9, 2019 at 12:34 PM Nick Coghlan  wrote:
> >> I find the "Our deprecation warnings were even less visible than
> >> normal" argument for extending the deprecation period compelling.
> > Outsider's 2 cents from reading this discussion (with no personal
> > experience with this warning):
> >
> > I am perplexed at the opinion, seemingly espoused by multiple people
> > in this thread, that because a major part of the problem is that the
> > warnings were not visible enough, somehow the proposed solution is
> > making them not visible enough again? It's too late, in my
> > understanding, in the 3.8 cycle to add a new feature like a change to
> > how these warnings are produced (it seems a significant change to the
> > .pyc structure is needed to emit them at runtime), so this supposed
> > "solution" is nothing but kicking the can down the road. When 3.9
> > rolls around, public exposure to the problem of invalid escape
> > sequences will still be approximately what it is now (because if
> > nobody saw the warnings in 3.7, they certainly won't see them in 3.8
> > with this "fix"), so you'll end up with the same complaints about
> > SyntaxWarning that started this discussion, end up back on
> > DeprecationWarning for 3.9 (hopefully with support for emitting them
> > at runtime instead of just compile-time), then have to wait until
> > 3.10/4.0 for SyntaxWarning and eventually the next version to actually
> > make them errors.
>
> Yes, I think that's the idea: Deprecation warning in 3.9, but more
> visible that what 3.7 has. That is, not just at compile time but at run
> time. What's required to make that happen is an open question.
>

i've lost track of who suggested what in this thread, but yes, that concept
has been rolling over in my mind as a potentially good idea after someone
suggested it.  Compile time warnings should turn into bytecode for a
warnings.warn call in the generated pyc.  I haven't spent time trying to
reason if that actually addresses the real issues we're having moving
forward with a syntax warning change though.  A reasonable feature to ask
for as a feature in 3.9 or later perhaps.

-gps
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BR7T76SXANRAGJ3QOMWZUEGRVPVP/


[Python-Dev] Re: What to do about invalid escape sequences

2019-08-09 Thread Gregory P. Smith
On Fri, Aug 9, 2019 at 8:43 AM Guido van Rossum  wrote:

> This discussion looks like there's no end in sight. Maybe the Steering
> Council should take a vote?
>

I've merged the PR reverting the behavior in 3.8 and am doing the same in
the master branch.

The sheer volume of email this is generating shows that we're not ready to
do this to our users.

Recall the nightmare caused by md5.py and sha.py DeprecationWarning's in
2.5...  this would be similar.

We need owners of code to see the problems, not end users of other peoples
code.

FWIW, lest people think I don't like this change and just pushed the revert
buttons as a result, wrong.  I agree with the ultimate SyntaxError and
believe we should move the language there (it is better for long term code
quality).  But it needs to be done in a way that disrupts the *right*
people in the process, not disrupting an exponentially higher number of
users of other peoples code.

If the steering council does anything it should be deciding if we're still
going to do this at all and, if so, planning how we do it without repeating
past mistakes.

-gps
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/W4BUFLMDX5FAFOVLYP4C2LQ2HOTJZEZX/


[Python-Dev] Re: What to do about invalid escape sequences

2019-08-06 Thread Gregory P. Smith
People distribute code via pypi.  if we reject uploads of packages with
these problems and link to fixers (modernize can be taught what to do), we
prevent them from spreading further.  A few years after doing that, we can
revisit how much pain and for whom making this a SyntaxWarning or even
SyntaxError would actually be.

it isn't useful to tell *users* of packages to spend time figuring out who
to complain to that some packages code that they somehow depend on (often
transitively) is not modern enough.

On Tue, Aug 6, 2019 at 5:39 PM Rob Cliffe via Python-Dev <
python-dev@python.org> wrote:

>
>
> On 07/08/2019 01:14:08, Chris Angelico wrote:
> > On Wed, Aug 7, 2019 at 10:03 AM Steven D'Aprano 
> wrote:
> >> - Keep the SyntaxWarning silent by default for 3.8. That gives us
> >> another year or more to gently pressure third-party libraries to fix
> >> their code, and to find ways to encourage developers to run with
> >> warnings enabled.
> > How do you propose to apply this pressure?
> >
> > How about: whenever a third-party library uses a potentially-wrong
> > escape sequence, it creates a message on the console. Then when
> > someone sees that message, they can post a bug report against the
> > package.
> >
> > In other words, a non-silent warning.
> >
> > ChrisA
> > ___
> >
> The interpreter knows which module contains the questionable string.
> So: is it feasible for the warning message to include something like
> "...  If you are not the maintainer of xxxmodule.py, please contact
> them, or post a bug report on ..."
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/JJULDRHOTETV2LDLJE6RWKZ6HCCPHTUP/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/MK5SN5OCCRFHYBNUXL5BDUIN2Q7FV4JB/


[Python-Dev] Re: What to do about invalid escape sequences

2019-08-06 Thread Gregory P. Smith
On Tue, Aug 6, 2019 at 10:06 AM Neil Schemenauer 
wrote:

>
> Making it an error so soon would be mistake, IMHO.  That will break
> currently working code for small benefit.  When Python was a young
> language with a few thousand users, it was easier to make these
> kinds of changes.  Now, we should be much more conservative and give
> people a long time and a lot of warning.  Ideally, we should provide
> tools to fix code if possible.
>
> Could PyPI and pip gain the ability to warn and even fix these
> issues?  Having a warning from pip at install time could be better
> than a warning at import time.  If linting was built into PyPI, we
> could even do a census to see how many packages would be affected by
> turning it into an error.
>

PyPI could warn on or reject packages at upload time when they contain
these kinds code compileall -> pyc time of issues.  that'd force the issue
to be in front of the package owners rather than anyone else.

Similarly, automation could proactively scan existing pypi packages
claiming py3 compatibility for the issue and reach out to owners.

But the issue becomes one of versions: we shouldn't be complaining about
older versions of packages to the package owners.  yet people are free to
list old versions or < constraints in their own packages requirements
regardless of what python version they're running on.

-gps


>
> On 2019-08-05, raymond.hettin...@gmail.com wrote:
> > P.S. In the world of C compilers, I suspect that if the relatively
> > new compiler warnings were treated as errors, the breakage would
> > be widespread. Presumably that's why they haven't gone down this
> > road.
>
> The comparision with C compilers is relevant.  C and C++ represent a
> fairly extreme position on not breaking working code.   E.g. K & R
> style functional declarations were supported for decades.  I don't
> think we need to go quite that far but also one or two releases is
> not enough time.
>
> Regards,
>
>   Neil
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/V2EDFDJGXRIDMKJU3FKIWC2NDLMUZA2Y/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/IH62W73S3WFSW6MUHJJ7XBTGFO4PXAWE/


[Python-Dev] Re: PEP 581 has been updated with "Downsides of GitHub" section

2019-07-02 Thread Gregory P. Smith
On Mon, Jul 1, 2019 at 9:01 AM Steven D'Aprano  wrote:

> On Sat, Jun 29, 2019 at 10:26:04AM -0500, Skip Montanaro wrote:
> > > You have missed at least one: the minimum technology requirement for
> > > using Github is a lot more stringent than for Roundup. Github's minimum
> > > system requirements are higher, and it doesn't degrade as well, so
> > > moving to Github will make it much harder for those who are using older
> > > technology. If not exclude them altogether.
> >
> > Is that Git or GitHub? If the latter, more JavaScript bits or something
> else?
>
> I'm referring to Github. I expect it is probably Javascript. Clicking
> hyperlinks is fine, but things like buttons with dropdown menus, the
> hamburger icon, the Clone/Download button etc do nothing when I click on
> them.
>
> For what it's worth, I'm not using an ad blocker but I am using a
> moderately old Firefox.
>

Document this kind of thing, that is fine.

But a *separate* issue is that we *should not try* to cater to people who
cannot use a modern up to date self updating trustworthy web browser.

A browser more than a few months old is a security risk; *we* are not going
to enforce that concept, but we shouldn't go out of our way to support it
either.
The same goes for anyone who disables javascript or uses a content
modifying plugins or extensions in their browser.  Problems caused by that
have a simple solution, it isn't on us to untangle.

-gps
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/NNRODVYN2DRDJ66TETY7JB5ZAHA4CUIK/


[Python-Dev] Re: python-ideas and python-dev migrated to Mailman 3/HyperKitty

2019-06-07 Thread Gregory P. Smith
On Thu, Jun 6, 2019 at 6:19 AM Victor Stinner  wrote:

> Le jeu. 6 juin 2019 à 14:18, Steven D'Aprano  a
> écrit :
> > i.e. 25-40% longer. Is there a shorter permalink form available, like
> > goo.gl, bitly, youtu.be etc use? That would be awesome if we could use
> > them instead.
>
> I really dislike URL shorteners.
>
> From my point of view, URL shorteners are the opposite of permanent
> links. It adds a new single point of failure.
>

Unless you run your own shortener... operated it part of the infrastructure
the shortened links will be going to.  In which case the shortened url
service prevents the need to modify each and every application to handle
shorter things on its own.  That way short urls can be made meaningful on
their own to indicate where they link to while still being a friendly to
avoid gross wrapping length.

We could implement our own automatic short namespaces such as
https://url.python.org/mail/ideas/48+bit-random-id

goo.gl and youtu.be are examples of site specific url shorteners already
used this way (now that goo.gl has stopped / is stopping offering itself as
a public bit.ly clone service)

This also avoids one (of many) ways of giving traffic log stats away to a
third party hosting the service.

-gps


> If a full URL becomes broken (HTTP error 404), I can still try to get
> find the information in a different way using the full URL. For
> example, there are some services like archive.org who archive public
> websites. Or look in other emails "around" this email, maybe the
> missing email is quoted from another email.
>
> Victor
> --
> Night gathers, and now my watch begins. It shall not end until my death.
> Python-Dev mailing list -- python-dev(a)python.org
> To unsubscribe send an email to python-dev-leave(a)python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BBEKSKJCNTUK4LMH435UAZOHZ4KXTP3D/


Re: [Python-Dev] [python-committers] PEP 595: Improving bugs.python.org

2019-05-24 Thread Gregory P. Smith
On Fri, May 24, 2019 at 1:48 PM Ezio Melotti  wrote:

>
> On Fri, May 24, 2019, 20:23 Gregory P. Smith  wrote:
>
>> -cc: committers to avoid crossposting.
>>
>
> +1 (I wanted to include committers, since the announcement about PEP 581
> was posted there too, but it's better to keep the discussion here)
>
>
>> I have feedback for roundup as experienced on BPO that should be
>> represented within PEP 595 if we are going to have a summary of "improving
>> roundup for BPO" captured in a PEP (presumably already rejected given 581?
>> But good to have documented regardless so _thank you_ for doing this
>> writeup even though I realize our plan of record may be demoralizing for
>> you).
>>
>
> We would like people to re-evaluate the decision, but if that doesn't
> happen I think the PEP is still useful, since it provides a fair view of
> Roundup capabilities and discusses things that we will have to take into
> account if we proceed with the migration -- that's why we decided to go
> ahead and write the PEP.
>
>
>> > **Flexible UI.**  While Roundup UI might look dated, it is convenient
>> and flexible.
>>
>> I wholly disagree with this statement.
>>
>
>> The BPO roundup UI drives me nuts. every. single. time. I have to use
>> it.  It is not optimized for common workflows users actually need to
>> accomplish when using a bug tracker.  Two example usability issues (of
>> many): Users can't read the latest update to a bug of length because it is
>> hidden within the *middle* of the scrolling region, they must hunt for it.
>>
>
> This came up in the past, and two solutions have been proposed already:
> 1) keyboard shortcuts have been added in the issue page to quickly jump to
> the first/previous/next/last message and to the response box [0].  They
> support both mnemonic (f/p/n/l/r) and vim-style (h/k/j/l/i) combinations.
> You can find a summary table in the left sidebar of the issue page, under
> help -> keyboard shortcuts.
> 2) a patch to collapse the history by default (so that the last message
> was at the end of the page) was proposed and merged [1], but it was
> reverted after a few days because some devs wanted direct access to the
> history without having to do an extra click every time to expand it.
>
> After reading, the text box to add to the discussion is oddly located near
>> the *top* of the scrolling region so that a user cannot see context of a
>> bug discussion they are responding to as they type.
>>
>
> This has also been discussed and different people had different opinion.
> Some suggested to reverse the order of the messages so that the last
> message is at the top near the reply box (like Twitter does), but other
> said it's unnatural to read.  Some suggested to put the reply box at the
> bottom; however if the other fields are left at the top you would have to
> go back up to set them, and if they are moved down you won't easily see
> them at the top when you open an existing issue.  Another solution is
> duplicating the fields and response box at the top and bottom.
>

The two other modern bugtracker UIs I use on a regular basis do this by
having all such non-conversation metainfo in a persistent top and side bar
boxes such that it is always present.  Keeping the conversation and
metainfo changes listed in order (latest message at the bottom, metainfo
deltas interspersed in between messages all ordered/grouped by timestamp,
and reply text entry box below that.  These two are typically big-screen
engineering UIs (github being one of them), if mobile is desired i expect
these would effectively wind up as a multi-pane UI.  There's a third ticket
that I use for non-engineering stuff which does things in reverse order
with the text entry up top and messages in reverse chronological order
below that.  This reversal always annoys me; probably because I spend most
of my time in the others so it forces me to do headstands. I don't know if
there is a *right* answer between the two approaches, I just know what I
prefer. But the common theme is keeping new the update UI elements right
next to the most recent entries which is what BPO lacks today.

What I mean with *flexible* when I wrote the PEP, is that, unlike GitHub
> (where they decide), we can customize bpo however we want (as long as we
> agree on what we want -- we could even have per-user settings if we really
> want to :)
>

Definitely true in that sense.  Custom things, no matter where, are custom
code.  diverge from an upstream project in our own fork and maintenance and
updates become harder.  Author a pile of bots to automate tasks within a
standard API or UI (as is working great for our Github PR workflow) and the
equivalent maintenance burden falls 

Re: [Python-Dev] [python-committers] PEP 595: Improving bugs.python.org

2019-05-24 Thread Gregory P. Smith
-cc: committers to avoid crossposting.

I have feedback for roundup as experienced on BPO that should be
represented within PEP 595 if we are going to have a summary of "improving
roundup for BPO" captured in a PEP (presumably already rejected given 581?
But good to have documented regardless so _thank you_ for doing this
writeup even though I realize our plan of record may be demoralizing for
you).

> **Flexible UI.**  While Roundup UI might look dated, it is convenient and
flexible.

I wholly disagree with this statement.

The BPO roundup UI drives me nuts. every. single. time. I have to use it.
It is not optimized for common workflows users actually need to accomplish
when using a bug tracker.  Two example usability issues (of many): Users
can't read the latest update to a bug of length because it is hidden within
the *middle* of the scrolling region, they must hunt for it.  After
reading, the text box to add to the discussion is oddly located near the
*top* of the scrolling region so that a user cannot see context of a bug
discussion they are responding to as they type.  I file things like this
under "User experience design is needed for the common workflows of all
classes of users".

Roundup needs a modern responsive web interface, not GET/POST request based
interface seen on BPO.  As a result of that, roundup *feels* like is
belongs in the Pre-2004 era interface wise by being web form and full page
reload server for everything. A responsive modern "async javascript
requests happen in the background of the UI" system that we all expect of
any web UI is needed. Not just tweaking the existing thing to have a mobile
friendly version of the web form. This includes persistent connections so
that updates to an issue show up live as they happen instead of getting an
error message "someone/something else has updated this issue since you
started typing, the action you wanted to take such as submitting that
comment or editing that field is now invalid and cannot be completed
without a lot of manual work figuring out what happened, cut and pasting,
and fixing things up on the you the users part".

I'm not going to try proposing a PR to this PEP encapsulating that, I'll
leave that up to anyone willing to wrangle such a PEP.  The list archive
has it regardless now. :)

-Greg


On Thu, May 23, 2019 at 1:17 PM Ezio Melotti  wrote:

> Hello,
> Berker and I have been working on a PEP that suggests we keep using
> and improving bugs.python.org and Roundup instead of switching to
> GitHub Issues as proposed by PEP 581.
>
> The PEP covers:
> * What are the advantages of Roundup over GitHub issues;
> * What features are missing in Roundup and how can we add them;
> * Issues with PEP 581;
> * Issues with the migration plan proposed by PEP 588;
>
> The rendered version of PEP 595 is available at
> https://www.python.org/dev/peps/pep-0595/
>
> For reference, you can consult PEP 581 and 588 at
> https://www.python.org/dev/peps/pep-0581/ and
> https://www.python.org/dev/peps/pep-0588/
>
> The full text of the PEP is include below.  We are planning to update
> the PEP to include the feedback we receive and to update the status of
> features as we implement them (we also have a Google Summer of Code
> students working on it).
>
> Best Regards,
> Ezio Melotti
>
>
> 
> PEP: 595
> Title: Improving bugs.python.org
> Author: Ezio Melotti , Berker Peksag
> 
> Status: Draft
> Type: Process
> Content-Type: text/x-rst
> Created: 12-May-2019
>
>
> Abstract
> 
>
> This PEP proposes a list of improvements to make bugs.python.org
> more usable for contributors and core developers.  This PEP also
> discusses why remaining on Roundup should be preferred over
> switching to GitHub Issues, as proposed by :pep:`581`.
>
>
> Motivation
> ==
>
> On May 14th, 2019 :pep:`581` has been accepted [#]_ without much
> public discussion and without a clear consensus [#]_.  The PEP
> contains factual errors and doesn't address some of the
> issues that the migration to GitHub Issues might present.
>
> Given the scope of the migration, the amount of work required,
> and how it will negatively affect the workflow during the
> transition phase, this decision should be re-evaluated.
>
>
> Roundup advantages over GitHub Issues
> =
>
> This section discusses reasons why Roundup should be preferred
> over GitHub Issues and Roundup features that are not available
> on GitHub Issues.
>
> * **Roundup is the status quo.**  Roundup has been an integral
>   part of the CPython workflow for years.  It is a stable product
>   that has been tested and customized to adapt to our needs as the
>   workflow evolved.
>
>   It is possible to gradually improve it and avoid the disruption
>   that a switch to a different system would inevitabily bring to
>   the workflow.
>
> * **Open-source and Python powered.**  Roundup is an open-source
>   project and is written in Python.  By using it and supporting
>   it, 

Re: [Python-Dev] Have a big machine and spare time? Here's a possible Python bug.

2019-05-23 Thread Gregory P. Smith
I suggest filing a bug to track this...

On Thu, May 23, 2019 at 10:13 AM Tim Peters  wrote:

> [Inada Naoki ]
> > ...
> > 2. This loop is cleary hot:
> >
> https://github.com/python/cpython/blob/51aa35e9e17eef60d04add9619fe2a7eb938358c/Objects/obmalloc.c#L1816-L1819
>
> Which is 3 lines of code plus a closing brace.  The OP didn't build
> their own Python, and the source from which it was compiled wasn't
> available to them.  But they did say that when they got into gdb and
> single-stepped, it was looping through the same three lines of code,
> in obmalloc.c, over & over.  Which is 100% consistent with the loop
> you identified as being "the hot spot".
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] bpo-36829: Add sys.unraisablehook()

2019-05-16 Thread Gregory P. Smith
I like the feature, we should have it.  It'll be useful for debugging and
probably more.

Which brings me to the annoying paint color question: These exceptions were
most definitely raised. Thus the term "unraisable" is wrong. I believe you
really mean "uncatchable".

-gps

On Thu, May 16, 2019 at 4:00 PM Steve Dower  wrote:

> On 16May2019 1441, Victor Stinner wrote:
> > Le jeu. 16 mai 2019 à 23:17, Steve Dower  a
> écrit :
> >> You go on to say "pass an error message" and "keep repr(obj) if you
> >> want", but how is this different from creating an exception that
> >> contains the custom message, the repr of the object, and chains the
> >> exception that triggered it?
> >
> > Well, "unraisable exceptions" are raised when something goes wrong.
> > I'm not comfortable with creating a new exception instance and
> > chaining it to the previous exception, because it could go wrong and
> > raise a 3rd exception. Issue a new error while trying to log an error
> > can be annoying :-(
>
> I mean, aren't they all? :) That's why they're exceptional.
>
> If we're not in a position to raise a new exception, then it's not safe
> to call into user's Python code. If it's safe to call into their Python
> code, then we have to be in a position to raise a new exception.
>
> You said earlier that we can safely do this, and now you're saying we
> can't safely do it. Which is it?
>
> > Moreover, when I looked at details of the "unraisable exception" (see
> > my reply to Petr Viktorin), I saw that the exception is not well
> > defined as you might expect. (exc_type, exc_value, exc_tb) can *not*
> > be replaced with (type(exc_value), exc_value,
> > exc_value.__traceback__). Sometimes, exc_value is None. Sometimes,
> > exc_tb is a traceback object, but exc_value.__traceback__ is None. So
> > I'm not comfortable neither to chain such badly shaped exception into
> > a new exception.
>
> Yeah, this is the normal optimization for raising and handling
> exceptions that never reach Python code. Exceptions are supposed to be
> normalized before re-entering the eval loop.
>
> > I prefer to pass "raw" values and let the hook decides how to handle
> them :-)
>
> This is fine, it's how we handle exceptions in most other places (such
> as __exit__, logging, sys, traceback and all through the C API), and
> skipping normalization for a last-chance "the-world-may-be-burning" hook
> isn't the worst thing ever.
>
> Cheers,
> Steve
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] Unicode identifiers in test files?

2019-05-03 Thread Gregory P. Smith
On Fri, May 3, 2019 at 10:46 PM Eric V. Smith  wrote:

> Is there a policy against using Unicode identifiers in test files?
>
> As part of adding !d to f-strings, there's a code path that's only
> executed if the text of the expression is non-ascii. The easiest way to
> exercise it, and the way I found a bug, is by using an identifier with
> Unicode chars. I know we have a policy against this in Lib/, but what
> about Lib/test/?
>
> I could work around this with exec-ing some strings, but that seems like
> added confusion that I'd avoid with a real Unicode identifier.
>

At this point I'd just do it, make sure the file has an explicit coding
utf-8 comment up top just so it is clear it is intentional but everything
should be happy with it.

-gps


>
> Eric
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] Need help to fix HTTP Header Injection vulnerability

2019-04-10 Thread Gregory P. Smith
On Wed, Apr 10, 2019 at 11:00 AM Ivan Pozdeev via Python-Dev <
python-dev@python.org> wrote:

>
> On 10.04.2019 7:30, Karthikeyan wrote:
>
> Thanks Gregory. I think it's a good tradeoff to ensure this validation
> only for URLs of http scheme.
>
> I also agree handling newline is little problematic over the years and the
> discussion over the level at which validation should occur also prolongs
> some of the patches. https://bugs.python.org/issue35906 is another
> similar case where splitlines is used but it's better to raise an error and
> the proposed fix could be used there too. Victor seemed to wrote a similar
> PR like linked one for other urllib functions only to fix similar attack in
> ftplib to reject newlines that was eventually fixed only in ftplib
>
> * https://bugs.python.org/issue30713
> * https://bugs.python.org/issue29606
>
> Search also brings multiple issues with one duplicate over another that
> makes these attacks scattered over the tracker and some edge case missing.
> Slightly off topic, the last time I reported a cookie related issue where
> the policy can be overriden by third party library I was asked to fix it in
> stdlib itself since adding fixes to libraries causes maintenance burden to
> downstream libraries to keep up upstream. With urllib being a heavily used
> module across ecosystem it's good to have a fix landing in stdlib that
> secures downstream libraries encouraging users to upgrade Python too.
>
> Validation should occur whenever user data crosses a trust boundary --
> i.e. when the library starts to assume that an extracted chunk now contains
> something valid.
>
> https://tools.ietf.org/html/rfc3986 defines valid syntax (incl. valid
> characters) for every part of a URL -- _of any scheme_ (FYI, \r\n are
> invalid everywhere and the test code for `data:' that Karthikeyan
> referred to is raw data to compare to rather than a part of a URL). It also
> obsoletes all the RFCs that the current code is written against.
>
> AFAICS, urllib.split* fns (obsoleted as public in 3.8) are used by both
> urllib and urllib2 to parse URLs. They can be made to each validate the
> chunk that they split off. urlparse can validate the entire URL altogether.
>
> Also, all modules ought to use the same code (urlparse looks like the best
> candidate) to parse URLs -- this will minimize the attack surface.
>
> I think I can look into this later this week.
>
My PR as of last night cites that RFC and does validation in http.client
while constructing the protocol request payload.  Doing it within split
functions was an initial hack that looked like it might work but didn't
feel right as that isn't what people expect of those functions and that
turned out to be the case as I tested things due to our mess of codepaths
for opening URLs, but they all end with http.client so yay!

I did *not* look at any of the async http client code paths.  (legacy
asyncore or new asyncio).  If there is an issue there, those deserve to
have their own bugs filed.

As for third party PyPI libraries such as urllib3, they are on their own to
fix bugs.  If they happen to use a code path that a stdlib fix helps, good
for them, but honestly they are much better off making and shipping their
own update to avoid the bug.  Users can get it much sooner as it's a mere
pip install -U away rather than a python runtime upgrade.

> Fixing this is going to break code that relies on the current code
> accepting invalid URLs. But the docs have never said that e.g. in urlopen,
> anything apart from a (valid) URL is accepted (in particular, this implies
> that the user is responsible for escaping stuff properly before passing
> it). So I would say that we are within our right here and whoever is
> relying on those quirks is and has always been on unsupported territory.
>
yep.  even http.client.HTTPConnection.request names the function parameter
"url" so anyone embedding whitespace newlines and http protocol strings
within that is well outside of supported territory (as one example in our
own test_xmlrpc was taking advantage of to test a malformed request).

I suggest following up on https://bugs.python.org/issue30458 rather than in
this thread.  the thread did its job, it directed our eyeballs at the
problems. :)

-gps

> Determining which of those quirks are exploitable and which are not to fix
> just the former is an incomparably larger, more error-prone and avoidable
> work. If anything, the history of the issue referenced to by previous
> posters clearly shows that this is too much to ask from the Python team.
>
> I also see other undocumented behavior like accepting '>' (also
> obsoleted as public in 3.8) which I would like to but it's of no harm.
>
> --
>
> Regards,
> Ivan
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>

Re: [Python-Dev] Need help to fix HTTP Header Injection vulnerability

2019-04-09 Thread Gregory P. Smith
On Tue, Apr 9, 2019 at 4:45 PM Karthikeyan  wrote:

> I would recommend fixing it since it's potentially remote code execution
> on systems like Redis (latest versions of Redis have this mitigated) though
> I must admit I don't fully understand the complexity since there are
> multiple issues linked. Go was also assigned a CVE for linked issue and it
> seemed to be the same reporter by username : CVE-2019-9741 . I tried using
> go's approach in the commit but urlopen accepts more URLs like data URLs
> [0] that seemed to accept \n as a valid case and the patch broke some
> tests. Looking at the issue discussion complexity also involves backwards
> compatibility. golang also pushed an initial fix that seemed to broke their
> internal tests [0] to arrive at a more simpler fix.
>
> [0]
> https://github.com/python/cpython/blob/a40681dd5db8deaf05a635eecb91498dac882aa4/Lib/test/test_urllib.py#L482
> [1]
> https://go-review.googlesource.com/c/go/+/159157/2#message-39c6be13a192bf760f6318ac641b432a6ab8fdc8
>
> --
> Regards,
> Karthikeyan S
>

useful references, thanks!  limiting the checks to only http and https as
those are the text based protocols with urls transmitted in text form makes
sense and avoids the data: test failures.

proposed simple fix in https://github.com/python/cpython/pull/12755

but tests are needed as is an audit of the code to see where else we may
potentially need to do such things.

-gps
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] Strange umask(?)/st_mode issue

2019-04-01 Thread Gregory P. Smith
On Mon, Apr 1, 2019 at 4:49 PM Cameron Simpson  wrote:

> On 01Apr2019 15:44, Steve Dower  wrote:
> >On 01Apr2019 1535, Cameron Simpson wrote:
> >>On 01Apr2019 09:12, Steve Dower  wrote:
> >>>On 30Mar2019 1130, Gregory P. Smith wrote:
> >>>>I wouldn't expect it to be the case in a CI environment but I
> >>>>believe a umask can be overridden if the filesystem is mounted
> >>>>and configured with acls set?  (oh, hah, Ivan just said the same
> >>>>thing)
> >>>
> >>>Yep, it appears this is the case. The Pipelines team got back to
> >>>me and it seems to be a known issue - the workaround they gave me
> >>>was to run "sudo setfacl -Rb /home/vsts" at the start, so I've
> >>>merged that in for now (to master and 3.7).
> >>
> >>Could that be done _without_ sudo to just the local directory
> >>containing the test tar file? If that works then you don't need any
> >>nasty privileged sudo use (which will just break on platforms
> >>without sudo anyway).
> >
> >I tried something similar to that and it didn't work. My guess is it's
> >to do with the actual mount point? (I also tried without sudo at
> >first, and when I didn't work, I tried it with sudo. I hear that's how
> >to decide whether you need it or not ;) )
> >
> >In any case, it only applies to the Azure Pipelines build definition,
> >so there aren't any other platforms where it'll be used.
>
> Ok then.
>
> Curious: is the sudo now in the build setup? I'm just thinking that this
> isn't a tarfile specific fix but a "get correct POSIX umask semantics"
> fix, so it should apply to the entire environment.
>
> Or am I naive?
>

I'm reading between the lines and assuming we're not the only user of their
CI complaining about this environment change. ;)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] Strange umask(?)/st_mode issue

2019-03-30 Thread Gregory P. Smith
I wouldn't expect it to be the case in a CI environment but I believe a
umask can be overridden if the filesystem is mounted and configured with
acls set?  (oh, hah, Ivan just said the same thing)

-gps

On Sat, Mar 30, 2019 at 9:05 AM Steve Dower  wrote:

> On 29Mar.2019 1944, Steve Dower wrote:
> > On 29Mar.2019 1939, Cameron Simpson wrote:
> >> Can you get a branch into your pipeline? Then you could just hack the
> >> tarfile test with something quick and dirty like:
> >>
> >>pid = os.getpid()
> >>system("strace -p %d 2>/path/to/strace.out &" % pid)
> >>time.sleep(2)   # get strace heaps of time to start
> >>
> >> just before the tarfile open. A ghastly hack but it would get you
> >> debugging info. You could even decide to remove the strace.out file if
> >> the umask issue doesn't show, if it is erratic (can't see why it would
> >> be though).
> >
> > Perfect, I'll give this a go. Thanks!
>
> I set up a PR to collect this trace and the results are at:
> https://github.com/python/cpython/pull/12625
>
> However, I suspect it's a non-result:
>
> umask(022)  = 022
>
> open("/home/vsts/work/1/s/build/test_python_5154/@test_5154_tmp-tardir/tmp.tar",
> O_WRONLY|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = 3
> write(3, "BZh91AY{\270\344\0\0\24P\0\300\0\4\0\0\10
> \\314\5)\246"..., 46) = 46
> close(3)= 0
>
> stat("/home/vsts/work/1/s/build/test_python_5154/@test_5154_tmp-tardir/tmp.tar",
> {st_mode=S_IFREG|0666, st_size=46, ...}) = 0
>
> Happy to take more suggestions if anyone has them.
>
> Thanks,
> Steve
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] PEP 556 threaded garbage collection & linear recursion in gc

2019-03-27 Thread Gregory P. Smith
Good point, I hadn't considered that it was regular common ref count 0
dealloc chaining.  The processes unfortunately didn't have faulthandler
enabled so there wasn't much info from where in the python code it happened
(now fixed).  I'll see if anything looks particularly unusual next time I
hear of such a report.

-G

On Wed, Mar 27, 2019, 5:38 PM Tim Peters  wrote:

> [Gregory P. Smith ]
> > ...
> > A situation came up the other day where I believe this could've helped.
> >
> > Scenario (admittedly not one most environments run into): A Python
> process
> > with a C++ extension module implementing a threaded server (threads
> > spawned by C++) that could call back into CPython within server request
> > handlers.  (ie: how all threaded servers tend to work regardless of core
> > loop implementation language)
> >
> > Python code in the application had done something (unknown to me, I
> > didn't dive into their code) that built up large enough presumably nested
> > or recursive data structures that the garbage collector, when triggered,
> > would wind up in very deep recursion.  This caused a stack overflow
> >  as the C++ spawned threads were only being given a 256k stack (to
> > conserve virtual address space - there can potentially be a _ton_ of
> > threads in this code).
> >
> > That had a C++ stack trace 1000+ levels deep repeating the pattern of
> >
> > ...
> > @ 0x564d59bd21de 32  func_dealloc
> > @ 0x564d59bce0c1 32  cell_dealloc
> > @ 0x564d5839db41 48  tupledealloc
> > @ 0x564d59bd21de 32  func_dealloc
> > @ 0x564d59bce0c1 32  cell_dealloc
> > @ 0x564d5839db41 48  tupledealloc
> > ...
> >
> > If our gc were done on a thread of its own spawned by Python, with a
> typical
> > normal larger default stack size (8mb) this would've been less likely
> > to trigger a crash (though obviously still possible if the recursion is
> linear).
>
> Just noting that gc is probably a red herring in that scenario.  gc
> (cleverly!) determines the set of trash objects without needing any
> recursive graph crawls.  And gc doesn't deallocate anything - not
> directly.  It simply applies Py_DECREF to trash objects, once for each
> PyObject* found pointing to them.  _If_ deallocation occurs as a
> result, it's the same as if the user had done `del` on the appropriate
> object manually.  The recursion then occurs in the chain of
> XXX_dealloc calls, which in turn do more Py_DECREFs of their own, and
> which have nothing in particular to do with gc.  Approximately the
> same stack depth would be hit in a Python built without gc if the user
> did the same sequence of `del`s by hand to break trash cycles.
>
> The good news is that the "trashcan" mechanism - which predates gc -
> was introduced to limit call depth in the presence of some potentially
> unbounded chains of dealloc calls.  So teach trashcan about the
> pattern here, and the stack depth problem goes away, with or without
> gc.
>
> The bad news is that the traschcan mechanism is excruciating, a
> long-time source of subtle bugs of its own :-(
>
> Note:  without actual code to run, it's possible that trashcan is
> _already_ trying to limit stack depth in this specific case, but the
> stack size is too small to accommodate the largest depth of recursive
> calls trashcan is willing to allow before interfering.  Or some bug
> snuck into trashcan, or its invoking code, that causes it to fail to
> work as intended due to some unknown detail of the code above.
> There's just no way to guess without code to provoke it.
>
>
> > Another take away from this is that it appears possible to cause our gc
> > to go into linear recursion.
>
> As above, it's probably not gc, but deallocation as a side effect of
> Py_DECREF dropping a refcount to 0.  gc is just one way Py_DECREF can
> get invoked.
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


[Python-Dev] PEP 556 threaded garbage collection & linear recursion in gc

2019-03-27 Thread Gregory P. Smith
https://www.python.org/dev/peps/pep-0556/

This PEP is currently Deferred as nobody is actively working on a test
implementation.

A situation came up the other day where I *believe* this could've helped.

Scenario (admittedly not one most environments run into): A Python process
with a C++ extension module implementing a threaded server (threads spawned
by C++) that could call back into CPython within server request handlers.
(ie: how all threaded servers tend to work regardless of core loop
implementation language)

Python code in the application had done something (unknown to me, I didn't
dive into their code) that built up large enough presumably nested or
recursive data structures that the garbage collector, when triggered, would
wind up in very deep recursion.  This caused a stack overflow as the C++
spawned threads were only being given a 256k stack (to conserve virtual
address space - there can potentially be a _ton_ of threads in this code).

That had a C++ stack trace 1000+ levels deep repeating the pattern of

...
@ 0x564d59bd21de 32  func_dealloc
@ 0x564d59bce0c1 32  cell_dealloc
@ 0x564d5839db41 48  tupledealloc
@ 0x564d59bd21de 32  func_dealloc
@ 0x564d59bce0c1 32  cell_dealloc
@ 0x564d5839db41 48  tupledealloc
...

If our gc were done on a thread of its own spawned by Python, with a
typical normal larger default stack size (8mb) this would've been less
likely to trigger a crash (though obviously still possible if the recursion
is linear).

I know, there are obvious workarounds to this situation of all sorts.  My
point is more that synchronously triggering gc _within_ a thread that
happens to invoke the periodic "hey, lets do another gc run" logic the eval
loop is undesirable.

I'm not making any plans to work on an implementation for this PEP,
deferred seems accurate. Just dropping a note that I'd still be interested
in seeing something other than synchronous gc in arbitrary threads happen
when a process has multiple threads.

*Another* take away from this is that it *appears* possible to cause our gc
to go into linear recursion.  Yuck!  I'd file an issue on that, but doing
so requires making some example code to construct such a scenario first...

Food for thought in case these kinds of things are something anyone else
has encountered.

-Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] Best way to specify docstrings for member objects

2019-03-20 Thread Gregory P. Smith
(answers above and below the quoting)

I like the idea of documenting attributes, but we shouldn't force the user
to use __slots__ as that has significant side effects and is rarely
something people should bother to use.  There are multiple types of
attributes.  class and instance.  but regardless of where they are
initialized, they all define the API shape of a class (or instance).

Q: Where at runtime regardless of syntax chosen would such docstrings
live?  One (of many) common conventions today is to just put them into an
Attributes: or similar section of the class docstring.  We could actually
do that automatically by appending a section to the class docstring, but
that unstructures the data enshrining one format and could break existing
code for the users of the few but existing APIs that treat docstrings as
structured runtime data instead of documentation if someone were to try and
use attribute docstrings on subclasses of those library types.  (ply does
this, I believe some database abstraction APIs do as well).

On Wed, Mar 20, 2019 at 12:41 AM Serhiy Storchaka 
wrote:

> 19.03.19 20:55, Raymond Hettinger пише:
> > I'm working on ways to make improve help() by giving docstrings to
> member objects.
> >
> > One way to do it is to wait until after the class definition and then
> make individual, direct assignments to __doc__ attributes.This way widely
> the separates docstrings from their initial __slots__ definition.   Working
> downstream from the class definition feels awkward and doesn't look pretty.
> >
> > There's another way I would like to propose¹.  The __slots__ definition
> already works with any iterable including a dictionary (the dict values are
> ignored), so we could use the values for the  docstrings.
>
> I think it would be nice to separate docstrings from the bytecode. This
> would be allow to have several translated sets of docstrings and load an
> appropriate set depending on user preferences. This would help in
> teaching Python.
>
> It is possible with docstrings of modules, classes, functions, methods
> and properties (created by using the decorator), because the compiler
> knows what string literal is a docstring. But this is impossible with
> namedtuple fields and any of the above ideas for slots.
>
> It would be nice to allow to specify docstrings for slots as for methods
> and properties. Something like in the following pseudocode:
>
> class NormalDist:
>  slot mu:
> '''Arithmetic mean'''
>  slot sigma:
>  '''Standard deviation'''
>

I don't think adding a 'slot' keyword even if limited in scope to class
body definition level is a good idea (very painful anytime we reserve a new
word that is already used in code and APIs).


> It would be also nice to annotate slots and add default values (used
> when the slot value was not set).
>
> class NormalDist:
>  mu: float = 0.0
> '''Arithmetic mean'''
>  sigma: float = 1.0
>  '''Standard deviation'''
>
>
Something along these lines is more interesting to me.  And could be
applied to variables in _any_ scope.  though there wouldn't be a point in
using a string in context where the name isn't bound to a class or module.

The best practice today remains "just use the class docstring to document
your public class and instance attributes".  FWIW other languages tend to
generate their documentation from code via comments rather than requiring a
special in language runtime accessible syntax to declare it as
documentation.

It feels like Python is diverging from the norm if we were encourage more
of this __doc__ carried around at runtime implicit assignment than we
already have.  I'm not convinced that is a good thing.

-gps
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] Is XML serialization output guaranteed to be bytewise identical forever?

2019-03-19 Thread Gregory P. Smith
On Tue, Mar 19, 2019, 4:53 AM Ned Batchelder  wrote:

> On 3/19/19 4:13 AM, Serhiy Storchaka wrote:
> > 19.03.19 00:41, Raymond Hettinger пише:
> >> 4) Fix the tests in the third-party modules to be more focused on
> >> their actual test objectives, the semantics of the generated XML
> >> rather than the exact serialization.  This option would seem like the
> >> right-thing-to-do but it isn't trivial because the entire premise of
> >> the existing test is invalid.  For every case, we'll actually have to
> >> think through what the test objective really is.
> >
>
> Option 4 is misleading.  Is anyone here really offering to "fix the
> tests in third-party modules"?  Option 4 is actually, "do nothing, and
> let a multitude of projects figure out how to fix their tests, slowing
> progress in those projects as they try to support Python 3.8."
>

We've done Option 4 for every past behavior change of any form on feature
.Next releases.  We do try to encourage projects to run their tests on the
3.Next betas so that they can be ready before 3.Next.0 lands, some of us
even do it ourselves when we're interested.  Many things won't get ready
ahead of time, but the actual .0 release forces the issue as their users
start demanding it on occasion offering patches.  We don't bock a release
on existing user code being ready for it.

In my case, the test code has a generic function to compare an actual
> directory of files to an expected directory of files, so it isn't quite
> as simple as "just use the right XML comparison."  And I support Python
> 2.7, 3.5, etc, so tests still need to work under those versions.  None
> of this is impossible, but please try not to preach to us maintainers
> that we are doing it wrong, that it will be easy to fix, etc.  Using
> language like "the entire premise of the test is invalid" seems
> needlessly condescending.
>

Agreed, that was poor wording.  Lets not let that type of wording escape
python-dev into docs about a behavior change.

Wording aside, a test relying on undefined behavior is testing for things
the code under test doesn't actually need to care about being true, even if
it has happened to work for years.  Such a test is overspecified.
Potentially without the authors previously consciously realizing that.
It'll need refactoring to loosen its requirements.  How to loosen it is
always an implementation detail based on the constraints imposed upon the
test.  Difficulty lies within range(0, "Port Mercurial to Python 3").  But
the end result is nice: The code is healthier as tests focus more on what
was actually important rather than what was quicker to write that got the
original job done many years ago.

One of the suggested solutions, a DOM comparison is not enough. I
> don't just want to know that my actual XML is different than my expected
> XML.  I want to know where and how it differs.
>
> Textual comparison may be the "wrong" way to check XML, but it gives me
> many tools for working with the test results.  It was simple and it
> worked.  Now in Python 3.8, because Python doesn't want to add an
> optional flag to continue doing what it has always done, I need to
> re-engineer my tests.
>
> --Ned.
>

I understand that from a code owners perspective having to do any work, no
matter what the reason, is counted as re-engineering.  But that doesn't
make it wrong.  If "what it has always done" was unspecified and arbitrary
despite happening to not change in the past rather than something easy to
continue the stability of such as sorted attributes, it is a burden to
maintain that **unspecifiable** behavior in the language or library going
forward.

(Note that I have no idea what order the xml code in question happened to
impose upon attributes; if it went from sorted to not a "fix" to provide
users is clear: provide a way to keep it sorted for those who need that.
If it relied on insertion order or hash table iteration order or the phase
of the moon when the release was cut, we are right to refuse to maintain
unspecifiable implementation side effect behavior)

-gps
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


  1   2   3   4   5   6   >