[Python-Dev] Re: Critique of PEP 657 -- Include Fine Grained Error Locations in Tracebacks

[Pablo Galindo Salgado]

Ok, we have implemented a rough prototype and we have decided not to go
with this for the following reasons:

* Is almost the same storage cost as the solution we already have. Since
storing the node id cost 4 bytes (an integer) per instruction
and our solution needs 2+ bytes per instruction for the offsets (normally
just 2 as offsets are generally < 256) + the compressed end line,
which is very small since is almost always the same as the start line.
* It actually doesn't have more advantages. The current solution in the PEP
can do exactly the same as this solution if you allow reparsing when
displaying tracebacks. This is because with the start line, end line, start
offset and end offset and the original file, you can extract the source that
is associated with the instruction, parse it (and this
is much faster because you just need to parse the tiny fragment) and then
you get an AST node that you can use for whatever you want.
* The proposed solution forces to reparse several times entire files just
to extract a single AST node. Even worse: for frames in the same file it
forces
to reparse those again and again unless you complicate the implementation
by adding AST caching. But even that it won't be free as it will incur in
quite a lot of extra memory and this is problematic, especially when
displaying exceptions as memory can be low, which the current design takes
into account.
* Nodes are created and modified after the optimization pass, so the AST
produced by the parser is not enough to reconstruct the actual
information, we need to also run the optimization passes, but
unfortunately, this is (by design) not don't in the Python API (to preserve
all possible information about the original code), so this complicates
quite a lot the API to get this, as `ast.parse` is not enough to get the
original tree.
* The implementation is quite more complex then the one the PEP has since
to do it right implies having to hash the source files, implement node id
propagation in the parser and a node visitor to find the correct node +
reparsing in the traceback.
* It makes the AST (both the C one and the Python one) bigger, which
increases the memory costs of parsing and very slightly affects performance
(we measured a 4% decrease in perf to add the new field).
* It makes the API for 3rd party tools very non-straightforward, forcing
reparsing and finding the AST nodes. Even if we provide
some new functionality in the ast module or similar to make this easier, it
quite a lot of overhead just to get the position information.

Even if is possible to solve many of these problems, we think the
complexity is not worth it, especially since it actually doesn't give more
functionality.

On Tue, 18 May 2021 at 13:47, Pablo Galindo Salgado 
wrote:

> > One integer is actually not enough to assign IDs.
>
> Actually, disregard this particular problem. I think that we could
> perfectly stop assigning IDs if we reach the overflow limit and call it a
> day
> since you need to have a truly horrendous file to reach 4,294,967,295 AST 
> nodes
> (I did some tests to check this).
>
> On Tue, 18 May 2021 at 13:25, Pablo Galindo Salgado 
> wrote:
>
>> Yet another problem that I found:
>>
>> One integer is actually not enough to assign IDs. One unsigned integer
>> can cover 4,294,967,295 AST nodes, but is technically possible
>> to have more than that in a single file. While in PEP 657 we are tracking
>> offsets that are normally very low < 100 typically or end lines
>> that are easily compressible (as end lines are normally equal to the
>> start of the line), node IDs can be arbitrarily big. Which is worse:
>> the parser creates some AST nodes that throw away if the parser fails
>> (that's how PEG parsers work), so there will be a lot of IDs that
>> don't get assigned (and the parser doesn't have an easy way to know how
>> many IDs has thrown away). This forces us to either use something
>> bigger than an integer (probably a size_t) or to deal with overflow.
>>
>> On Tue, 18 May 2021 at 10:23, Pablo Galindo Salgado 
>> wrote:
>>
>>> Hu, actually another problem of this approach:
>>>
>>> Nodes are created and modified after the optimization pass, so the AST
>>> produced by the parser is not enough to reconstruct the actual
>>> information, we need to also run the optimization passes, but
>>> unfortunately, this is (by design) not don't in the Python API (to preserve
>>> all possible information about the original code), so this complicates
>>> quite a lot the API to get this, as `ast.parse` is not enough to get the
>>> original tree.
>>>
>>> On Tue, 18 May 2021 at 08:53, Pablo Galindo Salgado 
>>> wrote:
>>>
 Hi Nathaniel,

 Thanks a lot for your suggestion! I like the idea although I still
 think is more complex than our current proposal, but on the other hand it
 allows for a much richer results so I'm quite excited to try it out. We are
 going to give it a go to explore it with a prototype and if we are
 

[Python-Dev] Re: March Steering Council update.

[Guido van Rossum]

Thanks, that's the context I was missing!

Package management is hard. No wonder some folks try to bypass the whole
thing and use Docker images instead. :-(

On Tue, May 18, 2021 at 8:56 AM Christian Heimes 
wrote:

> On 18/05/2021 16.19, Guido van Rossum wrote:
> > There are a few mentions of Debian, but no explanation of what the issue
> > is about. Can you elaborate on that?
>
> Debian and Debian-based distros like Ubuntu are applying downstream
> packages and split CPython interpreter and stdlib into multiple
> packages. For example venv and distutils is split out and ensurepip is
> patched out. Debian packages were also missing dependencies on
> ca-certificates and tzdata, which are required for Python's ssl and
> zoneinfo modules. The changes caused usability issues for new and
> sometimes even for seasoned users of Python.
>
> I have contacted the SC in the beginning of this year and asked them to
> work with Debian maintainers. Some issues have been addressed and will
> be available in future releases. Matthias' talk at the language summit
> is related to the effort of improving Debian packaging.
>
> Christian
>


-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YQ4VIRFVNH57QPORXYH34GPAGRDFQODX/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: March Steering Council update.

[Christian Heimes]

On 18/05/2021 16.19, Guido van Rossum wrote:
> There are a few mentions of Debian, but no explanation of what the issue
> is about. Can you elaborate on that?

Debian and Debian-based distros like Ubuntu are applying downstream
packages and split CPython interpreter and stdlib into multiple
packages. For example venv and distutils is split out and ensurepip is
patched out. Debian packages were also missing dependencies on
ca-certificates and tzdata, which are required for Python's ssl and
zoneinfo modules. The changes caused usability issues for new and
sometimes even for seasoned users of Python.

I have contacted the SC in the beginning of this year and asked them to
work with Debian maintainers. Some issues have been addressed and will
be available in future releases. Matthias' talk at the language summit
is related to the effort of improving Debian packaging.

Christian
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/VV5IVOSIKWV5END64PXNAFEQ2SNGP3VS/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: March Steering Council update.

[Nick Coghlan]

Thanks for the update!

I know this request is a little ironic coming from me, but would it be
possible to state the PEP titles the first time they're mentioned each
month?

Cross referencing is a little awkward when reading the summary on a phone
rather than a full computer.

Cheers,
Nick.

On Tue, 18 May 2021, 11:12 pm Thomas Wouters,  wrote:

>
> The SC has just published the community update for March:
>
>
> https://github.com/python/steering-council/blob/main/updates/2021-03-steering-council-update.md
>
> We're still trying to get these done every month, but between the rush of
> PEPs and other issues before the 3.10b1 deadline, and PyCon US, we're
> delayed a little. (The April update hopefully won't be too long.) In the
> meantime, the SC keynote from PyCon US covers our longer-term plans and our
> points of view on a few other subjects, and that will hopefully be up on
> YouTube (
> https://www.youtube.com/playlist?list=PL2Uw4_HvXqvYk1Y5P8kryoyd83L_0Uk5K)
> soon. (Since it's mentioned in the notes below, I just want to point out
> that we didn't record on April 19th; it was pushed back to May 3rd, the
> same day 3.10b1 was cut.)
>
> March 1
>
>- Steering Council synced up on the rejection draft for PEP 651. Group
>discussed the importance of conveying that all aspects of the proposal were
>reviewed and the decision is a holistic one. Group will continue working on
>the draft and will sync up via Slack during the week so it can be sent out.
>- The Steering Council discussed PEP 648. Barry created a doc and the
>group will consolidate feedback+questions into this document. Then the
>Steering Council will post this on Discourse as well as to python-dev@.
>- The Steering Council extensively discussed PEP 637. The group
>decided that it would reject the PEP based on the PEPs costs not being
>worth the benefits.
>- The Steering Council also discussed typing in general and who should
>own it.
>- Next week the SC will vote on the Documentation Work Group's charter.
>
>
> March
> 8
>
>- The Steering Council discussed Mark's response to the rejection of
>PEP 651.
>- The group also discussed the notification for PEP 637.
>- The group approved PEP 624 & PEP 597.
>- The Steering Council discussed PEP 644 and decided further
>communication was needed with Christian.
>- The Steering Council voted on and approved the Documentation Work
>Group.
>- The group discussed moving master to main and decided it needed to
>be done. The SC discussed communication around this change.
>
>
> March
> 15
>
>- Thomas updated the notifications for PEP 597, PEP 624 and PEP 637,
>and sent out the notifications March 15th.
>- Group checked in on the draft for PEP 648. Thomas has more text to
>add to it.
>- The Steering Council discussed what kind of presentation they will
>give at PyCon US. The group decided on a combination of presentation and
>Q Team is working on an outline. SC members were also reminded to
>register for PyCon US, and encourage others to do so.
>- Barry proposed to the SC that they create a Work Group with a subset
>of the Python Security Response Team members and that group can help scope
>the future of the PSRT group. Everyone is fine with this.
>- Steering Council discussed code of conduct situations around
>changing master to main. The group decided on a warning to S.D. that
>will be sent as a warning to all. Thomas is working on the initial draft.
>
>
> March
> 22
>
>- Barry and Thomas checked-in on PEP 648's draft response, they will
>be sending it out soon.
>- The SC reviewed PEP 644 more. Pablo raised a concern around users
>that can be in an env without OpenSSL 1.1.1 or newer and then not being
>able to use wheels. Pablo will email Christian to get clarification and
>will keep the SC informed.
>- Steering Council discussed their keynote at PyCon US 2021. They
>would like to gather questions from the community. Ewa will create a Slido,
>which will be live from April 5 to 11th. SC will review questions on the
>12th. The current plan is to record on April 19th.
>- Steering Council discussed the response to Debian. Carol's done with
>her draft. Pablo is going to take a look at it to see if anything else
>needs to be added.
>- SC discussed the behavior on python-dev@ pertaining to switching git
>from master to main.
>
>
> March
> 29
>
>- The Steering Council accepted PEP 644 (Require 

[Python-Dev] Re: March Steering Council update.

[Guido van Rossum]

There are a few mentions of Debian, but no explanation of what the issue is
about. Can you elaborate on that?

On Tue, May 18, 2021 at 06:15 Thomas Wouters  wrote:

>
> The SC has just published the community update for March:
>
>
> https://github.com/python/steering-council/blob/main/updates/2021-03-steering-council-update.md
>
> We're still trying to get these done every month, but between the rush of
> PEPs and other issues before the 3.10b1 deadline, and PyCon US, we're
> delayed a little. (The April update hopefully won't be too long.) In the
> meantime, the SC keynote from PyCon US covers our longer-term plans and our
> points of view on a few other subjects, and that will hopefully be up on
> YouTube (
> https://www.youtube.com/playlist?list=PL2Uw4_HvXqvYk1Y5P8kryoyd83L_0Uk5K)
> soon. (Since it's mentioned in the notes below, I just want to point out
> that we didn't record on April 19th; it was pushed back to May 3rd, the
> same day 3.10b1 was cut.)
>
> March 1
>
>- Steering Council synced up on the rejection draft for PEP 651. Group
>discussed the importance of conveying that all aspects of the proposal were
>reviewed and the decision is a holistic one. Group will continue working on
>the draft and will sync up via Slack during the week so it can be sent out.
>- The Steering Council discussed PEP 648. Barry created a doc and the
>group will consolidate feedback+questions into this document. Then the
>Steering Council will post this on Discourse as well as to python-dev@.
>- The Steering Council extensively discussed PEP 637. The group
>decided that it would reject the PEP based on the PEPs costs not being
>worth the benefits.
>- The Steering Council also discussed typing in general and who should
>own it.
>- Next week the SC will vote on the Documentation Work Group's charter.
>
>
> March
> 8
>
>- The Steering Council discussed Mark's response to the rejection of
>PEP 651.
>- The group also discussed the notification for PEP 637.
>- The group approved PEP 624 & PEP 597.
>- The Steering Council discussed PEP 644 and decided further
>communication was needed with Christian.
>- The Steering Council voted on and approved the Documentation Work
>Group.
>- The group discussed moving master to main and decided it needed to
>be done. The SC discussed communication around this change.
>
>
> March
> 15
>
>- Thomas updated the notifications for PEP 597, PEP 624 and PEP 637,
>and sent out the notifications March 15th.
>- Group checked in on the draft for PEP 648. Thomas has more text to
>add to it.
>- The Steering Council discussed what kind of presentation they will
>give at PyCon US. The group decided on a combination of presentation and
>Q Team is working on an outline. SC members were also reminded to
>register for PyCon US, and encourage others to do so.
>- Barry proposed to the SC that they create a Work Group with a subset
>of the Python Security Response Team members and that group can help scope
>the future of the PSRT group. Everyone is fine with this.
>- Steering Council discussed code of conduct situations around
>changing master to main. The group decided on a warning to S.D. that
>will be sent as a warning to all. Thomas is working on the initial draft.
>
>
> March
> 22
>
>- Barry and Thomas checked-in on PEP 648's draft response, they will
>be sending it out soon.
>- The SC reviewed PEP 644 more. Pablo raised a concern around users
>that can be in an env without OpenSSL 1.1.1 or newer and then not being
>able to use wheels. Pablo will email Christian to get clarification and
>will keep the SC informed.
>- Steering Council discussed their keynote at PyCon US 2021. They
>would like to gather questions from the community. Ewa will create a Slido,
>which will be live from April 5 to 11th. SC will review questions on the
>12th. The current plan is to record on April 19th.
>- Steering Council discussed the response to Debian. Carol's done with
>her draft. Pablo is going to take a look at it to see if anything else
>needs to be added.
>- SC discussed the behavior on python-dev@ pertaining to switching git
>from master to main.
>
>
> March
> 29
>
>- The Steering Council accepted PEP 644 (Require OpenSSL 1.1.1 or
>newer)
>- SC accepted PEP 652 (Maintaining the Stable ABI)
>- SC checked in on the draft response for PEP 648 (Extensible
>customizations of the 

[Python-Dev] Re: PEP 659: Specializing Adaptive Interpreter

[Chris Angelico]

On Tue, May 18, 2021 at 8:51 PM Stephen J. Turnbull
 wrote:
>
> Steve Holden writes:
>  > On Thu, May 13, 2021 at 11:07 PM Steven D'Aprano 
>  > wrote:
>  >
>  > > Steve
>  > > (one of the other ones)
>  > >
>  >
>  > We are all other Steves!
>
> +1
>
> There were five Steves (and one Stephanie) in my 6th grade class (of
> 27).  "Steve, move that " became an idiom 
>
> -- Other Steve since 1994
>

I feel the need to redress the balance of names here. This thread has
had a mere two Chrises so far, and I am improving that statistic by
50%.

ChrisA
(used to style his name as "chrisa" but people complained that I
looked like a girl)
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UMMFCI5NTHCF3PZVNJYVRH4ZKQQDWE5L/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] March Steering Council update.

[Thomas Wouters]

The SC has just published the community update for March:

https://github.com/python/steering-council/blob/main/updates/2021-03-steering-council-update.md

We're still trying to get these done every month, but between the rush of
PEPs and other issues before the 3.10b1 deadline, and PyCon US, we're
delayed a little. (The April update hopefully won't be too long.) In the
meantime, the SC keynote from PyCon US covers our longer-term plans and our
points of view on a few other subjects, and that will hopefully be up on
YouTube (
https://www.youtube.com/playlist?list=PL2Uw4_HvXqvYk1Y5P8kryoyd83L_0Uk5K)
soon. (Since it's mentioned in the notes below, I just want to point out
that we didn't record on April 19th; it was pushed back to May 3rd, the
same day 3.10b1 was cut.)

March 1

   - Steering Council synced up on the rejection draft for PEP 651. Group
   discussed the importance of conveying that all aspects of the proposal were
   reviewed and the decision is a holistic one. Group will continue working on
   the draft and will sync up via Slack during the week so it can be sent out.
   - The Steering Council discussed PEP 648. Barry created a doc and the
   group will consolidate feedback+questions into this document. Then the
   Steering Council will post this on Discourse as well as to python-dev@.
   - The Steering Council extensively discussed PEP 637. The group decided
   that it would reject the PEP based on the PEPs costs not being worth the
   benefits.
   - The Steering Council also discussed typing in general and who should
   own it.
   - Next week the SC will vote on the Documentation Work Group's charter.

March
8

   - The Steering Council discussed Mark's response to the rejection of PEP
   651.
   - The group also discussed the notification for PEP 637.
   - The group approved PEP 624 & PEP 597.
   - The Steering Council discussed PEP 644 and decided further
   communication was needed with Christian.
   - The Steering Council voted on and approved the Documentation Work
   Group.
   - The group discussed moving master to main and decided it needed to be
   done. The SC discussed communication around this change.

March
15

   - Thomas updated the notifications for PEP 597, PEP 624 and PEP 637, and
   sent out the notifications March 15th.
   - Group checked in on the draft for PEP 648. Thomas has more text to add
   to it.
   - The Steering Council discussed what kind of presentation they will
   give at PyCon US. The group decided on a combination of presentation and
   Q Team is working on an outline. SC members were also reminded to
   register for PyCon US, and encourage others to do so.
   - Barry proposed to the SC that they create a Work Group with a subset
   of the Python Security Response Team members and that group can help scope
   the future of the PSRT group. Everyone is fine with this.
   - Steering Council discussed code of conduct situations around changing
   master to main. The group decided on a warning to S.D. that will be sent
   as a warning to all. Thomas is working on the initial draft.

March
22

   - Barry and Thomas checked-in on PEP 648's draft response, they will be
   sending it out soon.
   - The SC reviewed PEP 644 more. Pablo raised a concern around users that
   can be in an env without OpenSSL 1.1.1 or newer and then not being able to
   use wheels. Pablo will email Christian to get clarification and will keep
   the SC informed.
   - Steering Council discussed their keynote at PyCon US 2021. They would
   like to gather questions from the community. Ewa will create a Slido, which
   will be live from April 5 to 11th. SC will review questions on the 12th.
   The current plan is to record on April 19th.
   - Steering Council discussed the response to Debian. Carol's done with
   her draft. Pablo is going to take a look at it to see if anything else
   needs to be added.
   - SC discussed the behavior on python-dev@ pertaining to switching git
   from master to main.

March
29

   - The Steering Council accepted PEP 644 (Require OpenSSL 1.1.1 or newer)
   - SC accepted PEP 652 (Maintaining the Stable ABI)
   - SC checked in on the draft response for PEP 648 (Extensible
   customizations of the interpreter at startup), and the draft response to
   the Debian discussion.
   - SC met with Ezio, PM for the GitHub Issues migration, to talk about
   status, progress and next steps, including keeping python-dev up to date.


-- 
Thomas Wouters 

Hi! I'm an email virus! Think twice before sending your email to help me
spread!

[Python-Dev] Re: Critique of PEP 657 -- Include Fine Grained Error Locations in Tracebacks

[Larry Hastings]

On 5/18/21 5:25 AM, Pablo Galindo Salgado wrote:

Yet another problem that I found:

One integer is actually not enough to assign IDs. One unsigned integer 
can cover 4,294,967,295 AST nodes, but is technically possibleto have 
more than that in a single file.



Surely you could use a 64-bit int for the node ID.


//arry/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/AEBB7YY2OYSTPJL5K44USRS3WCS2BTMI/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Critique of PEP 657 -- Include Fine Grained Error Locations in Tracebacks

[Pablo Galindo Salgado]

> One integer is actually not enough to assign IDs.

Actually, disregard this particular problem. I think that we could
perfectly stop assigning IDs if we reach the overflow limit and call it a
day
since you need to have a truly horrendous file to reach 4,294,967,295 AST nodes
(I did some tests to check this).

On Tue, 18 May 2021 at 13:25, Pablo Galindo Salgado 
wrote:

> Yet another problem that I found:
>
> One integer is actually not enough to assign IDs. One unsigned integer can
> cover 4,294,967,295 AST nodes, but is technically possible
> to have more than that in a single file. While in PEP 657 we are tracking
> offsets that are normally very low < 100 typically or end lines
> that are easily compressible (as end lines are normally equal to the start
> of the line), node IDs can be arbitrarily big. Which is worse:
> the parser creates some AST nodes that throw away if the parser fails
> (that's how PEG parsers work), so there will be a lot of IDs that
> don't get assigned (and the parser doesn't have an easy way to know how
> many IDs has thrown away). This forces us to either use something
> bigger than an integer (probably a size_t) or to deal with overflow.
>
> On Tue, 18 May 2021 at 10:23, Pablo Galindo Salgado 
> wrote:
>
>> Hu, actually another problem of this approach:
>>
>> Nodes are created and modified after the optimization pass, so the AST
>> produced by the parser is not enough to reconstruct the actual
>> information, we need to also run the optimization passes, but
>> unfortunately, this is (by design) not don't in the Python API (to preserve
>> all possible information about the original code), so this complicates
>> quite a lot the API to get this, as `ast.parse` is not enough to get the
>> original tree.
>>
>> On Tue, 18 May 2021 at 08:53, Pablo Galindo Salgado 
>> wrote:
>>
>>> Hi Nathaniel,
>>>
>>> Thanks a lot for your suggestion! I like the idea although I still think
>>> is more complex than our current proposal, but on the other hand it allows
>>> for a much richer results so I'm quite excited to try it out. We are going
>>> to give it a go to explore it with a prototype and if we are convinced we
>>> will update the PEP.
>>>
>>> One thing I'm not a fan of is that tools that want to leverage this
>>> information (notice our pep also offers this info for tools as an important
>>> aspect of it) will need to reparse the file AND search the AST node, which
>>> also involves transforming the full tree to Python objects, which is even
>>> slower. This is unfortunately a worse API than just get the full location
>>> given an instruction offset. Also, while displaying tracebacks may be a
>>> good scenario for the speed tradeoff, it may not be for other tools.
>>> Finally, if there is no file available all this information is lost,
>>> although one could argue that then is not extremely useful...
>>>
>>> Regards from sunny London,
>>> Pablo Galindo Salgado
>>>
>>> On Tue, 18 May 2021, 01:43 Nathaniel Smith,  wrote:
>>>
 On Mon, May 17, 2021 at 6:18 AM Mark Shannon  wrote:
 > 2. Repeated binary operations on the same line.
 >
 > A single location can also be clearer when all the code is on one
 line.
 >
 > i1 + i2 + s1
 >
 > PEP 657:
 >
 > i1 + i2 + s1
 > 
 >
 > Using a single location:
 >
 > i1 + i2 + s1
 >  ^

 It's true this case is a bit confusing with the whole operation span
 highlighted, but I'm not sure the single location version is much better. I
 feel like a Really Good UI would like, highlight the two operands in
 different colors or something, or at least underline the two separate items
 whose type is incompatible separately:

 TypeError: unsupported operand type(s) for +: 'int' + 'str':
 i1 + i2 + s1
 ^^^   ~~

 More generally, these error messages are the kind of thing where the UI
 can always be tweaked to improve further, and those tweaks can make good
 use of any rich source information that's available.

 So, here's another option to consider:

 - When parsing, assign each AST node a unique, deterministic id (e.g.
 sequentially across the AST tree from top-to-bottom, left-to-right).
 - For each bytecode offset, store the corresponding AST node id in an
 lnotab-like table
 - When displaying a traceback, we already need to go find and read the
 original .py file to print source code at all. Re-parse it, and use the ids
 to find the original AST node, in context with full structure. Let the
 traceback formatter do whatever clever stuff it wants with this info.

 Of course if the .py and .pyc files don't match, this might produce
 gibberish. We already have that problem with showing source lines, but it
 might be even more confusing if we get some random unrelated AST node. This
 could be avoided by storing some kind of hash in the code object, so that

[Python-Dev] Re: Critique of PEP 657 -- Include Fine Grained Error Locations in Tracebacks

[Pablo Galindo Salgado]

Yet another problem that I found:

One integer is actually not enough to assign IDs. One unsigned integer can
cover 4,294,967,295 AST nodes, but is technically possible
to have more than that in a single file. While in PEP 657 we are tracking
offsets that are normally very low < 100 typically or end lines
that are easily compressible (as end lines are normally equal to the start
of the line), node IDs can be arbitrarily big. Which is worse:
the parser creates some AST nodes that throw away if the parser fails
(that's how PEG parsers work), so there will be a lot of IDs that
don't get assigned (and the parser doesn't have an easy way to know how
many IDs has thrown away). This forces us to either use something
bigger than an integer (probably a size_t) or to deal with overflow.

On Tue, 18 May 2021 at 10:23, Pablo Galindo Salgado 
wrote:

> Hu, actually another problem of this approach:
>
> Nodes are created and modified after the optimization pass, so the AST
> produced by the parser is not enough to reconstruct the actual
> information, we need to also run the optimization passes, but
> unfortunately, this is (by design) not don't in the Python API (to preserve
> all possible information about the original code), so this complicates
> quite a lot the API to get this, as `ast.parse` is not enough to get the
> original tree.
>
> On Tue, 18 May 2021 at 08:53, Pablo Galindo Salgado 
> wrote:
>
>> Hi Nathaniel,
>>
>> Thanks a lot for your suggestion! I like the idea although I still think
>> is more complex than our current proposal, but on the other hand it allows
>> for a much richer results so I'm quite excited to try it out. We are going
>> to give it a go to explore it with a prototype and if we are convinced we
>> will update the PEP.
>>
>> One thing I'm not a fan of is that tools that want to leverage this
>> information (notice our pep also offers this info for tools as an important
>> aspect of it) will need to reparse the file AND search the AST node, which
>> also involves transforming the full tree to Python objects, which is even
>> slower. This is unfortunately a worse API than just get the full location
>> given an instruction offset. Also, while displaying tracebacks may be a
>> good scenario for the speed tradeoff, it may not be for other tools.
>> Finally, if there is no file available all this information is lost,
>> although one could argue that then is not extremely useful...
>>
>> Regards from sunny London,
>> Pablo Galindo Salgado
>>
>> On Tue, 18 May 2021, 01:43 Nathaniel Smith,  wrote:
>>
>>> On Mon, May 17, 2021 at 6:18 AM Mark Shannon  wrote:
>>> > 2. Repeated binary operations on the same line.
>>> >
>>> > A single location can also be clearer when all the code is on one line.
>>> >
>>> > i1 + i2 + s1
>>> >
>>> > PEP 657:
>>> >
>>> > i1 + i2 + s1
>>> > 
>>> >
>>> > Using a single location:
>>> >
>>> > i1 + i2 + s1
>>> >  ^
>>>
>>> It's true this case is a bit confusing with the whole operation span
>>> highlighted, but I'm not sure the single location version is much better. I
>>> feel like a Really Good UI would like, highlight the two operands in
>>> different colors or something, or at least underline the two separate items
>>> whose type is incompatible separately:
>>>
>>> TypeError: unsupported operand type(s) for +: 'int' + 'str':
>>> i1 + i2 + s1
>>> ^^^   ~~
>>>
>>> More generally, these error messages are the kind of thing where the UI
>>> can always be tweaked to improve further, and those tweaks can make good
>>> use of any rich source information that's available.
>>>
>>> So, here's another option to consider:
>>>
>>> - When parsing, assign each AST node a unique, deterministic id (e.g.
>>> sequentially across the AST tree from top-to-bottom, left-to-right).
>>> - For each bytecode offset, store the corresponding AST node id in an
>>> lnotab-like table
>>> - When displaying a traceback, we already need to go find and read the
>>> original .py file to print source code at all. Re-parse it, and use the ids
>>> to find the original AST node, in context with full structure. Let the
>>> traceback formatter do whatever clever stuff it wants with this info.
>>>
>>> Of course if the .py and .pyc files don't match, this might produce
>>> gibberish. We already have that problem with showing source lines, but it
>>> might be even more confusing if we get some random unrelated AST node. This
>>> could be avoided by storing some kind of hash in the code object, so that
>>> we can validate the .py file we find hasn't changed (sha512 if we're
>>> feeling fancy, crc32 if we want to save space, either way is probably fine).
>>>
>>> This would make traceback printing more expensive, but only if you want
>>> the fancy features, and traceback printing is already expensive (it does
>>> file I/O!). Usually by the time you're rendering a traceback it's more
>>> important to optimize for human time than CPU time. It would take less
>>> memory than PEP 657, and the 

[Python-Dev] Re: PEP 659: Specializing Adaptive Interpreter

[Stephen J. Turnbull]

Steve Holden writes:
 > On Thu, May 13, 2021 at 11:07 PM Steven D'Aprano 
 > wrote:
 > 
 > > Steve
 > > (one of the other ones)
 > >
 > 
 > We are all other Steves!

+1

There were five Steves (and one Stephanie) in my 6th grade class (of
27).  "Steve, move that " became an idiom 

-- Other Steve since 1994

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/FS6XIKMOD2JIUQINJ6UB7ZRJFCMMR4DN/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Speeding up CPython

[Stephen J. Turnbull]

Abdur-Rahmaan Janhangeer writes:

 > That's why i guess what i am proposing might seem simple

I'm saying that we already have the simple version, spelled

git clone; git checkout main~5000

then

git log -U0 main~5000..main | grep -v '^[-+ ]'

which provides very nice hints for the diligent student, including
description, file, and even line numbers.[1]  Variations on that theme
will provide less detailed hints or you can cheat just a little by
replacing the "| grep" with "; sleep 2; clear".  The possibilities are
endless!

The curated version you propose is, in my "I do that for $DAYJOB"
opinion, very hard to do better than that.

Steve

Footnotes: 
[1]  git is, in my opinion, the 42 of software development.  It is the
answer to life, the universe, and EVERYTHING. :-)

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PFBJQ7I7SDIA225RLAVD6JZ35MNCNP3U/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Critique of PEP 657 -- Include Fine Grained Error Locations in Tracebacks

[Pablo Galindo Salgado]

Hu, actually another problem of this approach:

Nodes are created and modified after the optimization pass, so the AST
produced by the parser is not enough to reconstruct the actual
information, we need to also run the optimization passes, but
unfortunately, this is (by design) not don't in the Python API (to preserve
all possible information about the original code), so this complicates
quite a lot the API to get this, as `ast.parse` is not enough to get the
original tree.

On Tue, 18 May 2021 at 08:53, Pablo Galindo Salgado 
wrote:

> Hi Nathaniel,
>
> Thanks a lot for your suggestion! I like the idea although I still think
> is more complex than our current proposal, but on the other hand it allows
> for a much richer results so I'm quite excited to try it out. We are going
> to give it a go to explore it with a prototype and if we are convinced we
> will update the PEP.
>
> One thing I'm not a fan of is that tools that want to leverage this
> information (notice our pep also offers this info for tools as an important
> aspect of it) will need to reparse the file AND search the AST node, which
> also involves transforming the full tree to Python objects, which is even
> slower. This is unfortunately a worse API than just get the full location
> given an instruction offset. Also, while displaying tracebacks may be a
> good scenario for the speed tradeoff, it may not be for other tools.
> Finally, if there is no file available all this information is lost,
> although one could argue that then is not extremely useful...
>
> Regards from sunny London,
> Pablo Galindo Salgado
>
> On Tue, 18 May 2021, 01:43 Nathaniel Smith,  wrote:
>
>> On Mon, May 17, 2021 at 6:18 AM Mark Shannon  wrote:
>> > 2. Repeated binary operations on the same line.
>> >
>> > A single location can also be clearer when all the code is on one line.
>> >
>> > i1 + i2 + s1
>> >
>> > PEP 657:
>> >
>> > i1 + i2 + s1
>> > 
>> >
>> > Using a single location:
>> >
>> > i1 + i2 + s1
>> >  ^
>>
>> It's true this case is a bit confusing with the whole operation span
>> highlighted, but I'm not sure the single location version is much better. I
>> feel like a Really Good UI would like, highlight the two operands in
>> different colors or something, or at least underline the two separate items
>> whose type is incompatible separately:
>>
>> TypeError: unsupported operand type(s) for +: 'int' + 'str':
>> i1 + i2 + s1
>> ^^^   ~~
>>
>> More generally, these error messages are the kind of thing where the UI
>> can always be tweaked to improve further, and those tweaks can make good
>> use of any rich source information that's available.
>>
>> So, here's another option to consider:
>>
>> - When parsing, assign each AST node a unique, deterministic id (e.g.
>> sequentially across the AST tree from top-to-bottom, left-to-right).
>> - For each bytecode offset, store the corresponding AST node id in an
>> lnotab-like table
>> - When displaying a traceback, we already need to go find and read the
>> original .py file to print source code at all. Re-parse it, and use the ids
>> to find the original AST node, in context with full structure. Let the
>> traceback formatter do whatever clever stuff it wants with this info.
>>
>> Of course if the .py and .pyc files don't match, this might produce
>> gibberish. We already have that problem with showing source lines, but it
>> might be even more confusing if we get some random unrelated AST node. This
>> could be avoided by storing some kind of hash in the code object, so that
>> we can validate the .py file we find hasn't changed (sha512 if we're
>> feeling fancy, crc32 if we want to save space, either way is probably fine).
>>
>> This would make traceback printing more expensive, but only if you want
>> the fancy features, and traceback printing is already expensive (it does
>> file I/O!). Usually by the time you're rendering a traceback it's more
>> important to optimize for human time than CPU time. It would take less
>> memory than PEP 657, and the same as Mark's proposal (both only track one
>> extra integer per bytecode offset). And it would allow for arbitrarily rich
>> traceback display.
>>
>> (I guess in theory you could make this even cheaper by using it to
>> replace lnotab, instead of extending it. But I think keeping lnotab around
>> is a good idea, as a fallback for cases where you can't find the original
>> source but still want some hint at location information.)
>>
>> -n
>>
>> --
>> Nathaniel J. Smith -- https://vorpus.org
>> ___
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-le...@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-dev@python.org/message/BUXFOSAEBXLIHH432PKBCXOGXUAHQIVP/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>

[Python-Dev] Re: PEP 1, PEP Purpose and Guidelines

[gopinathinchennai01]

thank yohttps://www.credosystemz.com/training-in-chennai/best-angularjs-training-in-chennai//;>u
 
it gives detailed information about email address
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/HNDJE6KTBDI3D4W36NZZJL3PS3OLRGVS/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Critique of PEP 657 -- Include Fine Grained Error Locations in Tracebacks

[Pablo Galindo Salgado]

Hi Nathaniel,

Thanks a lot for your suggestion! I like the idea although I still think is
more complex than our current proposal, but on the other hand it allows for
a much richer results so I'm quite excited to try it out. We are going to
give it a go to explore it with a prototype and if we are convinced we will
update the PEP.

One thing I'm not a fan of is that tools that want to leverage this
information (notice our pep also offers this info for tools as an important
aspect of it) will need to reparse the file AND search the AST node, which
also involves transforming the full tree to Python objects, which is even
slower. This is unfortunately a worse API than just get the full location
given an instruction offset. Also, while displaying tracebacks may be a
good scenario for the speed tradeoff, it may not be for other tools.
Finally, if there is no file available all this information is lost,
although one could argue that then is not extremely useful...

Regards from sunny London,
Pablo Galindo Salgado

On Tue, 18 May 2021, 01:43 Nathaniel Smith,  wrote:

> On Mon, May 17, 2021 at 6:18 AM Mark Shannon  wrote:
> > 2. Repeated binary operations on the same line.
> >
> > A single location can also be clearer when all the code is on one line.
> >
> > i1 + i2 + s1
> >
> > PEP 657:
> >
> > i1 + i2 + s1
> > 
> >
> > Using a single location:
> >
> > i1 + i2 + s1
> >  ^
>
> It's true this case is a bit confusing with the whole operation span
> highlighted, but I'm not sure the single location version is much better. I
> feel like a Really Good UI would like, highlight the two operands in
> different colors or something, or at least underline the two separate items
> whose type is incompatible separately:
>
> TypeError: unsupported operand type(s) for +: 'int' + 'str':
> i1 + i2 + s1
> ^^^   ~~
>
> More generally, these error messages are the kind of thing where the UI
> can always be tweaked to improve further, and those tweaks can make good
> use of any rich source information that's available.
>
> So, here's another option to consider:
>
> - When parsing, assign each AST node a unique, deterministic id (e.g.
> sequentially across the AST tree from top-to-bottom, left-to-right).
> - For each bytecode offset, store the corresponding AST node id in an
> lnotab-like table
> - When displaying a traceback, we already need to go find and read the
> original .py file to print source code at all. Re-parse it, and use the ids
> to find the original AST node, in context with full structure. Let the
> traceback formatter do whatever clever stuff it wants with this info.
>
> Of course if the .py and .pyc files don't match, this might produce
> gibberish. We already have that problem with showing source lines, but it
> might be even more confusing if we get some random unrelated AST node. This
> could be avoided by storing some kind of hash in the code object, so that
> we can validate the .py file we find hasn't changed (sha512 if we're
> feeling fancy, crc32 if we want to save space, either way is probably fine).
>
> This would make traceback printing more expensive, but only if you want
> the fancy features, and traceback printing is already expensive (it does
> file I/O!). Usually by the time you're rendering a traceback it's more
> important to optimize for human time than CPU time. It would take less
> memory than PEP 657, and the same as Mark's proposal (both only track one
> extra integer per bytecode offset). And it would allow for arbitrarily rich
> traceback display.
>
> (I guess in theory you could make this even cheaper by using it to replace
> lnotab, instead of extending it. But I think keeping lnotab around is a
> good idea, as a fallback for cases where you can't find the original source
> but still want some hint at location information.)
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/BUXFOSAEBXLIHH432PKBCXOGXUAHQIVP/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/SZPCHKVI32YI6EMRLLHKGTAU6M6VAKJZ/
Code of Conduct: http://python.org/psf/codeofconduct/