[Python-Dev] Restated typing issue for class fields.

[Vincent Risi]

I have written an xsd parser to generate python classes. I also have
written a utility to read in an xml file to populate the python classes.
As well as a utility to write xml from the python classes thus generated.
For example the code below that I include here shows a small sample of the
generated code.
The __annotations__ for these are very easy to use by the utility code for
doing their job.

Using the typing list[x] and tuple[a,b] do not work as well as
- [x] instead of list[x]
- (a, b) instead of tuple(a, b)

I seem to feel list[x] and tuple[a,b] are far more non Python, requiring
having to parse the string they return in a complicated way.

Looping through the annotations to get the field types it's easy to test
for list or tuple and then get the field type for the list or tuple and the
usage for the tuple.

I have been using Python from version 1.6 and have been involved in writing
and generating tons of lines of Python.

ATTRIB, PSEUDO, ASIS = range(3)

class RecordType:
isNew: str
column: (str,ATTRIB)
whizz: (str,PSEUDO)
def __init__(self):
self.isNew = ''
self.whizz = ''
self.column = ''

class MsgTableType:
record: [RecordType]
def __init__(self):
self.record = []

rt = RecordType()
for annote in rt.__annotations__:
print (annote, type(rt.__annotations__[annote]))

mtt = MsgTableType()
for annote in mtt.__annotations__:
print (annote, type(mtt.__annotations__[annote]))
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ANSJXQTUQDGADZRLAV5YFRU74NHUBVBL/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Request to revert unittest and configparser incompatible changes in Python 3.11

[Gregory P. Smith]

On Tue, Jan 18, 2022 at 6:24 AM Victor Stinner  wrote:

> Hi,
>
> My colleagues Tomáš Hrnčiar and Miro Hrončok made good progress on
> updating Python 3.10 to Python 3.11 in Fedora, but some specific
> Python 3.11 incompatible changes are causing more troubles than
> others:
> https://discuss.python.org/t/experience-with-python-3-11-in-fedora/12911
>
> We propose to revert the following 2 changes in Python 3.11 and
> postpone them in a later Python version, once most projects will be
> compatible with these changes:
>
> * Removal of unittest aliases (bpo-45162): it broke 61 Fedora packages
> * Removals from configparser module (bpo-45173) - broke 28 Fedora packages
>
> --
>
> We reported the issue to many affected projects, or when possible, we
> also wrote a pull request to make the code compatible (lot of those
> were made by others, e.g. Hugo van Kemenade, thank you!).
>

+1 to rolling both of these back for 3.11.  Deprecation removals are hard.
Surfacing these to the impacted upstream projects to provide time for those
to integrate the changes is the right way to make these changes stick in
3.12 or later.  Thanks for doing a significant chunk of that work!

As you've done the work to clean up a lot of other OSS projects, I suggest
we defer this until 3.12 with the intent that we won't defer it again. That
doesn't mean we can't hold off on it, just that we believe pushing for this
now and proactively pushing for a bunch of cleanups has improved the state
of the world such that the future is brighter.  That's a much different
strategy than our passive aggressive DeprecationWarnings.

-gps


>
> The problem is that fixing a Fedora package requires multiple steps:
>
> * (1) Propose a pull request upstream
> * (2) Get the pull request merged upstream
> * (3) Wait for a new release upstream
> * (4) Update the Fedora package downstream, or backport the change in
> Fedora (only needed by Fedora)
>
> Identifying the Python incompatible changes causing most troubles took
> us a lot of time, but we did this work. Reverting the two Python 3.11
> incompatible changes (causing most troubles) will save us "bug triage"
> time, to get more time on updating projects to Python 3.11 for the
> other remaining incompatible changes.
>
> We are not saying that these incompatible changes are bad, it's just a
> matter of getting most projects ready before Python 3.11 final will be
> released.
>
> --
>
> By the way, before making a change known to be incompatible, it would
> be nice to run a code search on PyPI top 5000 projects to estimate how
> many projects will be broken, and try to update these projects
> *before* making the change.
>
> For example, only introduce an incompatible change into Python once
> less than 15 projects are affected. Getting a pull request merged is
> nice, but a release including the fix is way better for practical
> reasons: people use "pip install ".
>
> --
>
> Fedora work on Python 3.11 is public and tracked at:
> https://bugzilla.redhat.com/show_bug.cgi?id=PYTHON3.11
>
> Click on "depends on" to see current issues:
>
> https://bugzilla.redhat.com/buglist.cgi?bug_id=2016048_id_type=anddependson=tvp
>
> Example of bz #2025600: mom fails to build with Python 3.11:
> AttributeError: module 'configparser' has no attribute
> 'SafeConfigParser'.
>
>
> Victor Stinner -- in the name of the Python Red Hat / Fedora maintenance
> team
> --
> Night gathers, and now my watch begins. It shall not end until my death.
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/GJTREADEXYAETECE5JDTPYWK4WMTKYGR/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/3J3VKNTKGPWACFVDWPRCS7FNED2A34R4/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Request to revert unittest and configparser incompatible changes in Python 3.11

[Gregory P. Smith]

On Tue, Jan 18, 2022 at 10:58 PM Christopher Barker 
wrote:

> On Tue, Jan 18, 2022 at 10:30 AM Brett Cannon  wrote:
>
>>  I remember that "noisy by default" deprecation warnings were widely
>>> despised.
>>>
>>> One thought, what if they were off by default UNLESS you were doing unit
>>> tests?
>>>
>>
>> I believe pytest already does this.
>>
>
> Indeed it does, at least in recent versions (1-2 yrs ago?)
>
> And even that is pretty darn annoying. It's really helpful for my code,
> but they often get lost in the noise of all the ones I get from upstream
> packages.
>
> I suppose I need to take the time to figure out how to silence the ones I
> don't want.
>
> And it does prompt me to make sure that the upstream packages are working
> on it.
>
> Now we just need to get more people to use pytest :-)
>

Our stdlib unittest already enables warnings by default per
https://bugs.python.org/issue10535.

Getting the right people to pay attention to them is always the hard part.


>
> -CHB
>
> --
> Christopher Barker, PhD (Chris)
>
> Python Language Consulting
>   - Teaching
>   - Scientific Software Development
>   - Desktop GUI and Web Development
>   - wxPython, numpy, scipy, Cython
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/LZAF3TOOIAAFXXK2KPAXA5V5SRBOSIIP/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5YOEMAIVXMVWCORAY54LZQO62755HQGX/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Request to revert unittest and configparser incompatible changes in Python 3.11

[Christopher Barker]

On Tue, Jan 18, 2022 at 10:30 AM Brett Cannon  wrote:

>  I remember that "noisy by default" deprecation warnings were widely
>> despised.
>>
>> One thought, what if they were off by default UNLESS you were doing unit
>> tests?
>>
>
> I believe pytest already does this.
>

Indeed it does, at least in recent versions (1-2 yrs ago?)

And even that is pretty darn annoying. It's really helpful for my code, but
they often get lost in the noise of all the ones I get from upstream
packages.

I suppose I need to take the time to figure out how to silence the ones I
don't want.

And it does prompt me to make sure that the upstream packages are working
on it.

Now we just need to get more people to use pytest :-)

-CHB

-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LZAF3TOOIAAFXXK2KPAXA5V5SRBOSIIP/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Minor inconvenience: f-string not recognized as docstring

[Jim J. Jewett]

Guido van Rossum wrote:
> I personally think F-strings should not be usable as docstrings. If you
> want a dynamically calculated docstring you should assign it dynamically,
> not smuggle it in using a string-like expression. We don't allow "blah {x}
> blah".format(x=1) as a docstring either, not "foo %s bar" % x.

Nor, last I checked, even "string1" + "string2", even though the result is a 
compile-time string in the appropriate location.  I think all of these should 
be allowed, but I'll grant that annotations reduce the need.  I'll even admit 
that scoping issues make the interpolating versions error prone, and the UI to 
clear that up may be more of a hassle than it is worth. 

-jJ
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZUWTCGK6KZJYCUDRR3JNB7H5W3ZHJWMT/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Suggestion: a little language for type definitions

[Jim J. Jewett]

Steven D'Aprano wrote:
> On Sat, Jan 08, 2022 at 12:59:38AM +0100, jack.jan...@cwi.nl wrote:

> For example, the arrow syntax for Callable `(int) -> str` (if accepted) 
> could be a plain old Python expression, usable anywhere the plain old 
> Python expression `Callable[[int], str]` would be.

In principle, yes.  In practice, I think the precedence of "->" might be 
tricky, particularly if the (int) part discourages people from wrapping the 
full expression in parentheses. 

> > What if we created a little language that is clearly flagged, for 
> > example as t”….” or t’….’? Then we could simply define the 
> > typestring language to be readable, so you could indeed say t”(int, 
> > str) -> bool”. And we could even allow escapes (similar to 
> > f-strings) so that the previous expression could also be specified, 
> > if you really wanted to, as t”{typing.Callable[[int, str], bool}”.

> The following are not rhetorical questions. I don't know the answers, 
> which is why I am asking.

> 1. Are these t-strings only usable inside annotations, or are they 
> expressions that are usable everywhere?

I assume they *could* be used anywhere, there just wouldn't be huge reasons to 
do so.  Sort of like a string expression can be an entire statement; there just 
usually isn't much reason (except as a docstring) to do it.

> 2. If only usable inside annotations, why bother with the extra prefix 
> t" and suffix "? What benefit do they give versus just the rule 
> "annotations use this syntax"?

It provides a useful box around the typing, so that people who are not 
currently worried about typing can more easily concentrate on the portion they 
do currently care about.

> 3. If usable outside of annotations, what runtime effect do they have? 

They create a string.  Which may or may not be a useful thing to do.

> The t-string must evaluate to an object. What object?

A string.  The various "let us delay annotation evaluation" proposals have made 
it clear that the people actually using typing don't want it to slow things 
down for extra evaluation until they explicitly call for that evaluation, 
perhaps as part of a special run.

> 4. If the syntax allowed inside the t-string is specified as part of the 
> Python language definition, why do we need the prefix and suffix?

Same answer as number 2 ... it allows typing to be a less intrusive neighbor.  
I don't think t" " is as good as some sort of braces, but ... we're out of 
conventional braces available in ASCII.

> Likewise, if this is allowed:
> def func(arr: t"array [1...10] of int") -> str: ...

How many arguments do I pass to func?  That is already tricky to see at a 
glance, but 

> def func(arr: array [1...10] of int) -> str: ...

is even more difficult to parse.  By the time I've mentally attached the "of" 
and "int" to the indexed (but not really) array that just describes a type, 
I've forgotten what I was looking for and why.

> 5. What difference, if any, is there between `t"{expression}"` and 
> `expression`?

In addition to the box (so readers can more easily filter it out), there is 
also a flag to typing systems saying that they *should* elaborate the string.  
What they elaborate it into will be very different from a regular string.  That 
won't happen every time the module is imported, but it will happen when the 
string is actually needed for something.

-jJ
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/HY7I52ILYN7IYN2S6UQT57XV4R3YEC2Z/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: 3.11 enhanced error location - can it be smarter?

[MRAB]

On 2022-01-18 21:58, Pablo Galindo Salgado wrote:
The code that computes the lines is already quite complex (to the point 
that has to do some AST analysis and post-parsing) so I am quite worried 
to introduce a lot of complexity in this area. I am fine doing something 
that we can easily check for (spawns all the line) but I would be 
against having to start doing even more analysis (especially for things 
that are not AST-based).


Take into account that code that displays exceptions need to be very 
resilient because it can be called in some tricky situations and 
validating that all the code works correctly is very hard, and complex 
to maintain.


As I said, I think I would be supportive of considering adding a check 
for the full line, but I think that adding more complexity here is quite 
dangerous.


It might be enough to check that there's only whitespace before the 
first ^ and only whitespace optionally followed by a comment (#) after 
the last ^ on that line.


On Tue, 18 Jan 2022 at 21:49, Patrick Reader <_...@pxeger.com 
> wrote:


On 18/01/2022 20:41, Pablo Galindo Salgado wrote:

We cannot base the computation on a % because is possible that the
location markers are relevant
but the variables, function names or constructs are just very
large. I think that the idea of "spawns
the whole line" is sensible, though.

On Tue, 18 Jan 2022 at 20:32, Steve Dower mailto:steve.do...@python.org>> wrote:


Omitting the line of ^ where over x% (75%? 90%?) of characters
on the
line would be marked would be fine by me.


It would also need to take into account cases where a line contains
an inline comment, which does not contribute to the code producing
the error, but where all of the rest of the line (the code) is the
source of the error and highlighting it is not useful

# test.py:

code_that_causes_an_error # a comment

$ python3.11 test.py

Traceback (most recent call last):
   File "test.py", line 1, in 
     code_that_causes_an_error # a comment
     ^
NameError: name 'code_that_causes_an_error' is not defined

(the traceback shown is from a current `main` build)


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/M62NSJW3S4Y4LNEZRJA4VJ3WXIPGHOVU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 646 (Variadic Generics): final call for comments

[Glenn Linderman]

On 1/18/2022 2:35 PM, Jim J. Jewett wrote:

The problem is that
 [*s3, *s4] = (a, b, 1, 2, 3)
is ambiguous ...

It wouldn't have to be... but as you say, it needs to be explicit.

*s3 could get all the content, and *s4 a tuple with no members.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CNTEQGZRHLPODTIR74SUHD7OWSEZ4FL3/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 646 (Variadic Generics): final call for comments

[Jim J. Jewett]

I'm seeing enough different interpretations to think things aren't quite 
specified -- but I'm not sure if it matters.

(1)  Is any of this something that should affect computation, or is it really 
just a question of how to interpret possibly ambiguous documentation? 

(2)  Are any of these troubling cases something that a person should actually 
write for a normal situation?  Or are they just arguments about which 
abbreviations are acceptable?  Or about how automatically-generated (inferred) 
type descriptions should be written?

(3)  Are the slice-expansion questions all assumed to be indexing an 
n-dimensional array, as opposed to [start, stop, step]?  Is that explicit in 
the PEP, and just not in the extracts here?

(4)  Expanding multiple * shouldn't be ambiguous; the problem is figuring out 
what to condense into which if two are adjacent.  So 
s1, s2 =[a,b], (1,2,3)
[*s1, *s2] should turn into [a, b, 1, 2, 3]
The problem is that 
[*s3, *s4] = (a, b, 1, 2, 3)
is ambiguous ... and I didn't really get that distinction from Petr's question 
or the answers.  I can't tell whether I've missed something crucial, or others 
are arguing over angels on a pinhead ... so whatever the PEP ends up deciding, 
it should be explicit.  (I *think* the earlier parts of this thread are 
consistent with this, and discussing whether to say explicitly that certain 
formats are forbidden (but maybe not enforced by the grammar), meaningless, or 
valid but currently meaningless outside of typing.)

-jJ
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YH4URO5EDQODG4QMGOCSXHV6RYTMLK5M/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: 3.11 enhanced error location - can it be smarter?

[Pablo Galindo Salgado]

The code that computes the lines is already quite complex (to the point
that has to do some AST analysis and post-parsing) so I am quite worried to
introduce a lot of complexity in this area. I am fine doing something that
we can easily check for (spawns all the line) but I would be against having
to start doing even more analysis (especially for things that are not
AST-based).

Take into account that code that displays exceptions need to be very
resilient because it can be called in some tricky situations and validating
that all the code works correctly is very hard, and complex to maintain.

As I said, I think I would be supportive of considering adding a check for
the full line, but I think that adding more complexity here is quite
dangerous.

On Tue, 18 Jan 2022 at 21:49, Patrick Reader <_...@pxeger.com> wrote:

> On 18/01/2022 20:41, Pablo Galindo Salgado wrote:
>
> We cannot base the computation on a % because is possible that the
> location markers are relevant
> but the variables, function names or constructs are just very large. I
> think that the idea of "spawns
> the whole line" is sensible, though.
>
> On Tue, 18 Jan 2022 at 20:32, Steve Dower  wrote:
>
>>
>> Omitting the line of ^ where over x% (75%? 90%?) of characters on the
>> line would be marked would be fine by me.
>
> It would also need to take into account cases where a line contains an
> inline comment, which does not contribute to the code producing the error,
> but where all of the rest of the line (the code) is the source of the error
> and highlighting it is not useful
>
> # test.py:
>
> code_that_causes_an_error # a comment
>
> $ python3.11 test.py
>
> Traceback (most recent call last):
>   File "test.py", line 1, in 
> code_that_causes_an_error # a comment
> ^
> NameError: name 'code_that_causes_an_error' is not defined
>
> (the traceback shown is from a current `main` build)
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/F4FSJY7OIPHAH5I6YBHCFI4DP3XLNUJW/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/G2D723CIGLMRRYJ43CGTTDHWXMRGOFIN/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Request to revert unittest and configparser incompatible changes in Python 3.11

[Victor Stinner]

On Tue, Jan 18, 2022 at 5:48 PM Steve Dower  wrote:
> Discovering during alpha that some packages haven't updated for the
> release that hasn't happened yet isn't the end of the world. Reverting
> the changes now is probably a bit premature - realistically we can undo
> these anytime during beta if we discover that packages are unable to be
> fixed over the next 9 months.

In Fedora, if a frequently used dependency is broken, a long list of
packages "fail to build". (In Fedora, the package test suite must pass
to build a package successfully.) If it takes 9 months to fix this
dependency, we will likely miss other issues before the Python final
version in dependent packages. The problem is when the issue is on the
Python side.

For example, we discovered that cloudpickle didn't work on Python
3.10. In fact, it was a Python regression. I had to modify how the
function object constructor inherits globals (bpo-42990). This kind of
change is fine during the beta phase, but it's bad to change the
Python behavior in a bugfix Python 3.10.x release :-(

Another example is when Cython was ported to Python 3.8, a
PyCode_New() change caused a lot of troubles. After multiple exchanges
between Cython and Python and a few more issues, it was decided to
*revert* the PyCode_New() change. I would not be possible to revert
the change in Python 3.8.1 or later :-(

There are many examples like that, I don't have an exhaustive list. If
issues are discovered earlier, we get more time to discuss and design
how to handle them.

I don't know where is the good trade-off here. That's why I'm opening
a discussion. Yes obviously, we can revert these 2 changes later. Or
we can just keep them ;-)

--

At the end of my first email, I also suggest thinking about
incompatible changes differently, try to make affected projects
compatible in advance. The problem are not the changes themselves, but
how they are introduced in Python, and more globally how they are
introduced "in the Python ecosystem" (!).

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/DI7FCMTSL2MGIZAAJ6I7QRJYTQQQ37VP/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: 3.11 enhanced error location - can it be smarter?

[Patrick Reader]

On 18/01/2022 20:41, Pablo Galindo Salgado wrote:
We cannot base the computation on a % because is possible that the 
location markers are relevant
but the variables, function names or constructs are just very large. I 
think that the idea of "spawns

the whole line" is sensible, though.

On Tue, 18 Jan 2022 at 20:32, Steve Dower  wrote:


Omitting the line of ^ where over x% (75%? 90%?) of characters on the
line would be marked would be fine by me.

It would also need to take into account cases where a line contains an 
inline comment, which does not contribute to the code producing the 
error, but where all of the rest of the line (the code) is the source of 
the error and highlighting it is not useful


# test.py:

code_that_causes_an_error # a comment

$ python3.11 test.py

Traceback (most recent call last):
  File "test.py", line 1, in 
    code_that_causes_an_error # a comment
    ^
NameError: name 'code_that_causes_an_error' is not defined

(the traceback shown is from a current `main` build)
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/F4FSJY7OIPHAH5I6YBHCFI4DP3XLNUJW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: 3.11 enhanced error location - can it be smarter?

[Barney Gale]

tox and pytest look at PY_COLORS:

https://github.com/tox-dev/tox/blob/9cc692d85c9ce84344ea7fee4b127755c6099a32/src/tox/session/commands/help.py
https://docs.pytest.org/en/6.2.x/reference.html#envvar-PY_COLORS

On Tue, 18 Jan 2022 at 21:17, Ethan Furman  wrote:

> On 1/18/22 11:59 AM, Pablo Galindo Salgado wrote:
>
>  > We considered using colours and other markers such as bold text, but
> that opens a considerable can of worms with
>  > detecting in all systems and configurations if that can be done. I have
> been told that some of these situations are
>  > quite tricky and is not as easy as checking for tty support.
>
> Maybe use an environment variable?
>
> --
> ~Ethan~
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/TRI25YRCTJSDQKERZM2DHCWOMGXHQA3M/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TH5HN7SKS6FV4QGB44IBN66VNHLLDMQ6/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: 3.11 enhanced error location - can it be smarter?

[Ethan Furman]

On 1/18/22 11:59 AM, Pablo Galindo Salgado wrote:

> We considered using colours and other markers such as bold text, but that 
opens a considerable can of worms with
> detecting in all systems and configurations if that can be done. I have been 
told that some of these situations are
> quite tricky and is not as easy as checking for tty support.

Maybe use an environment variable?

--
~Ethan~
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TRI25YRCTJSDQKERZM2DHCWOMGXHQA3M/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: 3.11 enhanced error location - can it be smarter?

[Pablo Galindo Salgado]

We cannot base the computation on a % because is possible that the location
markers are relevant
but the variables, function names or constructs are just very large. I
think that the idea of "spawns
the whole line" is sensible, though.

On Tue, 18 Jan 2022 at 20:32, Steve Dower  wrote:

> On 1/18/2022 7:59 PM, Pablo Galindo Salgado wrote:
> > We considered using colours and other markers such as bold text, but
> > that opens a considerable can of worms with detecting in all systems and
> > configurations if that can be done. I have been told that some of these
> > situations are quite tricky and is not as easy as checking for tty
> support.
> >
> > If someone wants to contribute a way to detect reliably (in C) if the
> > terminal supports ANSI colours, I'm happy to consider switching to that
> > instead of the underlying.
>
> I'll save some time - no such mechanism exists on Windows. There's no
> reliable way to detect what's displaying console output, and while many
> now support ANSI colours, we can't be assured of it.
>
> Omitting the line of ^ where over x% (75%? 90%?) of characters on the
> line would be marked would be fine by me.
>
> Cheers,
> Steve
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/DA7KDNGCWQXHQCFDJ3FNXMWTYPLPXOYC/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: 3.11 enhanced error location - can it be smarter?

[Steve Dower]

On 1/18/2022 7:59 PM, Pablo Galindo Salgado wrote:
We considered using colours and other markers such as bold text, but 
that opens a considerable can of worms with detecting in all systems and 
configurations if that can be done. I have been told that some of these 
situations are quite tricky and is not as easy as checking for tty support.


If someone wants to contribute a way to detect reliably (in C) if the 
terminal supports ANSI colours, I'm happy to consider switching to that 
instead of the underlying.


I'll save some time - no such mechanism exists on Windows. There's no 
reliable way to detect what's displaying console output, and while many 
now support ANSI colours, we can't be assured of it.


Omitting the line of ^ where over x% (75%? 90%?) of characters on the 
line would be marked would be fine by me.


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ICDRA5LFN4YZ45GBQM5E5GYDEWCSWT4I/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: 3.11 enhanced error location - can it be smarter?

[Pablo Galindo Salgado]

We considered using colours and other markers such as bold text, but that
opens a considerable can of worms with detecting in all systems and
configurations if that can be done. I have been told that some of these
situations are quite tricky and is not as easy as checking for tty support.

If someone wants to contribute a way to detect reliably (in C) if the
terminal supports ANSI colours, I'm happy to consider switching to that
instead of the underlying.

On Tue, 18 Jan 2022, 19:13 Ethan Furman,  wrote:

> On 1/18/22 10:43 AM, Mats Wichmann wrote:
>
>  > A thought - how about omitting the underline line if the
>  > to-be-underlined part would be the whole line?
>
> I would also like that change -- when the underlining is a portion of the
> whole it's quite useful, but when it's the
> whole line it's a lot of extra noise.
>
> In fact, if I can be permitted to dream for a moment, what would be really
> nice is using highlighting or bolding or some
> such and skipping the extra underline lines (where "underlining" means
> "extra line with pointy hats").
>
> ~Ethan~
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/JYC5D3L6QW7V3ZOWMTBLYIGUQ6UOMS2U/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PVQOEYH4JK5I4LP4JKREYC63R4CI2KML/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 677 (Callable Type Syntax): Final Call for Comments

[Steven Troxler]

We just merged changes to the PEP with a detailed runtime API specification.

Highlights are that:
- The new type should have an `inspect.Signature`- inspired structured API
- It should have a backward-compatible interface with the `__args__` and 
`__params__`
- As with the PEP 604 union syntax, `__eq__` should treat `typing.Callable` and 
equivalent builtin callable types as equal
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UB6VLDVDSCHWOGEGTPZAOX6BQFVZRHEK/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: 3.11 enhanced error location - can it be smarter?

[Ethan Furman]

On 1/18/22 10:43 AM, Mats Wichmann wrote:

> A thought - how about omitting the underline line if the
> to-be-underlined part would be the whole line?

I would also like that change -- when the underlining is a portion of the whole it's quite useful, but when it's the 
whole line it's a lot of extra noise.


In fact, if I can be permitted to dream for a moment, what would be really nice is using highlighting or bolding or some 
such and skipping the extra underline lines (where "underlining" means "extra line with pointy hats").


~Ethan~
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JYC5D3L6QW7V3ZOWMTBLYIGUQ6UOMS2U/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] 3.11 enhanced error location - can it be smarter?

[Mats Wichmann]

"When printing tracebacks, the interpreter will now point to the exact
expression that caused the error instead of just the line."

I get the motivation for better error messages, but there are scenarios
where you can't provide useful new information.  I've been lax in
getting to testing the project I work on with 3.11, but here's some spew
from one testcase (this is an intentionally triggered error - the
testcase is testing predictable behavior in the face of this error, 3.11
didn't *cause* this, as can be fairly easily seen):

Traceback (most recent call last):
  File "C:\Users\mats\Documents\github\scons\SCons\Script\Main.py", line
1403, in main
_exec_main(parser, values)
^^
  File "C:\Users\mats\Documents\github\scons\SCons\Script\Main.py", line
1366, in _exec_main
_main(parser)
^
  File "C:\Users\mats\Documents\github\scons\SCons\Script\Main.py", line
1034, in _main
SCons.Script._SConscript._SConscript(fs, script)

  File
"C:\Users\mats\Documents\github\scons\SCons\Script\SConscript.py", line
285, in _SConscript
exec(compile(scriptdata, scriptname, 'exec'), call_stack[-1].globals)
^
  File
"C:\Users\mats\AppData\Local\Temp\testcmd.16164.fpv9xt5b\SConstruct",
line 4, in 
raise InternalError('error inside')
^^^
SCons.Errors.InternalError: error inside


The grump here is all those pointy hats added absolutely zero new
information, because in each case the entire line is underlined.  So the
"benefit" of the change is the traceback went from 12 lines to 17. In
many cases tracebacks are much longer and adding lots of not-useful
lines is not a great thing.

A thought - how about omitting the underline line if the
to-be-underlined part would be the whole line?
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ST3PMOO6JPOYKI2EJAAFL5X2SZL353R6/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Request to revert unittest and configparser incompatible changes in Python 3.11

[Brett Cannon]

On Tue, Jan 18, 2022 at 9:43 AM Richard Damon 
wrote:

>
>
> On Jan 18, 2022, at 11:34 AM, Guido van Rossum  wrote:
>
> 
> At best it shows that deprecations are complicated no matter how well you
> plan them. I remember that "noisy by default" deprecation warnings were
> widely despised.
>
> On Tue, Jan 18, 2022 at 6:49 AM Antoine Pitrou  wrote:
>
>> On Tue, 18 Jan 2022 15:17:41 +0100
>> Victor Stinner  wrote:
>> > Hi,
>> >
>> > My colleagues Tomáš Hrnčiar and Miro Hrončok made good progress on
>> > updating Python 3.10 to Python 3.11 in Fedora, but some specific
>> > Python 3.11 incompatible changes are causing more troubles than
>> > others:
>> >
>> https://discuss.python.org/t/experience-with-python-3-11-in-fedora/12911
>> >
>> > We propose to revert the following 2 changes in Python 3.11 and
>> > postpone them in a later Python version, once most projects will be
>> > compatible with these changes:
>> >
>> > * Removal of unittest aliases (bpo-45162): it broke 61 Fedora packages
>> > * Removals from configparser module (bpo-45173) - broke 28 Fedora
>> packages
>>
>> Doesn't this show, once again, that making DeprecationWarning silent by
>> default was a mistake?
>>
>>
> --
> --Guido van Rossum (python.org/~guido)
> *Pronouns: he/him **(why is my pronoun here?)
> *
>
>
> One thought, what if they were off by default UNLESS you were doing unit
> tests?
>

I believe pytest already does this.

-Brett


>
> That would cut out a lot of the excessive noise issue, but put them in
> before the programmer when they are developing. Yes, they will get the
> warnings for dependencies but that lets the programmer apply pressure to
> fix it or warning they may need to change something if it won’t get fixed.
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/UAHZKJNQTMXDORVGX5U5EHKI4HQAV7BN/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BSGVVVO2DLPUZPU37ETVYYZLNNL3VEF4/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Request to revert unittest and configparser incompatible changes in Python 3.11

[Richard Damon]

> On Jan 18, 2022, at 11:34 AM, Guido van Rossum  wrote:
> 
> 
> At best it shows that deprecations are complicated no matter how well you 
> plan them. I remember that "noisy by default" deprecation warnings were 
> widely despised.
> 
>> On Tue, Jan 18, 2022 at 6:49 AM Antoine Pitrou  wrote:
>> On Tue, 18 Jan 2022 15:17:41 +0100
>> Victor Stinner  wrote:
>> > Hi,
>> > 
>> > My colleagues Tomáš Hrnčiar and Miro Hrončok made good progress on
>> > updating Python 3.10 to Python 3.11 in Fedora, but some specific
>> > Python 3.11 incompatible changes are causing more troubles than
>> > others:
>> > https://discuss.python.org/t/experience-with-python-3-11-in-fedora/12911
>> > 
>> > We propose to revert the following 2 changes in Python 3.11 and
>> > postpone them in a later Python version, once most projects will be
>> > compatible with these changes:
>> > 
>> > * Removal of unittest aliases (bpo-45162): it broke 61 Fedora packages
>> > * Removals from configparser module (bpo-45173) - broke 28 Fedora packages
>> 
>> Doesn't this show, once again, that making DeprecationWarning silent by
>> default was a mistake?
>> 
> 
> -- 
> --Guido van Rossum (python.org/~guido)
> Pronouns: he/him (why is my pronoun here?)

One thought, what if they were off by default UNLESS you were doing unit tests?

That would cut out a lot of the excessive noise issue, but put them in before 
the programmer when they are developing. Yes, they will get the warnings for 
dependencies but that lets the programmer apply pressure to fix it or warning 
they may need to change something if it won’t get fixed.___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UAHZKJNQTMXDORVGX5U5EHKI4HQAV7BN/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Request to revert unittest and configparser incompatible changes in Python 3.11

[Steve Dower]

On 1/18/2022 2:44 PM, Antoine Pitrou wrote:

We propose to revert the following 2 changes in Python 3.11 and
postpone them in a later Python version, once most projects will be
compatible with these changes:

* Removal of unittest aliases (bpo-45162): it broke 61 Fedora packages
* Removals from configparser module (bpo-45173) - broke 28 Fedora packages


Doesn't this show, once again, that making DeprecationWarning silent by
default was a mistake?


That's a hypothesis, but we don't have enough information to prove it. 
If we'd gone the other way, perhaps we'd be looking at massive 
complaints from "regular" end users about all the noisy warnings that 
they can't fix and saying that making it noisy was the mistake.


Discovering during alpha that some packages haven't updated for the 
release that hasn't happened yet isn't the end of the world. Reverting 
the changes now is probably a bit premature - realistically we can undo 
these anytime during beta if we discover that packages are unable to be 
fixed over the next 9 months.


That said, I'm fine with reverting the changes on the basis that they 
cause churn for no real benefit. If someone wants to argue that the 
benefit is worthwhile, that's fine, as long as they also argue in favour 
of the churn.


We shouldn't pretend to be surprised that something we changed causes 
others to have to change. We *know* that will happen. Either we push 
forward with the changes, or we admit we don't really need them. With 
this amount of time before the release, we can't blame downstream users 
for reverting it.


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PGR4WUPMVGF67IWX2O7EKA2NSHFYEX5P/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Request to revert unittest and configparser incompatible changes in Python 3.11

[Guido van Rossum]

At best it shows that deprecations are complicated no matter how well you
plan them. I remember that "noisy by default" deprecation warnings were
widely despised.

On Tue, Jan 18, 2022 at 6:49 AM Antoine Pitrou  wrote:

> On Tue, 18 Jan 2022 15:17:41 +0100
> Victor Stinner  wrote:
> > Hi,
> >
> > My colleagues Tomáš Hrnčiar and Miro Hrončok made good progress on
> > updating Python 3.10 to Python 3.11 in Fedora, but some specific
> > Python 3.11 incompatible changes are causing more troubles than
> > others:
> > https://discuss.python.org/t/experience-with-python-3-11-in-fedora/12911
> >
> > We propose to revert the following 2 changes in Python 3.11 and
> > postpone them in a later Python version, once most projects will be
> > compatible with these changes:
> >
> > * Removal of unittest aliases (bpo-45162): it broke 61 Fedora packages
> > * Removals from configparser module (bpo-45173) - broke 28 Fedora
> packages
>
> Doesn't this show, once again, that making DeprecationWarning silent by
> default was a mistake?
>
>
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/AQHDYW4RBXV7U5YDZVHE2I2BWEZAC6PB/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ISKNUHFT4W2GKUEIOL5HQTXBMJVQXK2M/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Request to revert unittest and configparser incompatible changes in Python 3.11

[Antoine Pitrou]

On Tue, 18 Jan 2022 15:17:41 +0100
Victor Stinner  wrote:
> Hi,
> 
> My colleagues Tomáš Hrnčiar and Miro Hrončok made good progress on
> updating Python 3.10 to Python 3.11 in Fedora, but some specific
> Python 3.11 incompatible changes are causing more troubles than
> others:
> https://discuss.python.org/t/experience-with-python-3-11-in-fedora/12911
> 
> We propose to revert the following 2 changes in Python 3.11 and
> postpone them in a later Python version, once most projects will be
> compatible with these changes:
> 
> * Removal of unittest aliases (bpo-45162): it broke 61 Fedora packages
> * Removals from configparser module (bpo-45173) - broke 28 Fedora packages

Doesn't this show, once again, that making DeprecationWarning silent by
default was a mistake?



___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/AQHDYW4RBXV7U5YDZVHE2I2BWEZAC6PB/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Request to revert unittest and configparser incompatible changes in Python 3.11

[Victor Stinner]

Hi,

My colleagues Tomáš Hrnčiar and Miro Hrončok made good progress on
updating Python 3.10 to Python 3.11 in Fedora, but some specific
Python 3.11 incompatible changes are causing more troubles than
others:
https://discuss.python.org/t/experience-with-python-3-11-in-fedora/12911

We propose to revert the following 2 changes in Python 3.11 and
postpone them in a later Python version, once most projects will be
compatible with these changes:

* Removal of unittest aliases (bpo-45162): it broke 61 Fedora packages
* Removals from configparser module (bpo-45173) - broke 28 Fedora packages

--

We reported the issue to many affected projects, or when possible, we
also wrote a pull request to make the code compatible (lot of those
were made by others, e.g. Hugo van Kemenade, thank you!).

The problem is that fixing a Fedora package requires multiple steps:

* (1) Propose a pull request upstream
* (2) Get the pull request merged upstream
* (3) Wait for a new release upstream
* (4) Update the Fedora package downstream, or backport the change in
Fedora (only needed by Fedora)

Identifying the Python incompatible changes causing most troubles took
us a lot of time, but we did this work. Reverting the two Python 3.11
incompatible changes (causing most troubles) will save us "bug triage"
time, to get more time on updating projects to Python 3.11 for the
other remaining incompatible changes.

We are not saying that these incompatible changes are bad, it's just a
matter of getting most projects ready before Python 3.11 final will be
released.

--

By the way, before making a change known to be incompatible, it would
be nice to run a code search on PyPI top 5000 projects to estimate how
many projects will be broken, and try to update these projects
*before* making the change.

For example, only introduce an incompatible change into Python once
less than 15 projects are affected. Getting a pull request merged is
nice, but a release including the fix is way better for practical
reasons: people use "pip install ".

--

Fedora work on Python 3.11 is public and tracked at:
https://bugzilla.redhat.com/show_bug.cgi?id=PYTHON3.11

Click on "depends on" to see current issues:
https://bugzilla.redhat.com/buglist.cgi?bug_id=2016048_id_type=anddependson=tvp

Example of bz #2025600: mom fails to build with Python 3.11:
AttributeError: module 'configparser' has no attribute
'SafeConfigParser'.


Victor Stinner -- in the name of the Python Red Hat / Fedora maintenance team
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/GJTREADEXYAETECE5JDTPYWK4WMTKYGR/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Discussion about crash tolerance feature for gdbm module

[Victor Stinner]

How does someone know if a database is corrupted? Why is a separated
script needed?

A single script cannot automatically detect a corrupted database and
load the latest snapshot?

How is different from simply copying the whole database file?

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/F4EKQAU7YLO6WDIYYNS4QF2DXVJ2LIPX/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Discussion about crash tolerance feature for gdbm module

[Dong-hee Na]

Hello Victor :)

> Do you ask us our opinion on the Python API that you propose? Or if
the whole feature is worth it?

Latter must be done first :)


> Can you please show me a short full example opening a database with
automatic snapshot recovery (with your proposed API)?

The following codes are demonstrating how the user sets the gdbm file to be
able to use the crash tolerance feature.

import dbm.gnu as dbm

db = dbm.open('x.db', 'nx') # For extension format
db.gdbm_failure_atomic('even_snapshot.bin', 'odd_snapshot.bin') # For
snapshot declaration
for k, v in zip('abcdef', 'ghijkl'):
db[k] = v

db.sync()
db.close()


The recovery task will be done separately because this situation is a
very special case.

import dbm.gnu as dbm

latest_snapshot = dbm.gdbm_latest_snapshot('even_snapshot.bin',
'odd_snapshot.bin')

db = dbm.open(latest_snapshot, 'r') # Open the latest valid snapshot

# Do what user want, os.rename(latest_snapshot, 'x.db') whatever.


Warm regards,

Dong-hee


2022년 1월 18일 (화) 오후 6:57, Victor Stinner 님이 작성:

> Hi Dong-hee,
>
> Can you please show me a short full example opening a database with
> automatic snapshop recovery (with your proposed API)?
>
> Do you ask us our opinion on the Python API that you propose? Or if
> the whole feature is worth it?
>
> Victor
>
> On Tue, Jan 18, 2022 at 2:41 AM Dong-hee Na  wrote:
> >
> > Hi folks,
> > From gdbm 1.21, gdbm supports the crash tolerance feature.
> > see: https://www.gnu.org.ua/software/gdbm/manual/Crash-Tolerance.html
> >
> > I would like to introduce this feature since python standard library is
> the only gdbm binding library that is available for end-users.
> > And this is also effort for end-users to provide the latest new features.
> > If this feature is added following methods will be added.
> >   - 'x' flag will be added for extended gdbm format.
> >   - gdbm.gdbm_failure_atomic('path1', 'path2') API will be added for
> snapshot paths.
> >   - gdbm.gdbm_latest_snapshot('path1', 'path2') API will be added for
> getting latest valid snapshot path.
> >
> > The above APIs will be used for people who need to recover their gdbm
> file if disaster situations happen.
> > However, this feature is not yet decided to land to CPython.
> > and we have already discussed this issue at
> https://bugs.python.org/issue45452
> > but I would like to hear other devs opinions.
> >
> > I cc the original authors of this feature and Serhiy who thankfully
> already participated in this discussion too :)
> >
> > Warm Regards,
> > Dong-hee
> > ___
> > Python-Dev mailing list -- python-dev@python.org
> > To unsubscribe send an email to python-dev-le...@python.org
> > https://mail.python.org/mailman3/lists/python-dev.python.org/
> > Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/RFICESJKJW3KZSHYSU4R7UEPUUAJL2B3/
> > Code of Conduct: http://python.org/psf/codeofconduct/
>
>
>
> --
> Night gathers, and now my watch begins. It shall not end until my death.
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YFDEPBDWRF35DWFSA4ZE6N2FOCLAUO2U/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Discussion about crash tolerance feature for gdbm module

[Victor Stinner]

Hi Dong-hee,

Can you please show me a short full example opening a database with
automatic snapshop recovery (with your proposed API)?

Do you ask us our opinion on the Python API that you propose? Or if
the whole feature is worth it?

Victor

On Tue, Jan 18, 2022 at 2:41 AM Dong-hee Na  wrote:
>
> Hi folks,
> From gdbm 1.21, gdbm supports the crash tolerance feature.
> see: https://www.gnu.org.ua/software/gdbm/manual/Crash-Tolerance.html
>
> I would like to introduce this feature since python standard library is the 
> only gdbm binding library that is available for end-users.
> And this is also effort for end-users to provide the latest new features.
> If this feature is added following methods will be added.
>   - 'x' flag will be added for extended gdbm format.
>   - gdbm.gdbm_failure_atomic('path1', 'path2') API will be added for snapshot 
> paths.
>   - gdbm.gdbm_latest_snapshot('path1', 'path2') API will be added for getting 
> latest valid snapshot path.
>
> The above APIs will be used for people who need to recover their gdbm file if 
> disaster situations happen.
> However, this feature is not yet decided to land to CPython.
> and we have already discussed this issue at https://bugs.python.org/issue45452
> but I would like to hear other devs opinions.
>
> I cc the original authors of this feature and Serhiy who thankfully already 
> participated in this discussion too :)
>
> Warm Regards,
> Dong-hee
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/RFICESJKJW3KZSHYSU4R7UEPUUAJL2B3/
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/FIOL3OD5GTPNCJKTWO4SCHYSVXDV6BBE/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] PEP 674 – Disallow using macros as l-value (version 2)

[Victor Stinner]

Hi,

I made the following changes to the PEP since the version 1 (November 30):

* Elaborate the HPy section and add a new "Relationship with the HPy
project" section
* Elaborate which projects are impacted and which changes are needed
* Remove the PyPy section

The PEP can be read online:
https://python.github.io/peps/ep-0674/

And you can find the plain text below.

Victor


PEP: 674
Title: Disallow using macros as l-value
Author: Victor Stinner 
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 30-Nov-2021
Python-Version: 3.11

Abstract


Incompatible C API change disallowing using macros as l-value to:

* Allow evolving CPython internals;
* Ease the C API implementation on other Python implementation;
* Help migrating existing C extensions to the HPy API.

On the PyPI top 5000 projects, only 14 projects (0.3%) are affected by 4
macro changes. Moreover, 24 projects just have to regenerate their
Cython code to use ``Py_SET_TYPE()``.

In practice, the majority of affected projects only have to make two
changes:

* Replace ``Py_TYPE(obj) = new_type;``
  with ``Py_SET_TYPE(obj, new_type);``.
* Replace ``Py_SIZE(obj) = new_size;``
  with ``Py_SET_SIZE(obj, new_size);``.


Rationale
=

Using a macro as a l-value
--

In the Python C API, some functions are implemented as macro because
writing a macro is simpler than writing a regular function. If a macro
exposes directly a structure member, it is technically possible to use
this macro to not only get the structure member but also set it.

Example with the Python 3.10 ``Py_TYPE()`` macro::

#define Py_TYPE(ob) (((PyObject *)(ob))->ob_type)

This macro can be used as a **r-value** to **get** an object type::

type = Py_TYPE(object);

It can also be used as **l-value** to **set** an object type::

Py_TYPE(object) = new_type;

It is also possible to set an object reference count and an object size
using ``Py_REFCNT()`` and ``Py_SIZE()`` macros.

Setting directly an object attribute relies on the current exact CPython
implementation. Implementing this feature in other Python
implementations can make their C API implementation less efficient.

CPython nogil fork
--

Sam Gross forked Python 3.9 to remove the GIL: the `nogil branch
`_. This fork has no
``PyObject.ob_refcnt`` member, but a more elaborated implementation for
reference counting, and so the ``Py_REFCNT(obj) = new_refcnt;`` code
fails with a compiler error.

Merging the nogil fork into the upstream CPython main branch requires
first to fix this C API compatibility issue. It is a concrete example of
a Python optimization blocked indirectly by the C API.

This issue was already fixed in Python 3.10: the ``Py_REFCNT()`` macro
has been already modified to disallow using it as a l-value.

These statements are endorsed by Sam Gross (nogil developer).

HPy project
---

The `HPy project `_ is a brand new C API for
Python using only handles and function calls: handles are opaque,
structure members cannot be accessed directly, and pointers cannot be
dereferenced.

Searching and replacing ``Py_SET_SIZE()`` is easier and safer than
searching and replacing some strange macro uses of ``Py_SIZE()``.
``Py_SIZE()`` can be semi-mechanically replaced by ``HPy_Length()``,
whereas seeing ``Py_SET_SIZE()`` would immediately make clear that the
code needs bigger changes in order to be ported to HPy (for example by
using ``HPyTupleBuilder`` or ``HPyListBuilder``).

The fewer internal details exposed via macros, the easier it will be for
HPy to provide direct equivalents. Any macro that references
"non-public" interfaces effectively exposes those interfaces publicly.

These statements are endorsed by Antonio Cuni (HPy developer).

GraalVM Python
--

In GraalVM, when a Python object is accessed by the Python C API, the C API
emulation layer has to wrap the GraalVM objects into wrappers that expose
the internal structure of the CPython structures (PyObject, PyLongObject,
PyTypeObject, etc). This is because when the C code accesses it directly or via
macros, all GraalVM can intercept is a read at the struct offset, which has
to be mapped back to the representation in GraalVM. The smaller the
"effective" number of exposed struct members (by replacing macros with
functions), the simpler GraalVM wrappers can be.

This PEP alone is not enough to get rid of the wrappers in GraalVM, but it
is a step towards this long term goal. GraalVM already supports HPy
which is a better
solution in the long term.

These statements are endorsed by Tim Felgentreff (GraalVM Python developer).

Specification
=

Disallow using macros as l-value


PyObject and PyVarObject macros
^^^

* ``Py_TYPE()``: ``Py_SET_TYPE()`` must be used instead
* ``Py_SIZE()``: ``Py_SET_SIZE()`` must be used instead

"GET" macros