[issue44242] enum.IntFlag regression: missing values cause TypeError

[Ethan Furman]

Ethan Furman  added the comment:

Yup, just had to get back from the weekend.  :-)

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44242] enum.IntFlag regression: missing values cause TypeError

[Jacob Walls]

Jacob Walls  added the comment:

With the followup patch merged, can this be closed now?

--
nosy: +jacobtylerwalls

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44242] enum.IntFlag regression: missing values cause TypeError

[Ethan Furman]

Ethan Furman  added the comment:


New changeset 0a186b1ec1fd094d825f08a4eb39fa83ef57067a by Miss Islington (bot) 
in branch '3.10':
bpo-44242: [Enum] improve error messages (GH-26669)
https://github.com/python/cpython/commit/0a186b1ec1fd094d825f08a4eb39fa83ef57067a


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44242] enum.IntFlag regression: missing values cause TypeError

[miss-islington]

Change by miss-islington :


--
nosy: +miss-islington
nosy_count: 9.0 -> 10.0
pull_requests: +25258
pull_request: https://github.com/python/cpython/pull/26671

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44242] enum.IntFlag regression: missing values cause TypeError

[Ethan Furman]

Ethan Furman  added the comment:


New changeset c956734d7af83ad31f847d31d0d26df087add9a4 by Ethan Furman in 
branch 'main':
bpo-44242: [Enum] improve error messages (GH-26669)
https://github.com/python/cpython/commit/c956734d7af83ad31f847d31d0d26df087add9a4


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44242] enum.IntFlag regression: missing values cause TypeError

[Ethan Furman]

Change by Ethan Furman :


--
pull_requests: +25256
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/26669

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44242] enum.IntFlag regression: missing values cause TypeError

[Ethan Furman]

Ethan Furman  added the comment:

Also changing error reporting to be less susceptible to DOS attacks.

--
resolution: fixed -> 
stage: resolved -> 
status: closed -> open

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44242] enum.IntFlag regression: missing values cause TypeError

[Ethan Furman]

Ethan Furman  added the comment:

Thank you everyone for your help.

--
priority: release blocker -> 
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed
versions: +Python 3.11

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44242] enum.IntFlag regression: missing values cause TypeError

[Ethan Furman]

Ethan Furman  added the comment:


New changeset 749648609de89f14581190ea34b9c0968787a701 by Ethan Furman in 
branch '3.10':
[3.10] bpo-44242: [Enum] remove missing bits test from Flag creation (GH-26586) 
(GH-26635)
https://github.com/python/cpython/commit/749648609de89f14581190ea34b9c0968787a701


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44242] enum.IntFlag regression: missing values cause TypeError

[Ethan Furman]

Change by Ethan Furman :


--
pull_requests: +25221
pull_request: https://github.com/python/cpython/pull/26635

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44242] enum.IntFlag regression: missing values cause TypeError

[Ethan Furman]

Ethan Furman  added the comment:


New changeset eea8148b7dff5ffc7b84433859ac819b1d92a74d by Ethan Furman in 
branch 'main':
bpo-44242: [Enum] remove missing bits test from Flag creation (GH-26586)
https://github.com/python/cpython/commit/eea8148b7dff5ffc7b84433859ac819b1d92a74d


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44242] enum.IntFlag regression: missing values cause TypeError

[Ethan Furman]

Change by Ethan Furman :


--
keywords: +patch
pull_requests: +25170
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/26586

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44242] enum.IntFlag regression: missing values cause TypeError

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

Raising this to a release blocker for 3.10.0 beta 3 as it breaks PyQt6 and 
other projects.

--
priority: normal -> release blocker

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44242] enum.IntFlag regression: missing values cause TypeError

[Thomas Caswell]

Thomas Caswell  added the comment:

This change also affects PyQt6:

Python 3.10.0b2+ (heads/3.10:d0991e2db3, Jun  1 2021, 11:42:08) [GCC 11.1.0] on 
linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import PyQt6.QtCore
>>> PyQt6.QtCore.PYQT_VERSION_STR
'6.1.1.dev2105301600'
>>> PyQt6.QtCore.Qt.Key
Traceback (most recent call last):
  File "", line 1, in 
  File "/home/tcaswell/.pybuild/bleeding/lib/python3.10/enum.py", line 615, in 
__call__
return cls._create_(
  File "/home/tcaswell/.pybuild/bleeding/lib/python3.10/enum.py", line 762, in 
_create_
return metacls.__new__(metacls, class_name, bases, classdict, 
boundary=boundary)
  File "/home/tcaswell/.pybuild/bleeding/lib/python3.10/enum.py", line 544, in 
__new__
raise TypeError(
TypeError: invalid Flag 'DropAction' -- missing values: 8, 16, 32, 64, 128, 
32768
>>> 


> remove the creation time check (which will remove the error for the OP), and 
> then add a decorator to the enum module that does that creation time check

I am very supportive of this plan!

Could this be labeled as a higher priority / release blocker?

--
nosy: +tcaswell

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44242] enum.IntFlag regression: missing values cause TypeError

[Ethan Furman]

Ethan Furman  added the comment:

I'm very much of the practicality beats purity philosophy, so I want to support 
the OP's flag without making them jump through hoops.

On the flip side, I also appreciate that there are folks that want the extra 
security...

So here's my plan:  remove the creation time check (which will remove the error 
for the OP), and then add a decorator to the enum module that does that 
creation time check -- just like the `unique` decorator enforces no aliases.

Now I just need a good name for that decorator:

- complete ?
- exhaustive ?
- all_named ?
- check_for_unnamed ?

--
assignee:  -> ethan.furman

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44242] enum.IntFlag regression: missing values cause TypeError

[John Belmonte]

John Belmonte  added the comment:

> Either [...] we should lose the creation time check (not apply the 
> transform), or we should still have the check (raise an error on invalid 
> bits) which would still leave us in this situation.

That is only one option (which undesirable consequences are already 
identified).  We should consider a few options carefully.

>From the start, if default Enum declarations are going to be subject to new 
>runtime exceptions, it's certain that they'll be hit in some existing code.  
>Anything other than the default being legacy-compatible, or defining the new 
>stuff as Enum2, or deferring all the changes to Python 4, is going to lead to 
>bug reports and disgruntled users.  I don't think that suggests we should 
>remove the runtime exceptions-- they are what supports the new API and 
>implementation.

Perhaps KEEP should be renamed to LEGACY, that should be the default, and there 
should be guidance on how to get the KEEP use cases you've found migrated to 
one of the other modes.  (And changing the default could be considered for 
Python 4.)

>> But the [CONFORM] class members themselves should not have that transform 
>> applied, and raise
>> an error on invalid bits.
>
> But I'm not sure I understand that.

It's an important point that should be fixed in the code.  CONFORM should be 
regarding transforming external values into valid member instances.  The enum 
author's declaration itself should be coherent and not rely on such transforms. 
 Users will be surprised when they define a "FOO = 0xFF" and they read the 
value back as 0x1F or something.  The runtime exception lets the enum author 
know about the issue in the declaration, and normalize it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44242] enum.IntFlag regression: missing values cause TypeError

[Ethan Furman]

Ethan Furman  added the comment:

I'll be offline for a couple hours, but I'll check back.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44242] enum.IntFlag regression: missing values cause TypeError

[Ethan Furman]

Ethan Furman  added the comment:

I see your point about the str/repr.

> But the class members themselves should not have that transform applied, and 
> raise
> an error on invalid bits.

But I'm not sure I understand that.  Either you are agreeing with me that we 
should lose the creation time check (not apply the transform), or we should 
still have the check (raise an error on invalid bits) which would still leave 
us in this situation.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44242] enum.IntFlag regression: missing values cause TypeError

[John Belmonte]

John Belmonte  added the comment:

> Do you agree that simply removing the unnamed member check that takes place 
> at Flag creation is a good way forward?

It's uncomfortable, because then STRICT is not actually strict, and it's going 
to be show raw values in str/repr.

> CONFORM -> unnamed bits are discarded (so the DEFAULT flag would be returned)

This behavior of CONFORM is a little dubious.  I expect it to conform new 
values after the class is constructed.  But the class members themselves should 
not have that transform applied, and raise an error on invalid bits.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44242] enum.IntFlag regression: missing values cause TypeError

[Ethan Furman]

Ethan Furman  added the comment:

That could be, and the user can set the boundary to whatever works best for 
their use-case, so long as the boundary they want to use does not conflict with 
the initial creation checks.

Do you agree that simply removing the unnamed member check that takes place at 
Flag creation is a good way forward?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44242] enum.IntFlag regression: missing values cause TypeError

[John Belmonte]

John Belmonte  added the comment:

> However, if the user is interfacing with other software/hardware, they may 
> not have a choice on which bits make up the mask.

I think it's the same as saying "unknown bits may come on the input".  An 
important use case is for forward compatibility, where new bits may be added 
but you don't want to break existing code.

That is what CONFORM is for.  The cited code should either declare the masks as 
I've shown if STRICT is desired, or use CONFORM if forward compatibility is 
desired.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44242] enum.IntFlag regression: missing values cause TypeError

[Ethan Furman]

Ethan Furman  added the comment:

Yes, that would be best practice.

However, if the user is interfacing with other software/hardware, they may not 
have a choice on which bits make up the mask.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44242] enum.IntFlag regression: missing values cause TypeError

[John Belmonte]

John Belmonte  added the comment:

Rather than make such masks containing unknown bits, this would be best 
practice if you want to use STRICT, correct?

NPTS_ROUND = 0x0
NPTS_CEIL = 0x1
NPTS_TRUNC = 0x2
NPTS_MASK = NPTS_ROUND | NPTS_CEIL | NPTS_TRUNC

Otherwise, if your input may have unknown bits, use CONFORM.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44242] enum.IntFlag regression: missing values cause TypeError

[Ethan Furman]

Ethan Furman  added the comment:

For example, if the default is CONFORM or KEEP, but the user wants an error if 
0x80 comes up, they would have to explicitly check for that value since the 
Flag would happily return it instead of raising.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44242] enum.IntFlag regression: missing values cause TypeError

[Ethan Furman]

Ethan Furman  added the comment:

Those are good points -- the difficulty is knowing which behavior the user 
wants.  And if the desired run-time behavior doesn't match the boundary flag 
the user is stuck.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44242] enum.IntFlag regression: missing values cause TypeError

[Ethan Furman]

Ethan Furman  added the comment:

Actually, thinking about that a little bit more, KEEP was added for exactly 
this situation, as some stdlib flags exhibit the same behavior.

So the real question is what should happen with, for example,

  GeodIntermediateFlag(0x80)

?

The idea behind boundary is what should happen when values are created that 
don't have names in the Enum/Flag?  The options for boundary are:

STRICT  -> an error is raised  (default for Enum)
EJECT   -> the integer 0x80 is returned (not a flag)
CONFORM -> unnamed bits are discarded (so the DEFAULT flag would be returned)
KEEP-> an unnamed flag with value 0x80 is returned

So KEEP is currently doing double-duty -- this reinforces my desire to go with 
option 2 and return KEEP to single-duty status.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44242] enum.IntFlag regression: missing values cause TypeError

[John Belmonte]

John Belmonte  added the comment:

I wonder if CONFORM could tolerate these, since it's ultimately going to 
discard invalid bits.  And then perhaps CONFORM is the default.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44242] enum.IntFlag regression: missing values cause TypeError

[John Belmonte]

John Belmonte  added the comment:

To clarify, it's caused by these mask entries in the enum:

NPTS_MASK = 0xF
DEL_S_MASK = 0xF0
AZIS_MASK = 0xF00

Since the masks are not an aggregation of individual bits defined in the enum, 
it's an error.

> I'm inclined to go with [removing the check], since `boundary` is designed to 
> answer the question of what to do when Flag.A | Flag.B does not exist in Flag.

I think that would cause various problems in the API and implementation.  For 
example, without underlying individual bits, repr() may be nonsensical.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44242] enum.IntFlag regression: missing values cause TypeError

[Ethan Furman]

Ethan Furman  added the comment:

That is an intentional change.  The cause is that the masks include bits that 
are not named in the Flag.

The user-side fix is to add a `boundary=KEEP` option to the flag:

class GeodIntermediateFlag(IntFlag, boundary=KEEP)

The enum library fix could be one of two things:

- automatically use the KEEP boundary when these conditions arise, and issue a 
DeprecationWarning; or

- lose that particular check.

I'm inclined to go with option 2, since `boundary` is designed to answer the 
question of what to do when Flag.A | Flag.B does not exist in Flag.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44242] enum.IntFlag regression: missing values cause TypeError

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

We are already blocked in Python 3.10 beta 2, Ethan, could you give a look at 
this so we can introduce the fix when we release the next beta?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44242] enum.IntFlag regression: missing values cause TypeError

[Ned Deily]

Change by Ned Deily :


--
nosy: +pablogsal

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44242] enum.IntFlag regression: missing values cause TypeError

[Miro Hrončok]

New submission from Miro Hrončok :

With the change introduced in https://bugs.python.org/issue38250 
https://github.com/python/cpython/commit/7aaeb2a3d682ecba125c33511e4b4796021d2f82
 I observe a regression in behavior of enum.IntFlag with missing values.

Consider this code (from pyproj):

from enum import IntFlag

class GeodIntermediateFlag(IntFlag):
DEFAULT = 0x0

NPTS_MASK = 0xF
NPTS_ROUND = 0x0
NPTS_CEIL = 0x1
NPTS_TRUNC = 0x2
 
DEL_S_MASK = 0xF0
DEL_S_RECALC = 0x00
DEL_S_NO_RECALC = 0x10
  
AZIS_MASK = 0xF00
AZIS_DISCARD = 0x000
AZIS_KEEP = 0x100

This is a valid code in Python 3.9, however produces a TypeError in Python 
3.10.0b1:

Traceback (most recent call last):
  File "intflag.py", line 3, in 
class GeodIntermediateFlag(IntFlag):
  File "/usr/lib64/python3.10/enum.py", line 544, in __new__
raise TypeError(
TypeError: invalid Flag 'GeodIntermediateFlag' -- missing values: 4, 8, 32, 
64, 128, 512, 1024, 2048


Since I don't see this behavior mentioned in 
https://docs.python.org/3.10/library/enum.html or 
https://docs.python.org/3.10/whatsnew/3.10.html or 
https://docs.python.org/3.10/whatsnew/changelog.html I believe this is a 
regression.

--
components: Library (Lib)
messages: 394457
nosy: John Belmonte, Manjusaka, ethan.furman, hauntsaninja, hroncok, jbelmonte, 
veky
priority: normal
severity: normal
status: open
title: enum.IntFlag regression: missing values cause TypeError
type: behavior
versions: Python 3.10

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com