[issue23591] enum: Add Flags and IntFlags

[Roundup Robot]

Roundup Robot added the comment:

New changeset f404a59d834e by Ethan Furman in branch '3.6':
closes issue23591: add NEWS entry
https://hg.python.org/cpython/rev/f404a59d834e

--
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23591] enum: Add Flags and IntFlags

[Josh Rosenberg]

Josh Rosenberg added the comment:

The "What’s New In Python 3.6" page should really get an update to mention 
Flag, IntFlag and auto as enhancements to the enum module. Right now, the ssl 
module update docs mentions (but does not properly link for some reason) 
IntFlag, but otherwise, there is no mention of the new feature in either the 
"What's New" docs or even in the theoretically comprehensive changelog; if you 
don't think to look at the enum module itself, you'd never know there were new 
features to use.

--
nosy: +josh.r

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23591] enum: Add Flags and IntFlags

[Roundup Robot]

Roundup Robot added the comment:

New changeset b56290a80ff7 by Ethan Furman in branch '3.6':
issue23591: fix flag decomposition and repr
https://hg.python.org/cpython/rev/b56290a80ff7

New changeset 7372c042e9a1 by Ethan Furman in branch 'default':
issue23591: fix flag decomposition and repr
https://hg.python.org/cpython/rev/7372c042e9a1

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23591] enum: Add Flags and IntFlags

[Ethan Furman]

Ethan Furman added the comment:

I'll respond to the bulk of your comment later.  For now let me assure you your 
feedback has been invaluable.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23591] enum: Add Flags and IntFlags

[Vedran Čačić]

Vedran Čačić added the comment:

About drama: I don't know how to approach the problem. I tell you about various 
concerns of mine, you seem to agree, and still you do exactly what I was 
warning you about. It could be that you just consider me a boring nag, but then 
saying so would be more useful than this situation.

---

Yes, you solved some problems I did have, and it's great, but still many of 
them remain. This thing is inherently complicated. You are trying to construct 
a discrete Boolean algebra, but without any guarantee of atoms. Imagine if the 
language was different:

class MySets(DiscreteSets):
a = {0, 1}
b = {2}
c = {1, 2}

You'd naturally view {0}, {1} and {2} as atoms, of different "sort" than {0,1} 
and such. That's what's missing here: a clear distinction between atoms and 
"molecules". It could be that the intended usage is for atoms to _always_ be 
constructed with auto(), and for molecules to be constructed from previous 
atoms, but then the documentation ought to say so.

So far, the only thing you say is "then it is on them to do it correctly", but 
you never really tell what "correctly" means. I understand your desire not to 
hard forbid anything, but the docs should tell how your class is _intended_ to 
be used, _AND_ the class should make every reasonable effort to preserve what 
it can of its behavior even in the face of not-quite-intended usage. Look at 
Raymond's collections.Counter's treatment of non-integer values for example.

BTW I'm not aware of any stdlib object whose repr raises ValueError. It surely 
makes debugging a lot harder.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23591] enum: Add Flags and IntFlags

[Ethan Furman]

Ethan Furman added the comment:

Vedran:

1) Thank you for the bug report, it's appreciated.

2) I would appreciate it even more if you just kept it to a bug report
   without the, for lack of a better word, drama.

---

>>>> import enum
>>>> class B(enum.Flag):
>b = 3
>c = 4
>d = 6
>>>> B.b | B.c
>Traceback (most recent call last): ...
>ValueError: 7 is not a valid B

This part is correct.  If the programmer wants to use actual values instead of 
auto() then it is on them to do it correctly.  I'd be happy to add a decorator 
to help ensure all bits have names, but I'm not sure what to call it.

---

>>>> t = B.b | B.c
>>>> t
>

>>>> ~B.d
>

These parts are not, and will be fixed.

---

>>>> class C(enum.Enum):
>   a = b = enum.auto() 
>>>> C.a
>
>>>> C.b
>

Bug, will be fixed.

>>>> def f():
>return enum.auto()
>>>> class E(enum.Enum):
>a = b = f()
>c = d = 2
>>>> E.a is E.b
>False

Same as above bug, will be fixed

>>>> E.c is E.d
>True
>>>> E.b is E.c
>True

Same as above bug, will be fixed.  (Had "a" and "b" been given different 
"auto()"s that last True would be correct as "b" would then have had a value of 
2 from auto(), and "c" and "d" also were given values of 2, so all three would 
have been the same.)

--
status: closed -> open

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23591] enum: Add Flags and IntFlags

[Vedran Čačić]

Vedran Čačić added the comment:

Now that I actually had the chance to play with the implementation, I see most 
of my worst fears were justified. :-( Look:

>>> import enum
>>> class B(enum.Flag):
b = 3
c = 4
d = 6
>>> B.b | B.c
Traceback (most recent call last): ...
ValueError: 7 is not a valid B
>>> t = B.b | B.c
>>> t

>>> B.b | B.c
>>> B.b | B.c

>>> ~B.d


Do you really find this behavior acceptable?? I see at least three bugs here.

At least you did say in the documentation

> Individual flags should have values that are powers of two (1, 2, 4, 8, ...)

but it seems to me it's not prominent enough.

---

Also, auto, despite parentheses, works exactly as it shouldn't.

>>> class C(enum.Enum):
a = b = enum.auto() 
>>> C.a

>>> C.b


>>> def f():
return enum.auto()
>>> class E(enum.Enum):
a = b = f()
c = d = 2
>>> E.a is E.b
False
>>> E.c is E.d
True
>>> E.b is E.c
True

In my opinion, this is simply horrible. Even _you_ said 
(http://bugs.python.org/issue23591#msg275093) this would be unacceptable. I'm 
really, really sad.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23591] enum: Add Flags and IntFlags

[Ethan Furman]

Ethan Furman added the comment:

Many thanks to everyone for their input and help.

--
resolution:  -> fixed
stage: commit review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23591] enum: Add Flags and IntFlags

[Ethan Furman]

Ethan Furman added the comment:

>> Which means it has methods such as __getitem__, __setitem__, etc.,
>> which means those methods can implement whatever is needed to give
>> the namespace the desired semantics (within Python syntax).

> Ah, _that_'s what you had in mind. (All this time, I thought _auto_
> was just a shorter name for _generate_next_value_.:) I'm ok with that.
> But aren't we then back to square one? (Using magic of the same kind
> as the "declarative style" that Raymond pronounced not Pythonic enough.)

Not at all.  The concern with that proposal (issue26988: AutoEnum via 
completely omitting values of any kind) was the transformation of a NameError 
into a value.  The current method is taking a unique object and transforming 
that into a value, which is entirely analagous to what Enum already does.

Besides being convenient, it's also necessary to present a cohesive, non-leaky 
abstraction to the user where Flag is concerned: using ints as the flag values 
is entirely an implementation detail, but without auto the user would be forced 
to manage that implementation detail when creating the Flag, which negates much 
of Flag's value.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23591] enum: Add Flags and IntFlags

[Vedran Čačić]

Vedran Čačić added the comment:

> Which means it has methods such as __getitem__, __setitem__, etc., which 
> means those methods can implement whatever is needed to give the namespace 
> the desired semantics (within Python syntax).

Ah, _that_'s what you had in mind. (All this time, I thought _auto_ was just a 
shorter name for _generate_next_value_.:) I'm ok with that. But aren't we then 
back to square one? (Using magic of the same kind as the "declarative style" 
that Raymond pronounced not Pythonic enough.)

Even Raymond says
> As long as _auto_ has been defined somewhere (i.e. from enum import _auto_), 
> it is normal Python

I'm sure he didn't think of the loophole you're currently exploiting. :-D

But anyway, I'm happy. Parentheses are there, so the intuition of calling a 
function to execute code is respected, and if you get out free with this hack, 
it opens a door a bit wider for complete declarative solution in the future.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23591] enum: Add Flags and IntFlags

[Ethan Furman]

Changes by Ethan Furman :


--
stage: patch review -> commit review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23591] enum: Add Flags and IntFlags

[Roundup Robot]

Roundup Robot added the comment:

New changeset aad7443e62be by Ethan Furman in branch 'default':
issue23591: add auto() for auto-generating Enum member values
https://hg.python.org/cpython/rev/aad7443e62be

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23591] enum: Add Flags and IntFlags

[Ethan Furman]

Ethan Furman added the comment:

Just so we're clear:  I'm going to go with "auto()" and (mostly) use Python's 
standard routines (with only a little bit of metaclass magic).

Vedran opined:
> Since you like examples, what do you say about

I love examples!  Examples are like pictures, which are worth a thousand words. 
 ;)

>class MyEnum(Enum):
>red = some_function()
>blue = red
>
> Now, is MyEnum.blue the same as MyEnum.red (watch: not "equal", but
> "same")?

Yes.

> Well, it depends on what some_function returns, right?

Nope.

> If it returns _auto_, they are not the same, but in all the other
> cases, blue is just an alias for red. So object identity depends on
> some value that could be external to the class. To me that's
> obviously unacceptable.

That would be unacceptable.  However, it would also be impossible*, because 
"_auto_" would be injected into the class namespace during "__prepare__()" and 
be unavailable for an external function to return.

* as impossible as anything is in Python -- so really, really difficult

> But even "the namespace class body is executed in" _should_ be a Python
> namespace, with all its rules.

One of those rules is:  the namespace is dict-like.

Which means it has methods such as __getitem__, __setitem__, etc., which means 
those methods can implement whatever is needed to give the namespace the 
desired semantics (within Python syntax).

Which, to some extent, is what will be used to transform an instance of "auto" 
into an appropriate integer.  In other words:

>>> class Color(Enum):
... red = auto()
... print(red)
...
1
>>> Color.red


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23591] enum: Add Flags and IntFlags

[Vedran Čačić]

Vedran Čačić added the comment:

Since you like examples, what do you say about

class MyEnum(Enum):
red = some_function()
blue = red

Now, is MyEnum.blue the same as MyEnum.red (watch: not "equal", but "same")? 
Well, it depends on what some_function returns, right? If it returns _auto_, 
they are not the same, but in all the other cases, blue is just an alias for 
red. So object identity depends on some value that could be external to the 
class. To me that's obviously unacceptable.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23591] enum: Add Flags and IntFlags

[Vedran Čačić]

Vedran Čačić added the comment:

Yes, I didn't explain well. I'm ok with postprocessing. After all, since we got 
PEP 520, it's quite obvious that "the namespace class body is executed in" is 
not the same as "the final __dict__ of a class".

But even "the namespace class body is executed in" _should_ be a Python 
namespace, with all its rules. If objects are the same, then after 
postprocessing they should also be same. _Especially_ if we currently do have 
semantics for reassignment: aliasing. "Name is not a part of object" is a 
fundamental tenet of Python. And `_auto_ is _auto_`.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23591] enum: Add Flags and IntFlags

[Martin Panter]

Changes by Martin Panter :


--
nosy:  -martin.panter

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23591] enum: Add Flags and IntFlags

[Ethan Furman]

Ethan Furman added the comment:

Vedran commented:
> This is something fundamental: it is breaking the promise that class body
> is a suite of commands, where Python statements (such as assignment) have
> their usual semantics.

I find it curious that you're okay with

>>> class Color(Enum):
... red = 1
...
>>> Color.red == 1
False

But you find this completely incomprehensible

>>> class Color(Enum):
...red = _auto_
...blue = _auto_
...
>>> Color.red == Color.blue
False

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23591] enum: Add Flags and IntFlags

[Raymond Hettinger]

Raymond Hettinger added the comment:

> Any problems with:
> 
> class Color(Enum):  # or Color(Flag)
>  red = _auto_
>  green = _auto_
>  blue = _auto_

As long as _auto_ has been defined somewhere (i.e. from enum import _auto_), it 
is normal Python and doesn't fight with the rest of language or its toolchains.

The idea does seem to be at odds with the enum module itself.  Heretofore, the 
rule for the module is that if the same object is assigned to multiple variable 
names, those names become synonyms.

>>> from enum import Enum
>>> class Color(Enum):
red = 1
magenta = 1
orange = 2
ochre = 2

>>> Color.orange is Color.ochre
True

Also, I still question the need, you already have multiple ways to do it:

Color = Enum('Color', ['red', 'green', 'blue'])

Color = Enum('Color', '''
red
green
blue
''')

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23591] enum: Add Flags and IntFlags

[Vedran Čačić]

Vedran Čačić added the comment:

_member_() is fine with me, though I prefer _auto_(). _auto_member_() is 
probably too much. :-)

The argument "we already do magic, so let's do also this bit of non-connected 
magic" seems very weak to me. But in fact those other things are not _magic_ in 
the same sense as this one (except repatching __new__, I agree that's weird, 
but Guido started that game when he declared bool unsubclassable:). They are 
just added behaviors.

This is something fundamental: it is breaking the promise that class body is a 
suite of commands, where Python statements (such as assignment) have their 
usual semantics. If I ever see, in any suite,

a = x
b = x

and after that `a is not b`, I feel cheated. Even more strongly: I _don't know 
how to think about the code_ anymore. Those look like assignment statements, 
but they are not. What are they? Is this Python?

And what if you really do want to make aliases? Are you saying that

a = x
b = a

should be fundamentally different than the above? Oh, please let's not go 
there. :-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23591] enum: Add Flags and IntFlags

[Ethan Furman]

Ethan Furman added the comment:

> For me, it is an issue. But you probably already know that.
>(http://bugs.python.org/issue26988#msg273125) (That's "the other thread".)

Ah, thanks.  I had forgotten about that one.

For what it's worth, I largely agree with you there.

> Try to explain: how exactly is that different than wanting "file.close"
> to close the file, or "quit" to quit the REPL?

Enum is already full of magic:
- class is iterable
- class is indexable
- member attributes are instances of class
- members are all created when class is created
- etc.

Flag, especially, needs a way to specify "give me a working value" even when 
the user doesn't care what that value is (as your, um, interesting examples 
have shown ;).

Whatever we choose as the placeholder (since it has been decided that no 
placeholder is too magical) will still have magic involved, so I don't want 
parenthesis there disguising that.

I'd also be happy with _member_ instead of _auto_ -- maybe that makes even more 
sense.

> And I surely hope we're not going in that direction [dropping parens from 
> function calls].

No, we're not.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23591] enum: Add Flags and IntFlags

[Vedran Čačić]

Vedran Čačić added the comment:

For me, it is an issue. But you probably already know that.
(http://bugs.python.org/issue26988#msg273125) (That's "the other thread".)

Try to explain: how exactly is that different than wanting "file.close" to 
close the file, or "quit" to quit the REPL? And I surely hope we're not going 
in that direction.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23591] enum: Add Flags and IntFlags

[Ethan Furman]

Ethan Furman added the comment:

Any problems with:

class Color(Enum):  # or Color(Flag)
  red = _auto_
  green = _auto_
  blue = _auto_

In other words:
- is the missing parenthesis an issue?
- are linters/checkers/IDEs going to have (or report) problems with that?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23591] enum: Add Flags and IntFlags

[Ethan Furman]

Ethan Furman added the comment:

I like it!  (Although I have no idea which other thread you are talking about.)

And yes, it needs the leading and trailing underscore so as not to clash with 
member names.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23591] enum: Add Flags and IntFlags

[Vedran Čačić]

Vedran Čačić added the comment:

I think we had that discussion in the other thread, and concluded that auto() 
(or _auto_() if you insist) is quite fine. I think it's important to emphasize 
it's automatically generated, not that it's really "next" in any particular 
order.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23591] enum: Add Flags and IntFlags

[STINNER Victor]

Changes by STINNER Victor :


--
nosy:  -haypo

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23591] enum: Add Flags and IntFlags

[Ethan Furman]

Ethan Furman added the comment:

I would like to add a function that can be used when creating a Flag (and Enum, 
but that's less important) that will generate the next value.

This is important because in Flag the values are an important internal detail, 
but are largely irrelevant to the user (if they weren't then an IntFlag is 
probably more appropriate).

Actually, this function already exists and is used to supply the values when 
using the Functional API: _generate_next_value_ .  But that's a bit clunky for 
use during class definition:

class Color(Flag):
red = _generate_next_value_()
blue = _generate_next_value_()
green = _generate_next_value_()

What would be a better name for class use?

- _next_value_
- _value_
- _flag_
- _next_flag_

Any others?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23591] enum: Add Flags and IntFlags

[Ethan Furman]

Ethan Furman added the comment:

> I am quite aware about what's the intended use, but you can't just
> assume people will know about it.

Fair enough.

> In my view, you must do one of two things:
> 
> 1) (at least in documentation, and preferably in the code by raising 
> Exceptions at class definition time) forbid the use of Flags whose values are 
> not either a) powers of two, or b) bitwise or of some already declared values

The closest I would get to forbidding something would be to forbid non-integers 
as values -- and I'm not going to do that.  In general, Python doesn't take 
such steps (one notable exception being sum refusing to work with strings 
because it was such a common trap).

> -or-
> 
> 2) Specify and implement a robust algorithm for finding the "cover of
> bits" for a given numeric value. If the intended algorithm is really
> "pick the largest member value smaller than given value, subtract and
> repeat until 0 remains", then it should be said so in the
> documentation, and preferably some reasons given for the usage of that
> exact algorithm. ("it was easiest to implement" does not rank very high
> on my list.)

How about "it works for me"?  Seriously, if every bit is named then the exact 
repr used is irrelevant.  If you don't like the standard method then write your 
own __repr__ method, because that's what we're talking about -- not the actual 
value (which isn't going to be affected by the algorithm for finding the "cover 
of bits", but the display of the names).

> In other words, MyFlags(7) should either be illegal, or I should be
> able to interpret what it will be by reading the documentation. Leaving
> it unspecified is not acceptable IMO.

Thankfully, docs can still change during beta.

> (In case it isn't clear: I'm for option 1. I _don't_ intend to write
> MyFlags ever in Python. But if I happen to write it unintentionally
> (e.g. if I forget to declare 2), I would like Python to tell me I'm
> doing something wrong.)

We could certainly add a decorator that double-checks that, just like we have 
the `unique` decorator to ensure no duplicates in an Enum.  Any idea what to 
call it?  complete?  named?  allbits?

> (But if you really want option 2 for some reason, that's ok too. I'm
> just saying I would like it specified, with a rationale if it's not
> too much of a problem.)

(1) isn't going to happen, except possibly as a decorator.  If you would like 
to contribute code and/or docs for (2) and/or a decorator for (1) I'd be happy 
to review.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23591] enum: Add Flags and IntFlags

[Vedran Čačić]

Vedran Čačić added the comment:

> As a side note, I suspect there would be much less confusion with Flag if we 
> had an AutoEnum.

No, there would be _different_ confusion. :-P

Anyway, let's get back to the discussion. I am quite aware about what's the 
intended use, but you can't just assume people will know about it. In my view, 
you must do one of two things:

1) (at least in documentation, and preferably in the code by raising Exceptions 
at class definition time) forbid the use of Flags whose values are not either 
a) powers of two, or b) bitwise or of some already declared values

-or-

2) Specify and implement a robust algorithm for finding the "cover of bits" for 
a given numeric value. If the intended algorithm is really "pick the largest 
member value smaller than given value, subtract and repeat until 0 remains", 
then it should be said so in the documentation, and preferably some reasons 
given for the usage of that exact algorithm. ("it was easiest to implement" 
does not rank very high on my list.)

In other words, MyFlags(7) should either be illegal, or I should be able to 
interpret what it will be by reading the documentation. Leaving it unspecified 
is not acceptable IMO.

(In case it isn't clear: I'm for option 1. I _don't_ intend to write MyFlags 
ever in Python. But if I happen to write it unintentionally (e.g. if I forget 
to declare 2), I would like Python to tell me I'm doing something wrong.)

(But if you really want option 2 for some reason, that's ok too. I'm just 
saying I would like it specified, with a rationale if it's not too much of a 
problem.)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23591] enum: Add Flags and IntFlags

[Roundup Robot]

Roundup Robot added the comment:

New changeset 8e9d3a5d47d5 by Ethan Furman in branch 'default':
issue23591: more docs; slight change to repr
https://hg.python.org/cpython/rev/8e9d3a5d47d5

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23591] enum: Add Flags and IntFlags

[Ethan Furman]

Ethan Furman added the comment:

As a side note, I suspect there would be much less confusion with Flag if we 
had an AutoEnum.

C'est la vie.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23591] enum: Add Flags and IntFlags

[Ethan Furman]

Ethan Furman added the comment:

> All my questions pertain to Flags.

Ah, okay.

> You said what to me seem like two contradictory things:
> 
>> Not having 2 named has different consequences for Flag vs IntFlag
>> (although *neither is an error*): Flag: no combination of flags will
>> ever have the 2 bit set [...]

and

>> if MyFlags is a Flag then MyFlags(impossible_combination) *will raise an 
>> exception.*

First, let me state the intended use of Flags:  primary flags (single-bit 
flags) will have values of powers of 2:

--> class Perm(Flag):
... R = 4
... W = 2
... X = 1
...
--> Perm.R | Perm.W | Perm. X

-->

If nicer names are desired for certain combinations (aka secondary flags, or 
multi-bit), then name them:

--> class Perm(Flag):
... ...
... RWX = 7
...
--> Perm.R | Perm.W | Perm.X


If, for whatever strange reason, you don't give a certain power of 2 a name, 
Flag doesn't care:

--> class Perm(Flag):
... R = 8
... W = 4
... X = 1
...
--> Perm.R | Perm.W | Perm. X

-->

But trying to use that missing value will be an error:

--> Perm(6)
Traceback (most recent call last):
  ...
ValueError: 6 is not a valid MyFlags

This is what I was referring to as "not naming a bit is not an error, but using 
an impossible value is".  What I missed in your example was that, although you 
hadn't named 2, you still had multi-bit values that included the 2 bit.  In 
other words, in my example there will never be a combination that has the 2 bit 
set, while in yours (because of your weird values) it is possible.


> Are you saying that after I write
> 
>  class MyFlags(Flags):
>  b001 = 1
>  b011 = 3
>  b100 = 4
>  b110 = 6
> 
> this is _not_ an error, but if after that I call
> 
>  print(MyFlags(7))
> 
> it _is_ an error? 

No, that's not what I'm saying, and hopefully my explanation above clears that 
up.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23591] enum: Add Flags and IntFlags

[Vedran Čačić]

Vedran Čačić added the comment:

Ok, I believe you that you have the interface for IntFlags right (I always did, 
and I apologize if I didn't make it clear from the start). All my questions 
pertain to Flags.

You said what to me seem like two contradictory things:

> Not having 2 named has different consequences for Flag vs IntFlag (although 
> *neither is an error*): Flag: no combination of flags will ever have the 2 
> bit set

> if MyFlags is a Flag then MyFlags(impossible_combination) *will raise an 
> exception.*

Are you saying that after I write

class MyFlags(Flags):
b001 = 1
b011 = 3
b100 = 4
b110 = 6

this is _not_ an error, but if after that I call

print(MyFlags(7))

it _is_ an error? It doesn't seem so bad now I think about it, but I'd like the 
error to be reported before ("3 has bit of value 2 set, but that bit is not a 
member" or something like that).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23591] enum: Add Flags and IntFlags

[Ethan Furman]

Ethan Furman added the comment:

> 1) IntFlags is not simply the meet of int and Flags (like IntEnum is the
> meet of int and Enum,
> https://docs.python.org/3.5/library/enum.html#others)?
> It seems a very different class.

No, I think it is.  The differences between Enum and IntEnum are entirely 
because IntEnum members are also ints, so they can do anything an int can do 
(but they lose their IntEnum-ness when non-Enum operations are performed on 
them).  Similarly, the differences between Flag and IntFlag are entirely 
because IntFlag members are also ints, so can do whatever ints can do (and, 
similarly, will lose their IntFlag-ness when non-Flag operations are performed 
on them).

> 2) (more important) If I give names to 1, 3, 4, and 6, you're in fact
> saying that MyFlags(7) == MyFlags(5) (since 2 is never set)?

(I think you messed up your example, as 4 | 1 == 5 and 6 | 1 == 7 -- 2 is never 
a factor.)

Absolutely not.  5 != 7, so the two will never be equal no matter which of Flag 
| IntFlag you are using.

Moreover, if MyFlags is a Flag then MyFlags(impossible_combination) will raise 
an exception.  If MyFlags is an IntFlag then MyFlags(impossible_combination) 
will have the value of  and the repr will show all the 
non-specified bits as base-2 numbers and the specified bits as names.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23591] enum: Add Flags and IntFlags

[Vedran Čačić]

Vedran Čačić added the comment:

Hmm... so if I read you right:

1) IntFlags is not simply the meet of int and Flags (like IntEnum is the meet 
of int and Enum, https://docs.python.org/3.5/library/enum.html#others)? It 
seems a very different class.

2) (more important) If I give names to 1, 3, 4, and 6, you're in fact saying 
that MyFlags(7) == MyFlags(5) (since 2 is never set)? Seems like succumbing to 
temptation to guess. ;-/ But maybe it's not bad in this case, since there is 
really no ambiguity.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23591] enum: Add Flags and IntFlags

[Roundup Robot]

Roundup Robot added the comment:

New changeset f33fc2117bb2 by Ethan Furman in branch 'default':
issue23591: bool(empty_flags) == False; more docs & tests
https://hg.python.org/cpython/rev/f33fc2117bb2

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23591] enum: Add Flags and IntFlags

[Roundup Robot]

Roundup Robot added the comment:

New changeset 31586a2f01b6 by Ethan Furman in branch 'default':
issue23591: optimize _high_bit()
https://hg.python.org/cpython/rev/31586a2f01b6

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23591] enum: Add Flags and IntFlags

[Ethan Furman]

Ethan Furman added the comment:

> Yes, you're correct here, but what about output? Do all relevant bits have
> to be named separately?

No.

> And will the output always talk about separate
> bits?

No.

> We have talked about this.

Indeed.  Here's my previous reply. ;)

>> The algorithm is simple: start with the biggest Flag and mask off
>> matching bits until all bits are are matched.

When I said "biggest flag" I meant the flag with the highest value (so a flag 
of 5 would get matched before a flag of four).

>> If any unmatched bits remain an error is raised.

This part is specifically for Flag; IntFlag will just show the values of any 
unmatched bits.

> If I give meaningful name to 1, 3, 4 and
> 6, but not to 2, is it supported?

Depends.  Not having 2 named has different consequences for Flag vs IntFlag 
(although neither is an error):

- Flag: no combination of flags will ever have the 2 bit set
- IntFlag: '2' will show in the name portion if the 2 bit is set

> And is the repr of 7 an implementation detail, or is it specified?

I should specify it in the docs, thanks.

> One more question: in the docs, you say 
>>
>> otherwise, all members evaluate as :data:`True`.
>
>I would like you to reconsider the case of `.0` (.NONE in my
> terminology). Many other things in the interface (e.g. `in` operator)
> are modelled as if flags instance were just a container (set, in fact)
> of bits. In that case, empty set _should_ be false. Otherwise, this will
> be a weird exception to the usual container semantics.

Very good point.  Unless I can think of a good reason not to, I'll make that 
change in the next couple days.

Thanks.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23591] enum: Add Flags and IntFlags

[Vedran Čačić]

Vedran Čačić added the comment:

> Nope.  You are welcome to give more meaningful names to different 
> combinations of powers of two.

Yes, you're correct here, but what about output? Do all relevant bits have to 
be named separately? And will the output always talk about separate bits? We 
have talked about this. If I give meaningful name to 1, 3, 4 and 6, but not to 
2, is it supported? And is the repr of 7 an implementation detail, or is it 
specified?

One more question: in the docs, you say 

> otherwise, all members evaluate as :data:`True`.

I would like you to reconsider the case of `.0` (.NONE in my terminology). Many 
other things in the interface (e.g. `in` operator) are modelled as if flags 
instance were just a container (set, in fact) of bits. In that case, empty set 
_should_ be false. Otherwise, this will be a weird exception to the usual 
container semantics.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23591] enum: Add Flags and IntFlags

[Roundup Robot]

Roundup Robot added the comment:

New changeset adbc7eec97f1 by Ethan Furman in branch 'default':
issue23591: add docs; code cleanup; more tests
https://hg.python.org/cpython/rev/adbc7eec97f1

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23591] enum: Add Flags and IntFlags

[Vedran Čačić]

Vedran Čačić added the comment:

Sure, errors should never pass silently. This function shouldn't be called with 
nonpositive arguments. And there is no highbit of 0 (and there are infinitely 
many of negative numbers;).

ValueError is probably best - though IndexError can also be argued for, since 
this is quite analogous to pop from empty list. :-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23591] enum: Add Flags and IntFlags

[STINNER Victor]

STINNER Victor added the comment:

As noticed by veky on the review, _high_bit() is slow and can be optimized 
using int.bit_length(). Attached bit_length.patch implements this.

_high_bit(0) returns -1. Maybe an exception must be raised if the argument is < 
1? (also fail for negative number)

--
Added file: http://bugs.python.org/file44319/bit_length.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23591] enum: Add Flags and IntFlags

[Raymond Hettinger]

Raymond Hettinger added the comment:

Still needs docs.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23591] enum: Add Flags and IntFlags

[Roundup Robot]

Roundup Robot added the comment:

New changeset 39661e2ff030 by Ethan Furman in branch 'default':
issue23591: add Flags, IntFlags, and tests
https://hg.python.org/cpython/rev/39661e2ff030

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23591] enum: Add Flags and IntFlags

[Ethan Furman]

Ethan Furman added the comment:

Since we're using re as the sample, here's where re.I is defined:

Lib/re.py:
-
I = IGNORECASE = sre_compile.SRE_FLAG_IGNORECASE # ignore case

As already mentioned, re.I results in 2, and a re.compile object is not usable 
as a re flag.

> Also, I suppose that means you've given up on the autocreation

We decided auto-created values were too magical for the stdlib (see 
issue26988).  They will exist in my aenum package, though.

> (since the values _are_ semantical here),

The values have meaning because the underlying library gave them meaning; so 
assuming a type of Flags are used, they will be IntFlags.

> and I suppose you'll require all the declared values to be powers of 2.

Nope.  You are welcome to give more meaningful names to different combinations 
of powers of two.

> With those conditions, I think this is a good enhancement of Python.

Hopefully you still think so. ;)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23591] enum: Add Flags and IntFlags

[Vedran Čačić]

Vedran Čačić added the comment:

The weirdest thing is that it already works pretty well in output of re.compile.

>>> re.compile('', re.I | re.M)
re.compile('', re.IGNORECASE|re.MULTILINE)
>>> _.flags  # re.UNICODE == 32 added automatically
42

So the only thing we should enhance is the output of .flags, and it seems that 
all the necessary code is already in the source code of __repr__ of SRE_Pattern 
objects.

Also, I suppose that means you've given up on the autocreation (since the 
values _are_ semantical here), and I suppose you'll require all the declared 
values to be powers of 2. With those conditions, I think this is a good 
enhancement of Python.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23591] enum: Add Flags and IntFlags

[Ethan Furman]

Ethan Furman added the comment:

Raymond, thanks for your support.

I am happy to have a flags variant.  The advice to wait and let Enum mature 
instead of adding the Flag variant at the beginning was sound, and we have a 
better product to show for it now.

The four base Enum-type classes will be:
- Enum
- IntEnum
- Flags
- IntFlags

I see no reason to add any more than these four to the stdlib -- further 
specializations can live in recipes, third-party modules, or private libraries.

While Enum and Flags are the heart of the enum module, IntEnum and IntFlags are 
present for two reasons:
- commonly needed when interfacing with other systems
- use in the stdlib itself

Pollution (otherwise known as enums escaping into other code where they don't 
make sense) is controlled by:
- IntEnums losing their enum status as soon as they are combined with
  any other number; and
- IntFlags losing their enum status as soon as a non-bitwise operator is
  used on them.

I'll commit as soon as the test suite is finished (and any failures are not 
attributable to these changes), and work on the docs later.

Serhiy, I'm not going to include your changes to re, os, etc., as they are not 
part of adding Flags/IntFlags.  Please open new issues for them.  I am not 
opposed to those changes, as

>>> re.I


is more useful than

>>> re.I
2

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23591] enum: Add Flags and IntFlags

[Raymond Hettinger]

Raymond Hettinger added the comment:

[Ethan]
> My experience is that a module maintainer, or somebody claiming to speak
> for the module maintainer, can close any issue in their area at any 
> time regardless of the number of core devs in favor of a change.

Ethan, it is still up to you and the other module maintainers to decide whether 
this goes in or whether to defer it for a release cycle.  If you have any 
concerns about having too many enum variants and think this is at odds with 
your goals for the module, say so here and we can resolve it at the sprints.  
On the other hand, if you're happy with it, we should get it applied as soon as 
possible.

[Serhiy]
> If general IntFlags will not added, I'm going to open separate issues
> for adding specialized class for every particular case (re, os, etc).

When enum went in, I thought Guido had said and everyone agreed to refrain 
sweeping through the standard lib and applying enums broadly to long standing 
and stable APIs.  Perhaps something has changed since them but there was a 
clear intention to show restraint, let enums mature, respect stable APIs, and 
to wait for demonstrated need (i.e. actual rather than imagined user 
difficulties).

--
nosy: +rhettinger

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23591] enum: Add Flags and IntFlags

[Ethan Furman]

Ethan Furman added the comment:

Patch includes tests, but not docs.

>From a previous comment:
---
> As it stands, Weird(7) would be , and if A was not
> defined an error would be raised.

Actually, it would be  since BC has a higher value than A.

--
stage:  -> patch review
Added file: http://bugs.python.org/file44278/issue23591.stoneleaf.02.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23591] enum: Add Flags and IntFlags

[STINNER Victor]

STINNER Victor added the comment:

> The repr is better -- which patch did you test?

Sorry, I wasn't clear. I didn't test any patch :-) I expect (not
expected) better repr when changes will be applied, it's just a
remark...

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23591] enum: Add Flags and IntFlags

[Ethan Furman]

Ethan Furman added the comment:

The repr is better -- which patch did you test?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23591] enum: Add Flags and IntFlags

[STINNER Victor]

STINNER Victor added the comment:

I expected better repr for such code:

>>> socket.SOCK_STREAM | socket.SOCK_CLOEXEC
524289
>>> os.O_RDONLY | os.O_APPEND
1024

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23591] enum: Add Flags and IntFlags

[STINNER Victor]

STINNER Victor added the comment:

I really like the idea of IntFlags.

--
nosy: +haypo
title: Add Flags and IntFlags -> enum: Add Flags and IntFlags

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com