[issue38659] enum classes cause slow startup time

[Ethan Furman]

Ethan Furman  added the comment:

The reason for that quadratic behavior is that for each new member (aka 
attribute), all the previous members must be checked to see if the new member 
is a duplicate.

In practice I wouldn't expect this to be a problem as most enums should be 
fairly small -- are there any real-world examples where there are more than, 
say, a hundred?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38659] enum classes cause slow startup time

[Armin Rigo]

Armin Rigo  added the comment:

Nobody seemed to mention it so I might as well: defining a regular Enum class 
takes an amount of time that is clearly quadratic in the number of attributes.  
That means that the problem is not Python-versus-C or small speed-ups or adding 
secret APIs to do the simple case faster.  The problem is in the algorithm 
which needs to be fixed somewhere.  My timings:

number of attributes time
1500 0.24s
3000 0.94s
6000 3.74s
1200015.57s

--
nosy: +arigo

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38659] enum classes cause slow startup time

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

Seems that the buildbots are going back to green so I will close the revert PR. 
THanks a lot, Ethan for the fix and the investigation!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38659] enum classes cause slow startup time

[Pablo Galindo Salgado]

Change by Pablo Galindo Salgado :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38659] enum classes cause slow startup time

[Ethan Furman]

Ethan Furman  added the comment:

Pablo, did my latest patch resolved the errors?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38659] enum classes cause slow startup time

[Ethan Furman]

Ethan Furman  added the comment:


New changeset 6c681e1a4aa2dbca61be9a26c9257d7d25fa29a7 by Ethan Furman in 
branch 'master':
bpo-38659: [Enum] do not check '_inverted_' during simple test (GH-25566)
https://github.com/python/cpython/commit/6c681e1a4aa2dbca61be9a26c9257d7d25fa29a7


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38659] enum classes cause slow startup time

[Ethan Furman]

Change by Ethan Furman :


--
pull_requests: +24285
pull_request: https://github.com/python/cpython/pull/25566

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38659] enum classes cause slow startup time

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

Thanks a lot Ethan. I will wait then for the investigation.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38659] enum classes cause slow startup time

[Ethan Furman]

Ethan Furman  added the comment:

My apologies, I was having hardware issues.

Checking it out now.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38659] enum classes cause slow startup time

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

Unfortunately, I am being forced to revert commit 
a02cb474f9c097c83cd444a47e9fb5f99b4aaf45 due to have it failing of all refleak 
buildbots for more than two days.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38659] enum classes cause slow startup time

[Pablo Galindo Salgado]

Change by Pablo Galindo Salgado :


--
pull_requests: +24279
pull_request: https://github.com/python/cpython/pull/25560

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38659] enum classes cause slow startup time

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

Commenting out the enum tests reveals that test_socket has additional problems:

㋹ ./python.exe -m test test_socket  -R 3:3
0:00:00 load avg: 2.32 Run tests sequentially
0:00:00 load avg: 2.32 [1/1] test_socket
beginning 6 repetitions
123456
/Users/pgalindo3/github/python/master/Lib/test/test_socket.py:2701: 
RuntimeWarning: received malformed or improperly-truncated ancillary data
  result = sock.recvmsg(bufsize, *args)
/Users/pgalindo3/github/python/master/Lib/test/test_socket.py:2792: 
RuntimeWarning: received malformed or improperly-truncated ancillary data
  result = sock.recvmsg_into([buf], *args)
./Users/pgalindo3/github/python/master/Lib/test/test_socket.py:2701: 
RuntimeWarning: received malformed or improperly-truncated ancillary data
  result = sock.recvmsg(bufsize, *args)
/Users/pgalindo3/github/python/master/Lib/test/test_socket.py:2792: 
RuntimeWarning: received malformed or improperly-truncated ancillary data
  result = sock.recvmsg_into([buf], *args)
./Users/pgalindo3/github/python/master/Lib/test/test_socket.py:2701: 
RuntimeWarning: received malformed or improperly-truncated ancillary data
  result = sock.recvmsg(bufsize, *args)
/Users/pgalindo3/github/python/master/Lib/test/test_socket.py:2792: 
RuntimeWarning: received malformed or improperly-truncated ancillary data
  result = sock.recvmsg_into([buf], *args)
./Users/pgalindo3/github/python/master/Lib/test/test_socket.py:2701: 
RuntimeWarning: received malformed or improperly-truncated ancillary data
  result = sock.recvmsg(bufsize, *args)
/Users/pgalindo3/github/python/master/Lib/test/test_socket.py:2792: 
RuntimeWarning: received malformed or improperly-truncated ancillary data
  result = sock.recvmsg_into([buf], *args)
./Users/pgalindo3/github/python/master/Lib/test/test_socket.py:2701: 
RuntimeWarning: received malformed or improperly-truncated ancillary data
  result = sock.recvmsg(bufsize, *args)
/Users/pgalindo3/github/python/master/Lib/test/test_socket.py:2792: 
RuntimeWarning: received malformed or improperly-truncated ancillary data
  result = sock.recvmsg_into([buf], *args)
./Users/pgalindo3/github/python/master/Lib/test/test_socket.py:2701: 
RuntimeWarning: received malformed or improperly-truncated ancillary data
  result = sock.recvmsg(bufsize, *args)
/Users/pgalindo3/github/python/master/Lib/test/test_socket.py:2792: 
RuntimeWarning: received malformed or improperly-truncated ancillary data
  result = sock.recvmsg_into([buf], *args)
.
test_socket leaked [20, 20, 20] file descriptors, sum=60
test_socket failed in 2 min 25 sec

== Tests result: FAILURE ==

1 test failed:
test_socket

Total duration: 2 min 25 sec
Tests result: FAILURE

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38659] enum classes cause slow startup time

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

Gentle ping

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38659] enum classes cause slow startup time

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

Thanks Ammar for the fix. Unfortunately there is still some failures related to 
this on test_socker:

==
ERROR: test_msgflag_enum (test.test_socket.GeneralModuleTests)
--
Traceback (most recent call last):
  File 
"/home/buildbot/buildarea/3.x.cstratak-RHEL7-ppc64le.refleak/build/Lib/test/test_socket.py",
 line 1969, in test_msgflag_enum
enum._test_simple_enum(CheckedMsgFlag, socket.MsgFlag)
  File 
"/home/buildbot/buildarea/3.x.cstratak-RHEL7-ppc64le.refleak/build/Lib/enum.py",
 line 1664, in _test_simple_enum
raise TypeError('enum mismatch:\n   %s' % '\n   '.join(failed))
TypeError: enum mismatch:
   'MSG_TRUNC' member mismatch:
  extra key '_inverted_' in simple enum member 'MSG_TRUNC'
   'MSG_CTRUNC' member mismatch:
  extra key '_inverted_' in simple enum member 'MSG_CTRUNC'
--

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38659] enum classes cause slow startup time

[STINNER Victor]

STINNER Victor  added the comment:

def setUp(self):
# Reset the module-level test variables to their original integer
# values, otherwise the already created enum values get converted
# instead.

Why not doing that in a tearDown() method instead? What if you run explicitly a 
single test method?

--
nosy: +vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38659] enum classes cause slow startup time

[Ethan Furman]

Ethan Furman  added the comment:

Actually, I think that fixed the refleak issue as well.

Thanks, Ammar!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38659] enum classes cause slow startup time

[Ethan Furman]

Ethan Furman  added the comment:


New changeset 37b173c5231b7b202859af97c17bc3d04b1e4c75 by Ammar Askar in branch 
'master':
bpo-38659: Properly re-intialize module variables in test_enum (GH-25516)
https://github.com/python/cpython/commit/37b173c5231b7b202859af97c17bc3d04b1e4c75


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38659] enum classes cause slow startup time

[Ethan Furman]

Ethan Furman  added the comment:

`format` issue resolved, but not the refleak issue.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38659] enum classes cause slow startup time

[Ammar Askar]

Change by Ammar Askar :


--
pull_requests: +24234
stage: resolved -> patch review
pull_request: https://github.com/python/cpython/pull/25516

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38659] enum classes cause slow startup time

[Ammar Askar]

Ammar Askar  added the comment:

Looks like this is the issue described in the comment here: 
https://github.com/python/cpython/blob/master/Lib/test/test_enum.py#L3691-L3692

On the first run you have the correct

  ('CONVERT_STRING_TEST_NAME_A', 5)

but later it turns into

  ('CONVERT_STRING_TEST_NAME_A', test.test_enum.CONVERT_STRING_TEST_NAME_A)

causing double-conversions of the enum elements. This causes the format(x) test 
to fail. You can re-create the same issue outside of the refleak by adding a 
simple:

def test_convert_repr_and_str_again(self):
self.test_convert_repr_and_str()

method.

--
nosy: +ammar2

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38659] enum classes cause slow startup time

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

Can someone take a look?

As per the buildbot policy 
(https://discuss.python.org/t/policy-to-revert-commits-on-buildbot-failure/404) 
we may need to revert it if is not fixed in 24 because of the risk of masking 
errors.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38659] enum classes cause slow startup time

[Pablo Galindo Salgado]

Change by Pablo Galindo Salgado :


--
priority: normal -> release blocker

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38659] enum classes cause slow startup time

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

Commit a02cb474f9c097c83cd444a47e9fb5f99b4aaf45 fails to build in the refleak 
buildbots:

https://buildbot.python.org/all/#/builders/75/builds/2/steps/5/logs/stdio

Example failure:

❯ ./python  -m test test_enum -R :
0:00:00 load avg: 1.81 Run tests sequentially
0:00:00 load avg: 1.81 [1/1] test_enum
beginning 9 repetitions
123456789
.test test_enum failed -- Traceback (most recent call last):
  File "/home/pablogsal/github/python/master/Lib/test/test_enum.py", line 3700, 
in test_convert_repr_and_str
self.assertEqual(format(test_type.CONVERT_STRING_TEST_NAME_A), '5')
AssertionError: 'CONVERT_STRING_TEST_NAME_A' != '5'
- CONVERT_STRING_TEST_NAME_A
+ 5


test_enum failed

== Tests result: FAILURE ==

1 test failed:
test_enum

Total duration: 586 ms
Tests result: FAILURE

--
nosy: +pablogsal
resolution: fixed -> 
status: closed -> open

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38659] enum classes cause slow startup time

[Ethan Furman]

Change by Ethan Furman :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38659] enum classes cause slow startup time

[Ethan Furman]

Ethan Furman  added the comment:


New changeset a02cb474f9c097c83cd444a47e9fb5f99b4aaf45 by Ethan Furman in 
branch 'master':
bpo-38659: [Enum] add _simple_enum decorator (GH-25497)
https://github.com/python/cpython/commit/a02cb474f9c097c83cd444a47e9fb5f99b4aaf45


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38659] enum classes cause slow startup time

[Ethan Furman]

Change by Ethan Furman :


--
pull_requests: +24219
pull_request: https://github.com/python/cpython/pull/25497

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38659] enum classes cause slow startup time

[Ethan Furman]

Change by Ethan Furman :


--
versions: +Python 3.10 -Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38659] enum classes cause slow startup time

[Ethan Furman]

Ethan Furman  added the comment:


New changeset 503cdc7c124cebbd777008bdf7bd9aa666b25f07 by Ethan Furman in 
branch 'master':
Revert "bpo-38659: [Enum] add _simple_enum decorator (GH-25285)" (GH-25476)
https://github.com/python/cpython/commit/503cdc7c124cebbd777008bdf7bd9aa666b25f07


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38659] enum classes cause slow startup time

[Ethan Furman]

Change by Ethan Furman :


--
pull_requests: +24202
pull_request: https://github.com/python/cpython/pull/25476

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38659] enum classes cause slow startup time

[Ethan Furman]

Ethan Furman  added the comment:


New changeset dbac8f40e81eb0a29dc833e6409a1abf47467da6 by Ethan Furman in 
branch 'master':
bpo-38659: [Enum] add _simple_enum decorator (GH-25285)
https://github.com/python/cpython/commit/dbac8f40e81eb0a29dc833e6409a1abf47467da6


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38659] enum classes cause slow startup time

[Ethan Furman]

Change by Ethan Furman :


--
keywords: +patch
pull_requests: +24021
stage: needs patch -> patch review
pull_request: https://github.com/python/cpython/pull/25285

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38659] enum classes cause slow startup time

[Ethan Furman]

Ethan Furman  added the comment:

I was just looking at this problem, and creating a bare-bones, no safety belts 
version for use in the stdlib (no patch yet) which decreases Enum creation from 
14x slower to only 6x slower.  (Comparing to a class with simple attributes.)

Not sure if that's enough improvement, though.

If it needs to be even faster, a C version of that simplified Enum shouldn't be 
too hard.  Anyone that uses the _simple_enum, though, should have a test that 
uses the full Enum and compares the two to make sure nothing got lost in 
translation.

--
assignee:  -> ethan.furman

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38659] enum classes cause slow startup time

[Karthikeyan Singaravelan]

Change by Karthikeyan Singaravelan :


--
nosy: +ethan.furman

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38659] enum classes cause slow startup time

[Gregory P. Smith]

New submission from Gregory P. Smith :

Creating an enum subclass (ie: defining an enum) is slow.  This dramatically 
impacts startup time of Python programs that import a bunch of potentially 
needed constant definitions at startup before any proper code executes.

How slow?  So slow that a module defining a ~300 enums takes nearly 100ms just 
to import from its pyc file.

Example code: 
https://github.com/googleads/google-ads-python/blob/96fd08bb62435f1930df4871033ba8689333b67f/google/ads/google_ads/v2/services/enums.py

We've known this, we should do something about it.  (Even if it means 
implementing the guts of the magic enum machinery in C.)  ie, it came up in 
https://bugs.python.org/issue28637 as a stdlib startup time regression and is 
likely to come up in similar contexts elsewhere.

--
components: Library (Lib)
messages: 355777
nosy: gregory.p.smith
priority: normal
severity: normal
stage: needs patch
status: open
title: enum classes cause slow startup time
type: performance
versions: Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com