[issue44938] Expose PyErr_ChainExceptions in the stable API
Gregory P. Smith added the comment: related: https://www.python.org/dev/peps/pep-0490/ (rejected pep to auto-chain from C API calls) I don't come across it often, but I did just review some code at work today where the chain might have been nice (though not a big deal). -- nosy: +gregory.p.smith ___ Python tracker <https://bugs.python.org/issue44938> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31370] Remove support for threads-less builds
Gregory P. Smith added the comment: https://web.dev/webassembly-threads/ suggests otherwise. -- ___ Python tracker <https://bugs.python.org/issue31370> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37584] Multiple test failures with OSError: [Errno 84] Invalid or incomplete multibyte or wide character on ZFS with utf8only=on
Gregory P. Smith added the comment: Confirmed. Repro: Do an ubuntu 20.04 install and choose "experimental zfs" support during install - https://ubuntu.com/blog/zfs-focus-on-ubuntu-20-04-lts-whats-new). On such a zfs filesystem, the following tests from a ./python -m test.regrtest run fail in 3.10: 11 tests failed: test_cmd_line_script test_httpservers test_imp test_import test_ntpath test_os test_posixpath test_socket test_unicode_file test_unicode_file_functions test_zipimport Move over to a tmpfs and all but test_httpservers now pass. test_httpservers tries to create such a path on /tmp == ERROR: test_undecodable_filename (test.test_httpservers.SimpleHTTPServerTestCase) -- Traceback (most recent call last): File "/home/greg/test/cpython/Lib/test/test_httpservers.py", line 400, in test_undecodable_filename with open(os.path.join(self.tempdir, filename), 'wb') as f: OSError: [Errno 84] Invalid or incomplete multibyte or wide character: '/tmp/tmpnt9ch98x/@test_124227_tmp\udce7w\udcf0.txt' I expect any filesystem mounted to reject non-UTF8 pathnames to cause similar failures. Our test suite needs to detect this environment and skip these tests there. -- nosy: +gregory.p.smith versions: +Python 3.10, Python 3.11 -Python 3.7, Python 3.8 ___ Python tracker <https://bugs.python.org/issue37584> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46056] Cannot use virtual environment on Windows 10 in corporate security settings
Gregory P. Smith added the comment: While the discussion is interesting for some who are trying to work around security policy mechanisms, we can't control what policies people enforce upon themselves that make their machines useless for software development. Marking as not a bug. The normal thing to do is to obtain an exemption to policies from the corporate security org managing your machine(s) because you have a business need for it. If you have specific implementation changes that would make life easier, file specific issues for those to be addressed. (ex: PEP 582 is pretty specific - and there is a package manager https://pdm.fming.dev/ that supports it) -- nosy: +gregory.p.smith resolution: -> not a bug stage: -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue46056> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46077] Include sha256 hashes of release downloads in announcement communications
New submission from Gregory P. Smith : The announcement email list (https://mail.python.org/archives/list/python-announce-l...@python.org/) and other places we make announcements beyond just the release pages like https://www.python.org/downloads/release/python-3101/ should include a list of sha256 hashes of the release binaries in the announcement text. This serves as an alternative confirmation that nothing has changed as such announcements are widely distributed and archived by independent parties and individuals and can thus convey a level of trust that a hash listed on the downloads page cannot (where an attacker would simply modify both). Yes there is a gpg signature on the downloads. I encourage people to use that. But this provides an alternate distributed mechanism to verify that nothing has changed at all since the release announcement. Something a gpg signature cannot fully do (consider this protection against the possibility of new signed binary being put into its place by a compromised key/signer/builder/RM before anyone happens to notice and poke around). A simple table of: filename.tar.gz | sha256 hash filename.msi | sha256 hash filename.dmg | sha256 hash ... At the end of the announcement email/post would suffice. Less of an issue on source packages as those can be verified against the git repo. But it's nice for people to know if binaries change without an announcement and explanation and is easy for us to provide. Bonus points if the release announcement email body itself is signed (if that is even feasible per our release signing GPG key management). [context: see recent python-dev subject: Python release announcement format] -- messages: 408579 nosy: gregory.p.smith priority: deferred blocker severity: normal status: open title: Include sha256 hashes of release downloads in announcement communications versions: Python 3.10, Python 3.11, Python 3.9 ___ Python tracker <https://bugs.python.org/issue46077> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41222] POpen bufsize=0 ignored with universal_newlines=True
Change by Gregory P. Smith : -- nosy: +gregory.p.smith ___ Python tracker <https://bugs.python.org/issue41222> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46167] Parse assert (x == y, "Descriptive text") as statement params instead of a tuple
New submission from Gregory P. Smith : Now that we have a shiny new parser, can we finally get rid of this language wart: assert thing, description # works as intended assert (thing, description) # always True as non-empty tuples are Truthy This most often happens when extending thing or description beyond a single line on assert statements as () are the natural way to do that and as it is with assert being a statement, knowing specifically where to place the ()s to not fall into the pit of snakes of unintentionally nerfing your assertion to be an always true tuple is hard for human authors. This would obsolete the pylint error about tuple assertion and enable more natural assert use. py.test framework users would presumably rejoice as well. This parsing change would need a PEP. I fail to see any obvious downsides though. -- components: Parser messages: 409101 nosy: gregory.p.smith, lys.nikolaou, pablogsal priority: normal severity: normal status: open title: Parse assert (x == y, "Descriptive text") as statement params instead of a tuple versions: Python 3.11 ___ Python tracker <https://bugs.python.org/issue46167> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46167] Parse assert (x == y, "Descriptive text") as statement params instead of a tuple
Gregory P. Smith added the comment: It's not about an advantage, it's about removing the problem of what edit to make when working on assert thing_that_has_a_meaningful_name.methods_have_good_names(value_from_somewhere) == other_thing_that_is_meaningful, "Description of what the issue is if this fails for humans, as if the names weren't enough" and making that fit within whatever line length limit your codebase has. put () in the wrong place and it triggers the long standing Python wart or parsing as a tuple. rather than warn about the syntax wart, we should just do the thing code authors want in the first place. -- ___ Python tracker <https://bugs.python.org/issue46167> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46150] test_pathlib assumes "fakeuser" does not exist as user
Change by Gregory P. Smith : -- assignee: -> gregory.p.smith nosy: +gregory.p.smith ___ Python tracker <https://bugs.python.org/issue46150> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46150] test_pathlib assumes "fakeuser" does not exist as user
Change by Gregory P. Smith : -- pull_requests: +28466 pull_request: https://github.com/python/cpython/pull/30244 ___ Python tracker <https://bugs.python.org/issue46150> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46150] test_pathlib assumes "fakeuser" does not exist as user
Gregory P. Smith added the comment: New changeset b8de8b7039cee47465b2af9950b0b9ed2d3f2903 by Nikita Sobolev in branch 'main': bpo-46150: ensure `fakeuser` does not exist in `PosixPathTest.test_expanduser` (GH-30240) https://github.com/python/cpython/commit/b8de8b7039cee47465b2af9950b0b9ed2d3f2903 -- ___ Python tracker <https://bugs.python.org/issue46150> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10202] ftplib doesn't check close status after sending file
Gregory P. Smith added the comment: [pasting in my investigation that I replied with on a mailing list:] The possible problem does appear real but it likely not frequently observed and is something most people reading the code wouldn't see as it's rather esoteric: https://blog.netherlabs.nl/articles/2009/01/18/the-ultimate-so_linger-page-or-why-is-my-tcp-not-reliable via https://stackoverflow.com/questions/8874021/close-socket-directly-after-send-unsafe describes the scenario. So does this apply in Python? Apparently. Our socket module close() is simply calling close as expected. https://github.com/python/cpython/blob/main/Modules/socketmodule.c#L3132 -> https://github.com/python/cpython/blob/main/Modules/socketmodule.c#L443 Our ftplib code that implicitly calls close on the data connection when exiting the `with self.transfercmd(cmd, rest) as conn: ` context manager https://github.com/python/cpython/blob/main/Lib/ftplib.py#L498` Triggers a close() without a prior shutdown(socket.SHUT_RDWR) + read() -> EOF or timeout dance. In most protocols this isn't a big deal. ftp being so old as to rely solely on the TCP connection itself is the outlier. How often does this happen in practice? I don't know. I haven't heard of it happening, but how many people honestly use ftplib to send data between operating systems where a socket close triggering a TCP RST rather than FIN on the sender might be more likely in the past 20 years? I suspect not many. The code can be improved to prevent it. But I don't believe it'll be feasible to construct a real world not-mock-filled regression test that would fail without it. Potential solution: Do the shutdown+recv EOF dance as the final step inside of the storbinary and storlines `with self.transfercmd as conn` blocks. Technically speaking socket.socket.shutdown() is conditionally compiled but I don't know if any platforms lacking the socket shutdown API exist anymore (my guess is that conditional compilation for shutdown is a leftover from the 1990s). If so, a "if hasattr(conn, 'shutdown'):" conditional before the added logic would suffice. Looking at various ftp client source code (netkit ftp & netbsd ftp - both bsd derivatives) as well as a modern gonuts golang one I do not see them explicitly doing the shutdown dance. (I didn't dive in to figure out if it is hidden within their flclose() or conn.Close() implementations as that'd be surprising) -- nosy: +gregory.p.smith ___ Python tracker <https://bugs.python.org/issue10202> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10202] ftplib doesn't check close status after sending file
Change by Gregory P. Smith : -- versions: +Python 3.10, Python 3.11, Python 3.9 -Python 2.7, Python 3.1, Python 3.2 ___ Python tracker <https://bugs.python.org/issue10202> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10202] ftplib doesn't check close status after sending file
Change by Gregory P. Smith : -- stage: test needed -> needs patch ___ Python tracker <https://bugs.python.org/issue10202> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46120] Add note to `typing.Union` that it is recommended to use `|` instead
Gregory P. Smith added the comment: New changeset 1b30660c3b40da557050948ac8635f2dc50c4ee2 by Nikita Sobolev in branch 'main': bpo-46120: State that `|` is preferred over `Union` (GH-30222) https://github.com/python/cpython/commit/1b30660c3b40da557050948ac8635f2dc50c4ee2 -- nosy: +gregory.p.smith ___ Python tracker <https://bugs.python.org/issue46120> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46150] test_pathlib assumes "fakeuser" does not exist as user
Change by Gregory P. Smith : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue46150> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46339] PEG parser segfault from ast.literal_eval
New submission from Gregory P. Smith :
cpython/b$ ./python
Python 3.11.0a3+ (heads/main-dirty:081a214008, Jan 11 2022, 02:48:22) [GCC
11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ast
>>> ast.literal_eval('''F"""
...
...
... {6
... 0="""''')
Segmentation fault
this was discovered by oss-fuzz.
This may be a recent regression? It didn't happen on my build from a few weeks
ago. worth testing on older versions.
--
assignee: pablogsal
components: Parser
messages: 410270
nosy: gregory.p.smith, lys.nikolaou, pablogsal
priority: normal
severity: normal
stage: needs patch
status: open
title: PEG parser segfault from ast.literal_eval
type: crash
versions: Python 3.11
___
Python tracker
<https://bugs.python.org/issue46339>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46349] inspect.getdoc() should append parent class method docs when encountering a local one line docstring
New submission from Gregory P. Smith :
Today `inspect.getdoc()` is quite simple. If a method has any docstring, it
returns it.
There is one idiom where this is not very useful to our users. A """See base
class.""" docstring on a method.
Rather than having to repeat the canonical base class documentation in all
methods or argue with tooling about not having a docstring in this common
scenario, a single line docstring being used as a hint to just bring in the
parent's docstring would be more helpful to the user.
Today's inspect.getdoc() could would turn from:
https://github.com/python/cpython/blob/main/Lib/inspect.py#L850
into something like:
```
...
if doc is None or (isinstance(doc, str) and '\n' not in doc.strip()):
try:
parent_doc = _finddoc(object)
if doc is None:
doc = parent_doc
elif isinstance(parent_doc, str):
doc = f'{doc}\nParent class MRO doc:\n{parent_doc}'
except (AttributeError, TypeError):
return doc
...
```
Why not just tell people to omit docstrings when they want parent docs?
Because not all coding styles and linter tooling allows for this as they aim to
enforce that documentation _is_ written. Source analysis tooling and IDEs may
not have a whole transitive dependency view of where the base classes even come
from and are thus incapable of reliably analyzing whether a parent MRO method
even exists or has a docstring. This allows for trivial one line docstrings in
that situation while still providing better information to the help() user.
This would add value to when using
help(typical_subclass_of_abstract_thing.method) in notebooks.
--
components: Library (Lib)
messages: 410335
nosy: gregory.p.smith
priority: normal
severity: normal
stage: needs patch
status: open
title: inspect.getdoc() should append parent class method docs when
encountering a local one line docstring
type: enhancement
versions: Python 3.11
___
Python tracker
<https://bugs.python.org/issue46349>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46349] inspect.getdoc() should append parent class method docs when encountering a local one line docstring
Gregory P. Smith added the comment: >From the context of help(typical_subclass_of_abstract_thing.method), the user >isn't told what the base classes are or which one might have documentation. >So it could become an exercise in frustration. and LOL yes the parameter name is silly but that's a separate module wide cleanup that could be done - inspect is full of public APIs that shadow object as a parameter name. it is often quite old code. -- ___ Python tracker <https://bugs.python.org/issue46349> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46347] memory leak in PyEval_EvalCodeEx
Gregory P. Smith added the comment: when was the regression introduced? I wouldn't necessarily rush an urgent release out unless this was just introduced in the 3.10.1 patch release. Could it wait for 3.10.2 already scheduled for four weeks from now? -- nosy: +gregory.p.smith ___ Python tracker <https://bugs.python.org/issue46347> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27273] subprocess.run(cmd, input='text') should pass universal_newlines=True to Popen
Gregory P. Smith added the comment: agreed. -- resolution: -> rejected stage: patch review -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue27273> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40255] Fixing Copy on Writes from reference counting
Gregory P. Smith added the comment:
[data] I finally dug up the old YouTube doc at work with their findings. Mostly
just posting this here for future public reference if anyone wants. Nothing
surprising.
When youtube experimented with a modified 2.7 adding "eternal refcounts" in
2015, they saw a 3-5% CPU performance regression. (not the 10% I had in my mind)
Their version of this simply set a high bit on the refcount as the indicator
and added the obvious conditional into the Py_INCREF/Py_DECREF macros.
Unsurprisingly in line with what others have since found. For their preforked
server and decision of what to mark eternal before forking, it saved them 10%
ram (fewer copy on writes). The -ram vs +cpu +maintenance cost tradeoff wound
up not being worthwhile to them though. Their motivation for trying was
entirely COW memory savings.
=== CPython 2.7 object.h modification they used:
```
+#ifdef GOOGLE_ETERNAL_REFCOUNT_SUPPORT
+
+#define PY_ETERNAL_REFCOUNT (PY_SSIZE_T_MAX / 2)
+
+#define Py_IS_ETERNAL(op) ( \
+ ((PyObject*)(op))->ob_refcnt >= PY_ETERNAL_REFCOUNT)
+
+#define Py_SET_ETERNAL(op)\
+ do {\
+ ((PyObject*)(op))->ob_refcnt = PY_ETERNAL_REFCOUNT; \
+ if (PyObject_IS_GC(op)) { \
+PyObject_GC_UnTrack(op); \
+ } \
+ } while (0)
+
+#define Py_INCREF(op) ( \
+ Py_IS_ETERNAL(op) \
+? PY_ETERNAL_REFCOUNT\
+: (_Py_INC_REFTOTAL _Py_REF_DEBUG_COMMA \
+ ((PyObject*)(op))->ob_refcnt++)\
+ )
+
+#define Py_DECREF(op) \
+do {\
+if (Py_IS_ETERNAL(op)) break; \
+if (_Py_DEC_REFTOTAL _Py_REF_DEBUG_COMMA \
+--((PyObject*)(op))->ob_refcnt != 0)\
+_Py_CHECK_REFCNT(op)\
+else\
+_Py_Dealloc((PyObject *)(op)); \
+} while (0)
+
+#else
```
--
___
Python tracker
<https://bugs.python.org/issue40255>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46406] optimize int division
New submission from Gregory P. Smith : Based on a python-dev thread, we've come up with faster int division code for CPython's bignums. https://mail.python.org/archives/list/python-...@python.org/thread/ZICIMX5VFCX4IOFH5NUPVHCUJCQ4Q7QM/#NEUNFZU3TQU4CPTYZNF3WCN7DOJBBTK5 filing this issue for starters to attach a PR to. details forthcoming. -- messages: 410735 nosy: gregory.p.smith priority: normal severity: normal stage: needs patch status: open title: optimize int division type: performance versions: Python 3.11 ___ Python tracker <https://bugs.python.org/issue46406> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46406] optimize int division
Change by Gregory P. Smith : -- keywords: +patch pull_requests: +28829 stage: needs patch -> patch review pull_request: https://github.com/python/cpython/pull/30626 ___ Python tracker <https://bugs.python.org/issue46406> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46406] optimize int division
Gregory P. Smith added the comment: The PR was directly inspired by Mark Dickinson's code in the email thread directly using __asm__ to get the instruction he wanted. There is usually a way to make the compiler actually do what you intend. This appears to be it. Interestingly, experimenting with small code snippets rather than the entire longobject.c on gotbolt.org to check various compilers output does not always yield as nice of a result. (clang 11+ showed promise there, but this change benefits gcc equally as well in real world CPython microbenchmark timeit tests). https://godbolt.org/z/63eWPczjx was my playground code. ``` $ ./b-clang13/python -m timeit -n 150 -s 'x = 10**1000; r=x//10; assert r == 10**999, r' 'x//17' 150 loops, best of 5: 450 nsec per loop $ ./b-clang13-new-basic-divrem1/python -m timeit -n 150 -s 'x = 10**1000; r=x//10; assert r == 10**999, r' 'x//17' 150 loops, best of 5: 375 nsec per loop $ ./b-gcc9/python -m timeit -n 150 -s 'x = 10**1000; r=x//10; assert r == 10**999, r' 'x//17' 150 loops, best of 5: 448 nsec per loop $ ./b-gcc9-new-basic-divrem1/python -m timeit -n 150 -s 'x = 10**1000; r=x//10; assert r == 10**999, r' 'x//17' 150 loops, best of 5: 370 nsec per loop ``` That's on an AMD zen3 (x86_64). Also tested with other divisors, 17 is not specialized by the compiler. These were not --enable-optimizations builds, though the results remain similar on those for non-specialized values as x//10 turns into when using -fprofile-values on gcc9. Performance tests using other architectures forthcoming. A pyperformance suite run on a benchmark-stable host is worthwhile. I don't actually expect this to show up as significant in most things there; we'll see. The new code is not any more difficult to maintain than the previous code regardless. -- ___ Python tracker <https://bugs.python.org/issue46406> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46406] optimize int division
Change by Gregory P. Smith : -- nosy: +mark.dickinson ___ Python tracker <https://bugs.python.org/issue46406> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46406] optimize int division
Gregory P. Smith added the comment: I tested my PR branch on 32-bit arm (raspbian bullseye) and the microbenchmark timing shows no change (within the noise across repeated runs). Unsurprising as division is entirely different on 32-bit arm. Raspbian uses armv6 for compatibility with the original rpi and rpi0. armv6 does not have an integer division instruction. (how RISCy of it) But that doesn't make a difference in this code as the final 32-bit arm ISA, armv7-a, only has a 32:32 divider. (armv8 aka aarch64 is 64-bit and uses a UDIV as one would expect) anyways, that satisfies me that it isn't making anything worse elsewhere. -- ___ Python tracker <https://bugs.python.org/issue46406> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45452] Support crash tolerance feature for gdbm module
Gregory P. Smith added the comment: The gdbm module's purpose is effectively one of just exposing the underlying C library APIs to Python as you said. Consider this a +1 in favor of exposing the new APIs in the Python gdbm module. I'm not concerned about anyone wanting these in older Python versions. It really requires a combination of modern software in order to use. Running on a recent kernel version, using a non-default fancy filesystem, and linking against a recent gdbm library. So being a new feature in 3.11 makes sense. Anyone satisfying all of the above is highly likely to already also use a recent CPython. If anyone _really_ wants it for older things, they can maintain backport on PyPI. -- nosy: +gregory.p.smith ___ Python tracker <https://bugs.python.org/issue45452> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40120] Undefined C behavior going beyond end of struct via a [1] arrays.
Gregory P. Smith added the comment: off the top of my head that might actually work as I _think_ "empty" things are required to consume an unused byte of size no matter what meaning sizeof shouldn't change? Some testing and standards perusing for C99 is in order to confirm that though. -- ___ Python tracker <https://bugs.python.org/issue40120> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39207] concurrent.futures.ProcessPoolExecutor does not properly reap jobs and spawns too many workers
Gregory P. Smith added the comment: fyi - this appears to have caused https://bugs.python.org/issue46464 -- nosy: +gregory.p.smith ___ Python tracker <https://bugs.python.org/issue39207> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46464] concurrent.futures.ProcessPoolExecutor can deadlock when tcmalloc is used
Gregory P. Smith added the comment: For context, the fundamental problem is that os.fork() is unsafe to use in any process with threads. concurrent/futures/process.py violates this. So long as its design involves a thread, it can never be guaranteed not to deadlock. POSIX forbids execution of most code after fork() has been called. Returning to the CPython interpreter after os.fork() in the child process is never safe unless the parent process had no threads at fork time. The change that triggered the issue moved from doing all of the os.fork() calls for the workers up front *before the thread was spawned* to doing them on demand after the thread was running. The bugfix for this is simply a rollback to revert the bpo-39207 change. It was a performance enhancement, but it causes a new deadlock in code that didn't already have one when thread tuned malloc implementations are used. The only way to safely launch worker processes on demand is to spawn a worker launcher process spawned prior to any thread creation that remains idle, with a sole job of spawn new worker processes for us. That sounds complicated. That'd be a feature. Lets go with the bugfix first. -- assignee: -> gregory.p.smith keywords: +3.9regression nosy: +gregory.p.smith stage: -> needs patch type: -> resource usage ___ Python tracker <https://bugs.python.org/issue46464> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46493] Add an API to indicate if the process might have multiple threads
New submission from Gregory P. Smith :
It'd be handy to have a function to determine if there are multiple threads in
the current process or not - at least on POSIXish systems. This would be
useful anytime a library is trying to use os.fork(), as fork() is not safe in a
multi-threaded process.
Motivation: I want to _use_ this API to consider raising RuntimeWarnings in
libraries that call fork() to highlight the problem to people attempting to use
that code in multithreaded processes.
As POSIXy OSes don't usually have a simple API to get this information, one
implementation such as this could make sense:
```
def is_process_multithreaded() -> bool:
"""Are there multiple threads in this process? OSError if no such OS API is
available."""
threads = 0
ourself = str(os.gettid())
i_feel_seen = False
try:
# Linux, NetBSD, any others?
with os.scandir('/proc/self/task') as tasks:
for task_dir_entry in tasks:
# tid named subdirs should be the only thing that exists.
# We do a simple conformity check just in case.
if task_dir_entry.name.isdigit():
if task_dir_entry.name == ourself:
i_feel_seen = True
threads += 1
if i_feel_seen and threads > 1:
return True # Multiple threads confirmed.
except EnvironmentError:
raise OSError('Unable to count threads on this platform.')
if i_feel_seen and threads == 1:
return False
else:
raise OSError(f'Unclear. Found {threads} in /proc/self/task and did not
find ourself.')
```
macOS has mach darwin kernel APIs that can do this. Not well documented but
they do work -
https://stackoverflow.com/questions/21478229/how-to-count-number-of-alive-threads-in-ios
FreeBSD has APIs that can do this (see FreeBSD usr.bin/procstat/ source).
configure.ac checks and an extension module would be needed to integrate those.
My use case is not relevant to Windows (a platform unburdened by fork) but
Windows APIs to answer the question exist if anyone has a reason to want this
there as well -
https://stackoverflow.com/questions/3749668/how-to-query-the-thread-count-of-a-process-using-the-regular-windows-c-c-apis
--
components: Library (Lib)
messages: 411420
nosy: gregory.p.smith
priority: normal
severity: normal
stage: needs patch
status: open
title: Add an API to indicate if the process might have multiple threads
type: enhancement
versions: Python 3.11
___
Python tracker
<https://bugs.python.org/issue46493>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5404] Cross-compiling CPython - need a buildbot
Change by Gregory P. Smith : -- title: Cross-compiling Python -> Cross-compiling CPython - need a buildbot versions: +Python 3.11 -Python 3.4 ___ Python tracker <https://bugs.python.org/issue5404> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44733] Feature request: maxtasksperchild for ProcessPoolExecutor
Gregory P. Smith added the comment: [bpo-46464](https://bugs.python.org/issue46464) describes a deadlock that the pre-requisite for this feature causes. Spawning new children directly via fork() is impossible once a thread in the parent process has been started and concurrent.futures.process starts a thread. -- nosy: +gregory.p.smith ___ Python tracker <https://bugs.python.org/issue44733> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44733] Feature request: maxtasksperchild for ProcessPoolExecutor
Gregory P. Smith added the comment: If we want this feature to stay in, it needs an implementation that does not rely on a multithreaded processing calling fork(). I'm going to have to revert the existing implementation for bpo-46464. -- stage: resolved -> needs patch status: closed -> open ___ Python tracker <https://bugs.python.org/issue44733> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44733] Feature request: maxtasksperchild for ProcessPoolExecutor
Change by Gregory P. Smith : -- assignee: -> gregory.p.smith resolution: fixed -> ___ Python tracker <https://bugs.python.org/issue44733> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46464] concurrent.futures.ProcessPoolExecutor can deadlock when tcmalloc is used
Gregory P. Smith added the comment: https://bugs.python.org/issue44733 for 3.11 attempts to build upon the dynamic spawning ability and will need reverting unless a non-thread+fork implementation is provided. -- ___ Python tracker <https://bugs.python.org/issue46464> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46464] concurrent.futures.ProcessPoolExecutor can deadlock when tcmalloc is used
Change by Gregory P. Smith : -- keywords: +patch pull_requests: +29029 stage: needs patch -> patch review pull_request: https://github.com/python/cpython/pull/30847 ___ Python tracker <https://bugs.python.org/issue46464> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46503] assertion failure in Parser/string_parser.c
New submission from Gregory P. Smith :
```
>>> v = r"""f'\N '"""
>>> import ast
>>> ast.literal_eval(v)
python: ../gpshead/Parser/string_parser.c:487: fstring_find_literal: Assertion
`s == end || *s == '{' || *s == '}'' failed.
Aborted
```
this comes from oss-fuzz after enabling assert checks in its cpython builds. :)
https://oss-fuzz.com/testcase-detail/4805529363415040 &
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=43730 for those who have
access.
--
components: Parser
messages: 411503
nosy: gregory.p.smith, lys.nikolaou, pablogsal
priority: high
severity: normal
stage: needs patch
status: open
title: assertion failure in Parser/string_parser.c
type: crash
versions: Python 3.11
___
Python tracker
<https://bugs.python.org/issue46503>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46503] assertion failure in Parser/string_parser.c
Change by Gregory P. Smith : -- assignee: -> pablogsal ___ Python tracker <https://bugs.python.org/issue46503> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11322] encoding package's normalize_encoding() function is too slow
Change by Gregory P. Smith : -- nosy: +gregory.p.smith ___ Python tracker <https://bugs.python.org/issue11322> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46508] codec name acceptance became way too lenient in 3.9
New submission from Gregory P. Smith :
in 3.8 this was not a valid codec name: "เ_เ_เ_iDnA"
in 3.9 it gets treated as idna and triggers the punycode decoder when passed to
bytes.decode(codec).
Discovered by oss-fuzz.
_Likely_ a consequence of https://bugs.python.org/issue37751
The consequences of this change are that anyone can stuff heinous strings into
codec names and get a non-LookupError behavior out of them. Anywhere codecs can
be part of user input this has many interesting potential negative consequences.
<=3.8 gave `LookupError("unknown encoding: ...`
--
keywords: 3.9regression
messages: 411535
nosy: gregory.p.smith
priority: normal
severity: normal
stage: needs patch
status: open
title: codec name acceptance became way too lenient in 3.9
type: behavior
versions: Python 3.10, Python 3.11, Python 3.9
___
Python tracker
<https://bugs.python.org/issue46508>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.
Gregory P. Smith added the comment: https://bugs.python.org/issue46508 filed to track fixing the acceptance and use of garbage codec values regression that this caused. -- nosy: +gregory.p.smith ___ Python tracker <https://bugs.python.org/issue37751> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.
Gregory P. Smith added the comment: (note: this might not be the true cause of that issue; though it sounds potentially related - I haven't investigated far enough yet) -- ___ Python tracker <https://bugs.python.org/issue37751> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.
Gregory P. Smith added the comment: note that Bodo's own followup issue about the breaking change for python-iconv was filed as https://bugs.python.org/issue44723 -- ___ Python tracker <https://bugs.python.org/issue37751> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44723] Codec name normalization breaks custom codecs
Change by Gregory P. Smith : -- nosy: +gregory.p.smith ___ Python tracker <https://bugs.python.org/issue44723> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32958] socket module calls with long host names can fail with idna codec error
Gregory P. Smith added the comment: ablack: the basic auth username:password@ part of the string is not part of a hostname. What code are you seeing that is trying to send that to a name resolver rather than stripping the obviously private info up through the @ sign? -- nosy: +gregory.p.smith ___ Python tracker <https://bugs.python.org/issue32958> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46508] codec name acceptance became way too lenient in 3.9
Gregory P. Smith added the comment: while figuring this issue out, it may also make sense to address https://bugs.python.org/issue44723 as well. -- ___ Python tracker <https://bugs.python.org/issue46508> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46464] concurrent.futures.ProcessPoolExecutor can deadlock when tcmalloc is used
Gregory P. Smith added the comment: It sounds like we need to introspect the mp_context= passed to ProcessPoolExecutor (and it's default when None) to raise an error when max_tasks_per_child is incompatible with it. -- ___ Python tracker <https://bugs.python.org/issue46464> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46464] concurrent.futures.ProcessPoolExecutor can deadlock when tcmalloc is used
Gregory P. Smith added the comment: and similarly, the dynamic spawning could be kept when the mp_context is spawn (and possibly forkserver). -- ___ Python tracker <https://bugs.python.org/issue46464> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44733] Feature request: maxtasksperchild for ProcessPoolExecutor
Gregory P. Smith added the comment: from discussion on the other bug it looks like we should have a way to keep this; we just need to not allow it when the mp_context to be used is the 'fork' one. -- ___ Python tracker <https://bugs.python.org/issue44733> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44733] Feature request: maxtasksperchild for ProcessPoolExecutor
Gregory P. Smith added the comment: As this is a new feature, it would also be reasonable to have specifying max_tasks_per_child without explicitly specifying a mp_context default to a safe mp_context. -- ___ Python tracker <https://bugs.python.org/issue44733> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45162] Remove old deprecated unittest features
Gregory P. Smith added the comment: Per https://mail.python.org/archives/list/python-...@python.org/thread/GJTREADEXYAETECE5JDTPYWK4WMTKYGR/ we want to revert this change for 3.11 as it causes adoption pain for little benefit. To determine if we should proceed with this in 3.12 or future versions, run a stable Python with this change applied on the test suites of external projects to find regressions. (many will have been addressed by then thanks to the work by RedHat) and remember that those are only open source projects. Peoples internal codebases are full of use of these names and this will prevent upgrading until they take the time to address those. There are targeted pieces of this change that may make sense to go forward with individually. But it is easier to reason about after a rollback and consider new PRs to move those forward. -- nosy: +gregory.p.smith priority: normal -> deferred blocker resolution: fixed -> stage: resolved -> needs patch status: closed -> open type: enhancement -> behavior ___ Python tracker <https://bugs.python.org/issue45162> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45173] Remove configparser deprecations
Gregory P. Smith added the comment: Per https://mail.python.org/archives/list/python-...@python.org/thread/GJTREADEXYAETECE5JDTPYWK4WMTKYGR/ we want to revert https://github.com/python/cpython/commit/1fc41ae8709e20d741bd86c2345173688a5e84b0 for 3.11 to avoid causing pain. per that thread and https://discuss.python.org/t/experience-with-python-3-11-in-fedora/12911 it sounds like you may be on top of filing upstream issues/PRs to get OSS projects to stop using the deprecated APIs. great! that should help this stick in 3.12. as the commit removed several things at once, I suggest a full rollback, and smaller PRs for individual things if your upstream python project testing reveals things that nobody appears to be depending on. -- nosy: +gregory.p.smith priority: normal -> deferred blocker stage: patch review -> needs patch ___ Python tracker <https://bugs.python.org/issue45173> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45162] Remove old deprecated unittest features
Change by Gregory P. Smith : -- assignee: -> gregory.p.smith ___ Python tracker <https://bugs.python.org/issue45162> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45162] Remove old deprecated unittest features
Change by Gregory P. Smith : -- pull_requests: +29114 stage: needs patch -> patch review pull_request: https://github.com/python/cpython/pull/30935 ___ Python tracker <https://bugs.python.org/issue45162> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45162] Remove old deprecated unittest features
Gregory P. Smith added the comment: New changeset b50322d20337ca468f2070eedb051a16ee1eba94 by Gregory P. Smith in branch 'main': bpo-45162: Revert "Remove many old deprecated unittest features" (GH-30935) https://github.com/python/cpython/commit/b50322d20337ca468f2070eedb051a16ee1eba94 -- ___ Python tracker <https://bugs.python.org/issue45162> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45162] Remove old deprecated unittest features
Change by Gregory P. Smith : -- assignee: gregory.p.smith -> ___ Python tracker <https://bugs.python.org/issue45162> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42982] Update suggested number of iterations for pbkdf2_hmac()
Gregory P. Smith added the comment: New changeset 897ce9018775bcd679fb49aa17258f8f6e818e23 by Illia Volochii in branch 'main': bpo-42982: Improve the text on suggested number of iterations of PBKDF2 (GH-24276) https://github.com/python/cpython/commit/897ce9018775bcd679fb49aa17258f8f6e818e23 -- nosy: +gregory.p.smith ___ Python tracker <https://bugs.python.org/issue42982> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42982] Update suggested number of iterations for pbkdf2_hmac()
Gregory P. Smith added the comment: I reworked the PR and went with less specific text and linking to the NIST 800 132 appendix as guidance on how people should determine what is right for them. there is no one right number. it is application specific. thanks for everyone's valuable input! -- assignee: docs@python -> gregory.p.smith resolution: -> fixed stage: patch review -> commit review status: open -> closed ___ Python tracker <https://bugs.python.org/issue42982> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46166] Get "self" args or non-null co_varnames from frame object with C-API
Change by Gregory P. Smith : -- keywords: +3.11regression nosy: +gregory.p.smith priority: normal -> high ___ Python tracker <https://bugs.python.org/issue46166> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42982] Update suggested number of iterations for pbkdf2_hmac()
Change by Gregory P. Smith : -- pull_requests: +29145 pull_request: https://github.com/python/cpython/pull/30966 ___ Python tracker <https://bugs.python.org/issue42982> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46548] macOS installers cannot be signed on Monterey
Change by Gregory P. Smith : -- nosy: +ned.deily ___ Python tracker <https://bugs.python.org/issue46548> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40746] test_gdb failing on 32-bit armv7l when built with GCC -Og: Cannot access memory at address 0xfffffedc
Gregory P. Smith added the comment: I manually tested this on 3.11 main and it appears to be working. -- resolution: -> works for me stage: -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue40746> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46576] test_peg_generator is extremely slow
New submission from Gregory P. Smith : test_peg_generator is an extremely slow test. This bug tracks any changes to reduce its runtime and test latency. -- components: Tests messages: 412104 nosy: gregory.p.smith priority: normal severity: normal status: open title: test_peg_generator is extremely slow versions: Python 3.11 ___ Python tracker <https://bugs.python.org/issue46576> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46576] test_peg_generator is extremely slow
Change by Gregory P. Smith : -- keywords: +patch pull_requests: +29194 stage: -> patch review pull_request: https://github.com/python/cpython/pull/31015 ___ Python tracker <https://bugs.python.org/issue46576> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46524] test_peg_generator takes 8 minutes on Windows
Gregory P. Smith added the comment: re: slow tests in the first half of the list. the same total amount of time is going to be spent regardless. In our test suite on a modern fast 16 thread system, all but 10 tests are completed in parallel within the first 30 seconds. The remaining ~10 take 10x+ that wall time more minutes. So the most latency you will shave off on a modern system is probably <30 seconds. On a slower system the magnitude of that will remain the same in proportion. CI systems are not workstations. On -j1 or -j2 system I doubt it will make a meaningful difference at all. Picture test execution as a utilization graph: ``` |ttt | | ttt | tt + ``` The total area under that curve is going to remain the same no matter what so long as we execute everything. Reordering the tests can pull the final long tail in a bit by pushing out the top layer. You move more towards an optimal rectangle, but you're still limited by the area. **The less -jN parallelism you have as CPU cores the less difference any reordering change makes.** What actual parallelism do our Github CI systems offer? The fundamental problem is that we do a LOT in our test suite and have no concept of what depends on what and thus _needs_ to be run. So we run it all. For specialized tests like test_peg_generator and test_tools it should be easy to determine from a list of modified files if those tests are relevant. That gets a lot more complicated to accurately express for things like test_multiprocessing and test_concurrent_futures. test_peg_generator and test_tools are also *packages of tests* that themselves should be parallelized individually instead of considered a single serialized unit. At work we even shard test methods within TestCase classes so that big ones can be split across test executor tasks: See the _setup_sharding() function in absltest here: https://github.com/abseil/abseil-py/blob/main/absl/testing/absltest.py#L2368 In absence of implementing an approach like that within test.regrtest to shard at a more granular level thus enabling us to approach the golden rectangle of optimal parallel test latency, we're left with manually splitting long running test module/packages up into smaller units to achieve a similar effect. -- nosy: +gregory.p.smith ___ Python tracker <https://bugs.python.org/issue46524> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46524] test_peg_generator takes 8 minutes on Windows
Gregory P. Smith added the comment: If a decent parallelism CI systems are not available from github (they seem stuck at 2-3 threads per https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners), an alternative approach could be to shard across multiple parallel CI tasks. Shard such that each one gets only one of the slow tests. Unfortunately if each of these were a line item in Github's poor CI UI sharding a single config's tests across 5-10 tasks would be a nightmare to navigate on a PR. I expect everyone would hate that. Providing our own runners with decent parallelism could help: https://docs.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners But we're always going to be bound by our longest tail test if we don't fix our test parallelism to be more granular. -- ___ Python tracker <https://bugs.python.org/issue46524> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46524] test_peg_generator takes 8 minutes on Windows
Change by Gregory P. Smith : -- pull_requests: +29197 pull_request: https://github.com/python/cpython/pull/31015 ___ Python tracker <https://bugs.python.org/issue46524> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45173] Remove configparser deprecations
Gregory P. Smith added the comment: New changeset e8659b47dece5a272111c0af5e340c364a9f807b by Hugo van Kemenade in branch 'main': bpo-45173: Keep configparser deprecations until Python 3.12 (GH-30952) https://github.com/python/cpython/commit/e8659b47dece5a272111c0af5e340c364a9f807b -- ___ Python tracker <https://bugs.python.org/issue45173> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46524] test_peg_generator takes 8 minutes on Windows
Gregory P. Smith added the comment: New changeset 164a017e13ee96bd1ea1ae79f5ac9e25fe83994e by Gregory P. Smith in branch 'main': bpo-46576: bpo-46524: Disable compiler optimization within test_peg_generator. (#31015) https://github.com/python/cpython/commit/164a017e13ee96bd1ea1ae79f5ac9e25fe83994e -- ___ Python tracker <https://bugs.python.org/issue46524> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46576] test_peg_generator is extremely slow
Gregory P. Smith added the comment: New changeset 164a017e13ee96bd1ea1ae79f5ac9e25fe83994e by Gregory P. Smith in branch 'main': bpo-46576: bpo-46524: Disable compiler optimization within test_peg_generator. (#31015) https://github.com/python/cpython/commit/164a017e13ee96bd1ea1ae79f5ac9e25fe83994e -- ___ Python tracker <https://bugs.python.org/issue46576> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46524] test_peg_generator takes 8 minutes on Windows
Change by Gregory P. Smith : -- pull_requests: +29274 pull_request: https://github.com/python/cpython/pull/31089 ___ Python tracker <https://bugs.python.org/issue46524> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46576] test_peg_generator is extremely slow
Change by Gregory P. Smith : -- pull_requests: +29273 pull_request: https://github.com/python/cpython/pull/31089 ___ Python tracker <https://bugs.python.org/issue46576> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46524] test_peg_generator takes 8 minutes on Windows
Gregory P. Smith added the comment: New changeset f5ebec4d3e1199ec38b88920cfde8e460e5120dd by Gregory P. Smith in branch '3.10': [3.10] bpo-46576: bpo-46524: Disable compiler optimization within test_peg_generator. (GH-31015) (GH-31089) https://github.com/python/cpython/commit/f5ebec4d3e1199ec38b88920cfde8e460e5120dd -- ___ Python tracker <https://bugs.python.org/issue46524> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46576] test_peg_generator is extremely slow
Gregory P. Smith added the comment: New changeset f5ebec4d3e1199ec38b88920cfde8e460e5120dd by Gregory P. Smith in branch '3.10': [3.10] bpo-46576: bpo-46524: Disable compiler optimization within test_peg_generator. (GH-31015) (GH-31089) https://github.com/python/cpython/commit/f5ebec4d3e1199ec38b88920cfde8e460e5120dd -- ___ Python tracker <https://bugs.python.org/issue46576> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45629] Tools/freeze needs tests in the test suite.
Change by Gregory P. Smith : -- nosy: +gregory.p.smith nosy_count: 2.0 -> 3.0 pull_requests: +29280 pull_request: https://github.com/python/cpython/pull/31094 ___ Python tracker <https://bugs.python.org/issue45629> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45629] Tools/freeze needs tests in the test suite.
Gregory P. Smith added the comment: New changeset 8726067ace98a27557e9fdf1a8e1c509c37cfcfc by Gregory P. Smith in branch 'main': bpo-45629: Improve test.support.skip_if_buildbot (GH-31094) https://github.com/python/cpython/commit/8726067ace98a27557e9fdf1a8e1c509c37cfcfc -- ___ Python tracker <https://bugs.python.org/issue45629> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46576] test_peg_generator is extremely slow
Gregory P. Smith added the comment: for my purposes those PRs make it reasonable, even though it remains one of our slowest tests. we can reopen/reuse this issue for any further test_peg_generator performance improvements as needed. -- assignee: -> gregory.p.smith resolution: -> fixed stage: patch review -> commit review status: open -> closed versions: +Python 3.10, Python 3.9 ___ Python tracker <https://bugs.python.org/issue46576> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46524] test_peg_generator takes 8 minutes on Windows
Gregory P. Smith added the comment: test_peg_generator is significantly less of the long tail on optimized builds now. CI extremely noisy performance wise (times vary by 2-3x without any differences anways) so I can't easily judge if this made a notable difference in windows CI latency. -- ___ Python tracker <https://bugs.python.org/issue46524> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46524] test_peg_generator takes 8 minutes on Windows
Gregory P. Smith added the comment: Thanks to Kumar for contributing Windows compiler flags side of this (the point of this issue). -- resolution: -> fixed stage: patch review -> commit review status: open -> closed ___ Python tracker <https://bugs.python.org/issue46524> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33178] Add support for BigEndianUnion and LittleEndianUnion in ctypes
Change by Gregory P. Smith : -- assignee: -> gregory.p.smith nosy: +gregory.p.smith versions: +Python 3.11 -Python 2.7, Python 3.6 ___ Python tracker <https://bugs.python.org/issue33178> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46636] Bugs of 2to3 on built-in function or types(classes)
Gregory P. Smith added the comment: lib2to3 is going away per https://bugs.python.org/issue40360 but it is behaving as intended above, it does not understand logic. it was only ever intended to maybe work on python 2-only code. not python 2-and-3 compatible code. -- nosy: +gregory.p.smith resolution: -> not a bug stage: -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue46636> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46639] Ceil division with math.ceildiv
Change by Gregory P. Smith : -- nosy: +mark.dickinson ___ Python tracker <https://bugs.python.org/issue46639> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42353] Proposal: re.prefixmatch method (alias for re.match)
Change by Gregory P. Smith : -- assignee: -> gregory.p.smith versions: +Python 3.11 -Python 3.10 ___ Python tracker <https://bugs.python.org/issue42353> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42353] Proposal: re.prefixmatch method (alias for re.match)
Change by Gregory P. Smith : -- keywords: +patch pull_requests: +29314 stage: needs patch -> patch review pull_request: https://github.com/python/cpython/pull/31137 ___ Python tracker <https://bugs.python.org/issue42353> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42353] Proposal: re.prefixmatch method (alias for re.match)
Gregory P. Smith added the comment: What do other APIs in widely used languages do with regex terminology? We appear to be the only popular language who anchors to the start of a string with an API named "match". libpcre C: uses "match" to mean what we call "search" - https://www.pcre.org/current/doc/html/pcre2_match.html Go: Uses "Match" to mean what we call "search" - https://pkg.go.dev/regexp#Match JavaScript: Uses "match" to mean what we call "search" - https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/match Java: Uses "matches" (I think meaning what we call fullmatch?) - https://docs.oracle.com/javase/7/docs/api/java/util/regex/Pattern.html C++ RE2: explicit "FullMatch" and "PartialMatch" APIs - https://github.com/google/re2 Jave re2j: uses "matches" like Java regex.Pattern - https://github.com/google/re2j Ruby: Uses "match" as we do "search" - https://ruby-doc.org/core-2.4.0/Regexp.html Rust: Uses match as we do "search" - https://docs.rs/regex/latest/regex/ -- ___ Python tracker <https://bugs.python.org/issue42353> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46648] `test.test_urllib2.MiscTests.test_issue16464` flaky due to external connection
Gregory P. Smith added the comment: The merged PRs disable the test without anything tracking fixing or re-re-enabling it. Reopening & retitling this issue. This specific test does not need to use an external server. That has nothing to do with what the purposes of the test is which is all around a content-length issue per https://bugs.python.org/issue16464 The appropriate fix is to have the test manage it's own http server on localhost in another thread or process. We do not need to test it with https. -- nosy: +gregory.p.smith resolution: fixed -> stage: resolved -> needs patch status: closed -> open title: `test.test_urllib2.MiscTests.test_issue16464` started to fail -> `test.test_urllib2.MiscTests.test_issue16464` flaky due to external connection versions: +Python 3.10, Python 3.8, Python 3.9 ___ Python tracker <https://bugs.python.org/issue46648> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24904] Provide a way for potentially long runtime difflib algorithms to be aborted by the caller (and report progress?)
Gregory P. Smith added the comment: The way I'd go about this is to support some form of periodic checkpoint in the algorithm where it checks in with code supplied by the difflib user. Traditionally that'd have been via callbacks, there might be an async style way to express that these days. Those could indicate that they want the operation to be aborted. If it is possible to estimate progress, supplying that as input to the checkpoint API would be useful. This leaves decision logic on when to abort something entirely up to the user rather than being clock based which is often not what the user wants. I'm re-titling the issue as the original patch and proposal of a timeout isn't the direction several core devs have suggested we head. -- nosy: +gregory.p.smith stage: -> needs patch title: Patch: add timeout to difflib SequenceMatcher ratio() and quick_ratio() -> Provide a way for potentially long runtime difflib algorithms to be aborted by the caller (and report progress?) ___ Python tracker <https://bugs.python.org/issue24904> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20342] Endianness not detected correctly due to AC_RUN_IFELSE macros
Gregory P. Smith added the comment: any idea if this is still relevant? -- components: +Cross-Build nosy: +Alex.Willmer, gregory.p.smith ___ Python tracker <https://bugs.python.org/issue20342> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46648] `test.test_urllib2.MiscTests.test_issue16464` flaky due to external connection
Change by Gregory P. Smith : -- assignee: -> sobolevn ___ Python tracker <https://bugs.python.org/issue46648> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20342] Endianness not detected correctly due to AC_RUN_IFELSE macros
Gregory P. Smith added the comment: I'll leave it open, nothing about this should be ppc64 vs ppc64el specific. Our cross compilation story has historically been not great. Building for a target of one endianness from a build host of other endianness... sounds like exactly one of many kinds of thing that we might have a mess with until someone can confirm. (I'm not setup with a big endian toolchain to build for a qemu target right now so I can't easily confirm) -- ___ Python tracker <https://bugs.python.org/issue20342> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46784] Duplicated symbols when linking embedded Python with libexpat
Change by Gregory P. Smith : -- nosy: +gregory.p.smith ___ Python tracker <https://bugs.python.org/issue46784> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46793] expose expat XML billion laughs attack mitigation APIs
New submission from Gregory P. Smith : Quoting from https://github.com/python/cpython/pull/31397#issuecomment-1044796561 """ XML_SetBillionLaughsAttackProtectionActivationThreshold XML_SetBillionLaughsAttackProtectionMaximumAmplification I still hope that someone can make those two^^ accessible (with additional glue code) to the user on pyexpat level in CPython. """ - Sebastian Pipping @hartwork -- messages: 413513 nosy: gregory.p.smith priority: normal severity: normal stage: needs patch status: open title: expose expat XML billion laughs attack mitigation APIs type: enhancement versions: Python 3.11 ___ Python tracker <https://bugs.python.org/issue46793> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46784] Duplicated symbols when linking embedded Python with libexpat
Change by Gregory P. Smith : -- pull_requests: +29557 pull_request: https://github.com/python/cpython/pull/31420 ___ Python tracker <https://bugs.python.org/issue46784> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46200] Discourage logging f-strings due to security considerations
Gregory P. Smith added the comment: A new system of logging APIs has been on several of our (core dev and otherwise) minds ever since f-strings were introduced. For this specific issue, agreed that documentation is key. The old logging APIs cannot change. And practically preventing passing non-literal constant values in as the first parameter within their implementation while there are theoretical ways to _maybe_ do that, I expect none would be maintainable or performant. For https://www.python.org/dev/peps/pep-0675/#logging-format-string-injection, my gut feeling is you want logging functions annotated as is a duality using overload. ``` @overload def info(str) -> None: ... @overload def info(msg: LiteralString, *args: object) -> None: ``` As that prevents the untrusted template in msg so you could use a lone f-str. This larger conversation is better turned into a python-dev or discuss.python.org thread than done in our poor bug tracker as anything more than the doc update and musing around the annotations turns into an evolved logging system design discussion. (a nice thread internally at work just restarted about this and linked here... people want f-logging!) -- nosy: +gregory.p.smith ___ Python tracker <https://bugs.python.org/issue46200> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46784] Duplicated symbols when linking embedded Python with libexpat
Gregory P. Smith added the comment: New changeset e05e3d20d309603010f2c1194e612f894ad8a985 by Gregory P. Smith in branch '3.10': [3.10] bpo-46784: Add newly exported expat symbols to the namespace. (GH-31397) (GH-31420) https://github.com/python/cpython/commit/e05e3d20d309603010f2c1194e612f894ad8a985 -- ___ Python tracker <https://bugs.python.org/issue46784> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46784] Duplicated symbols when linking embedded Python with libexpat
Gregory P. Smith added the comment: PRs for 3.7 and 3.8 remain up to release blockers. -- components: +Build resolution: -> fixed stage: patch review -> commit review status: open -> closed ___ Python tracker <https://bugs.python.org/issue46784> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46784] Duplicated symbols when linking embedded Python with libexpat
Gregory P. Smith added the comment: err "release managers" same thing right? ;) -- ___ Python tracker <https://bugs.python.org/issue46784> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45373] ./configure --enable-optimizations should enable LTO
Gregory P. Smith added the comment: FWIW I agree that we should try adding LTO to --enable-optimizations now. -- nosy: +gregory.p.smith ___ Python tracker <https://bugs.python.org/issue45373> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
