[issue41100] Support macOS 11 and Apple Silicon Macs
Maxime Belanger added the comment: I had some time today, so I took a crack at merging this into Python 3.8. I've filed GH-25274 with my proposal. To summarize, I'm suggesting back porting all changes from bpo-41100 (as well as the subsequent bpo-42688) but without the weak-linking changes to `posixmodule` and co, the installer changes and the `universal2` variant. This means we don't support `arm64` officially, but someone can build for it from source if they'd like. This should unblock Homebrew (which currently manually patches based on some obsolete PRs) as well as folks who build from source and have a particular need (as we would for example at Dropbox). -- ___ Python tracker <https://bugs.python.org/issue41100> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43377] _PyErr_Display should be available in the CPython-specific API
New submission from Maxime Belanger : We have found `_PyErr_Display` to be quite helpful in embedding situations, in particular as a way to capture errors to a custom buffer rather than to `stderr`. Otherwise, embedders often have to replicate logic in `PyErr_Print`, etc. Since the header restructuring in Python 3.8+, that function is a bit harder to call. It's exported, but is considered "internal" and thus requires defining `Py_BUILD_CORE`. I was wondering: why not expose it under "Include/cpython"? It seems like a generic-enough helper, similar to `_PyErr_WriteUnraisableMsg`. -- components: C API messages: 387965 nosy: Maxime Belanger priority: normal severity: normal status: open title: _PyErr_Display should be available in the CPython-specific API type: enhancement versions: Python 3.10 ___ Python tracker <https://bugs.python.org/issue43377> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41100] Build failure on macOS 11 (beta)
Change by Maxime Belanger : -- nosy: +Maxime Belanger ___ Python tracker <https://bugs.python.org/issue41100> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34297] Windows py.exe launcher fail to handle quote correctly
Change by Maxime Belanger : -- nosy: +Maxime Belanger ___ Python tracker <https://bugs.python.org/issue34297> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37966] is_normalized is much slower at "no" than the standard's algorithm
Maxime Belanger added the comment: Thanks for that! -- ___ Python tracker <https://bugs.python.org/issue37966> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33376] [pysqlite] Duplicate rows can be returned after rolling back a transaction
Change by Maxime Belanger : -- versions: +Python 3.9 -Python 2.7, Python 3.7 ___ Python tracker <https://bugs.python.org/issue33376> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32280] Expose `_PyRuntime` through a section name
Maxime Belanger added the comment: Thanks for taking a look! To answer your question, the need for the named section comes not from the API being being "internal", but because we need to access it at runtime from a tool running in a separate process. We have augmented Crashpad (Google's crash reporter) tool, which inspects the memory of a crashing process to create/upload a report, to access each native thread's stack and reconstruct its Python stack frames. This allows us to quickly make sense of native crashes involving Python code. To do this, the tool needs to know where the Python runtime stores state within thread-local storage. This is actually the only reason we need to access `_PyRuntime`: it allows us to retrieve `autoTSSKey` for the process, which we can use to look up the `PyThreadState` for each underlying/native thread. At the time, `_PyRuntime` seemed like a natural structure to expose and a named section a simple-enough way of doing so. We're certainly open to alternatives if you think there's a better way. Given your plans for 3.9: I'm assuming `autoTSSKey` will remain the same per-process? -- ___ Python tracker <https://bugs.python.org/issue32280> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35139] Statically linking pyexpat in Modules/Setup fails to compile on macOS
Maxime Belanger added the comment: The line in question being: ``` #pyexpat expat/xmlparse.c expat/xmlrole.c expat/xmltok.c pyexpat.c -I$(srcdir)/Modules/expat -DHAVE_EXPAT_CONFIG_H -DUSE_PYEXPAT_CAPI ``` -- ___ Python tracker <https://bugs.python.org/issue35139> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35139] Statically linking pyexpat in Modules/Setup fails to compile on macOS
New submission from Maxime Belanger : Uncommenting the following line in `Modules/Setup` leads to a compiler error on macOS (tested w/ Xcode 10): ``` ./Modules/expat/xmlparse.c:92:3: error: You do not have support for any sources of high quality entropy enabled. For end user security, that is probably not what you want. Your options include: * Linux + glibc >=2.25 (getrandom): HAVE_GETRANDOM, * Linux + glibc <2.25 (syscall SYS_getrandom): HAVE_SYSCALL_GETRANDOM, * BSD / macOS >=10.7 (arc4random_buf): HAVE_ARC4RANDOM_BUF, * BSD / macOS <10.7 (arc4random): HAVE_ARC4RANDOM, * libbsd (arc4random_buf): HAVE_ARC4RANDOM_BUF + HAVE_LIBBSD, * libbsd (arc4random): HAVE_ARC4RANDOM + HAVE_LIBBSD, * Linux / BSD / macOS (/dev/urandom): XML_DEV_URANDOM * Windows (RtlGenRandom): _WIN32. If insist on not using any of these, bypass this error by defining XML_POOR_ENTROPY; you have been warned. If you have reasons to patch this detection code away or need changes to the build system, please open a bug. Thank you! ``` I believe this is due to `setup.py` being out of sync with `Modules/Setup`, which defines `XML_POOR_ENTROPY`. I'll attach a patch that resolves the issue for me. -- components: Build, Extension Modules, macOS messages: 329089 nosy: Maxime Belanger, ned.deily, ronaldoussoren priority: normal severity: normal status: open title: Statically linking pyexpat in Modules/Setup fails to compile on macOS type: compile error versions: Python 3.6, Python 3.7, Python 3.8 ___ Python tracker <https://bugs.python.org/issue35139> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33376] [pysqlite] Duplicate rows can be returned after rolling back a transaction
Change by Maxime Belanger : -- nosy: +Maxime Belanger ___ Python tracker <https://bugs.python.org/issue33376> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35080] The tests for the `dis` module can be too rigid when changing opcodes
Maxime Belanger added the comment: Ah, quite right; apologies for the not-so-useful PR! -- resolution: -> wont fix stage: patch review -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue35080> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35080] The tests for the `dis` module can be too rigid when changing opcodes
Change by Maxime Belanger : -- components: +Library (Lib), Tests ___ Python tracker <https://bugs.python.org/issue35080> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35080] The tests for the `dis` module can be too rigid when changing opcodes
New submission from Maxime Belanger : For various reasons, one may want to change the `Lib/opcode.py` module. In doing so, this has caused us to frequently rewrite `test_dis.py`. It would be great if those tests used `dis.opmap` rather than hard-coded/magic numbers, as this would allow easier changes. -- messages: 328615 nosy: Maxime Belanger priority: normal severity: normal status: open title: The tests for the `dis` module can be too rigid when changing opcodes type: enhancement versions: Python 3.8 ___ Python tracker <https://bugs.python.org/issue35080> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32285] In `unicodedata`, it should be possible to check a unistr's normal form without necessarily copying it
Change by Maxime Belanger : -- versions: +Python 3.8 -Python 3.7 ___ Python tracker <https://bugs.python.org/issue32285> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35025] Compiling `timemodule.c` can fail on macOS due to availability warnings
New submission from Maxime Belanger : We build Python on macOS with `-Werror=unguarded-availability` and `-mmacosx-version-min=` to ensure `libpython` is binary-compatible with earlier versions of macOS. This can create problems when building some modules, including `timemodule.c`, which was recently altered to fix bpo-28081. The initial fix is inappropriate, because attempting to reference `CLOCK_REALTIME` et al when `HAVE_CLOCK_GETTIME` is unset (in our case, due to being too "new"), results in a compiler error: ``` ./Modules/timemodule.c:1368:29: error: '_CLOCK_REALTIME' is only available on macOS 10.12 or newer [-Werror,-Wunguarded-availability] PyModule_AddIntMacro(m, CLOCK_REALTIME); ^~ /usr/include/time.h:154:24: note: expanded from macro 'CLOCK_REALTIME' #define CLOCK_REALTIME _CLOCK_REALTIME ^~~ ./Include/modsupport.h:78:67: note: expanded from macro 'PyModule_AddIntMacro' #define PyModule_AddIntMacro(m, c) PyModule_AddIntConstant(m, #c, c) ``` A more correct patch (I'm attaching ours) is to only add the macros to the module if at least one of the three functions is defined. This should continue to work for the author of the original issue as well as fix our problem. -- components: Extension Modules, macOS messages: 328030 nosy: Maxime Belanger, ned.deily, ronaldoussoren priority: normal severity: normal status: open title: Compiling `timemodule.c` can fail on macOS due to availability warnings type: compile error versions: Python 3.6, Python 3.7, Python 3.8 ___ Python tracker <https://bugs.python.org/issue35025> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35022] MagicMock should support `__fspath__`
New submission from Maxime Belanger : We have plenty of tests calling into `os.path.*` functions, and as Python 3.6+ gets more stringent about checking for `os.PathLike` compliance, it would greatly simplify our lives for `MagicMock` to support `__fspath__`. -- components: Library (Lib) messages: 328014 nosy: Maxime Belanger priority: normal severity: normal status: open title: MagicMock should support `__fspath__` type: enhancement versions: Python 3.7, Python 3.8 ___ Python tracker <https://bugs.python.org/issue35022> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32282] When using a Windows XP compatible toolset, `socketmodule.c` fails to build
Maxime Belanger <m...@dropbox.com> added the comment: Works for me. We no longer rely on the `_xp` toolset so no objections. -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue32282> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32280] Expose `_PyRuntime` through a section name
Maxime Belanger <m...@dropbox.com> added the comment: Interesting, would this imply potentially multiple GILs? The major thing we need out of the structure is the (`Py_tss_t`) `autoTSSKey` in order to associate a native thread with its `PyThreadState`. -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue32280> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32285] In `unicodedata`, it should be possible to check a unistr's normal form without necessarily copying it
New submission from Maxime Belanger <m...@dropbox.com>: In our deployment of Python 2.7, we've patched `unicodedata` to introduce a new function: `is_normalized` can check whether a unistr is in a given normal form. This currently has to be done by creating a normalized copy, then checking whether it is equal to the source string. This function uses the internal helper (also called `is_normalized`) that can "quick check" normalization, but falls back on creating a normalized copy and comparing (when necessary). We're contributing this change in case this can helpful to others. Feedback is welcome! -- components: Unicode messages: 308085 nosy: Maxime Belanger, ezio.melotti, vstinner priority: normal severity: normal status: open title: In `unicodedata`, it should be possible to check a unistr's normal form without necessarily copying it versions: Python 2.7, Python 3.7 ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue32285> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32282] When using a Windows XP compatible toolset, `socketmodule.c` fails to build
New submission from Maxime Belanger <m...@dropbox.com>: If Python is built using the Windows XP "variant" of the toolset (e.g. `v140_xp`), build errors can occur in `socketmodule.c`. This is due to the include of `VersionHelpers.h` being gated on `_MSC_VER`, which "lies" if `v140_xp` is in use. We'll be contributing our patch to this issue here. -- components: Windows messages: 308081 nosy: Maxime Belanger, paul.moore, steve.dower, tim.golden, zach.ware priority: normal severity: normal status: open title: When using a Windows XP compatible toolset, `socketmodule.c` fails to build versions: Python 3.5, Python 3.6, Python 3.7 ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue32282> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32280] Expose `_PyRuntime` through a section name
New submission from Maxime Belanger <m...@dropbox.com>: We've recently been toying with more sophisticated crash reporting machinery for our Desktop Python application (which we deploy on macOS, Windows and Linux). To assist in debugging, we are storing `_PyRuntime` in a predictable location our crash reporter can later look up without needing the symbol table (this can grow complicated across multiple platforms). Taking a page from `crashpad`'s book (https://chromium.googlesource.com/crashpad/crashpad/+/master/client/crashpad_info.cc), we've patched `pylifecycle.c` to store the `_PyRuntime` struct in a section of the same name. Upon a crash, this section is then used by the tool to annotate each report with enough information to reconstruct the Python stack frames in each thread (as applicable). We're contributing our patch here in the hopes this can be helpful to others. -- components: Interpreter Core, Windows, macOS messages: 308079 nosy: Maxime Belanger, ned.deily, paul.moore, ronaldoussoren, steve.dower, tim.golden, zach.ware priority: normal severity: normal status: open title: Expose `_PyRuntime` through a section name type: enhancement versions: Python 3.7 ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue32280> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31903] `_scproxy` calls SystemConfiguration functions in a way that can cause deadlocks
New submission from Maxime Belanger <m...@dropbox.com>: Through the use of various Python packages (such as `pyobjc`), it is possible for a deadlock to occur due to how `_scproxy.c` calls `SCDynamicStoreCopyProxies`. In more recent versions of macOS (10.7+), this function can block on acquiring a lock deep inside `NSUserPreferences`. As `pyobjc` allows Python-wrapped `NSString`s to be stored in `CFPreferences`, it is thus possible for one thread to hold the `CFPreferences` lock and block on the GIL while another thread holds the GIL and blocks on the `CFPreferences` lock. We've observed this on a significant number of macOS devices before fixing ourselves by wrapping the calls to `SCDynamicStoreCopyProxies` with `Py_BEGIN/END_ALLOW_THREADS`. -- components: macOS messages: 305247 nosy: Maxime Belanger, ned.deily, ronaldoussoren priority: normal severity: normal status: open title: `_scproxy` calls SystemConfiguration functions in a way that can cause deadlocks type: crash versions: Python 2.7, Python 3.5, Python 3.6, Python 3.7 ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue31903> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31359] `configure` script incorrectly detects symbols as available on Mac w/ Xcode 8+
Maxime Belanger added the comment: That'd be even better :). I'll also note that this is slightly worse in Python 3 vs Python 2 as there are way more of these to deal with. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue31359> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31359] `configure` script incorrectly detects symbols as available on Mac w/ Xcode 8+
Changes by Maxime Belanger <m...@dropbox.com>: -- components: +Build, macOS nosy: +ned.deily, ronaldoussoren type: -> compile error ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue31359> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31359] `configure` script incorrectly detects symbols as available on Mac w/ Xcode 8+
New submission from Maxime Belanger: At Dropbox, we use a custom fork of Python to deploy our Desktop Client onto various platforms, including macOS. We currently use the Mac OS X 10.11 SDK, but are considering updating to the macOS 10.12 SDK, which introduced new low-level functions, such as `getentropy`. Apple uses a series of "availability macros" with new warnings in Clang to consider the case when the `MACOSX_DEPLOYMENT_TARGET` is below the introduction version for a given symbol. This has caused significant issues for `autoconf`-based projects, as running `configure` mistakes the built version into believing certain symbols are available, when they're not. As a result, deploying a viable build to users on older versions of Mac is rather complicated. Current "solutions" (in recent versions of Xcode) include: * Passing `-no_weak_imports` to the linker, which will fail any build using weakly-linked symbols (a pretty big hammer). * A new compiler warning `-Wunguarded-availability`, which detects uses that contradict the `MACOSX_DEPLOYMENT_TARGET` using some AST tricks. In our testing, neither had any effect on Python's `configure`, though these solutions worked for some other projects. We've implemented a workaround based on patching `pyconfig.h` after `configure` is complete, but ideally, Python's `configure` should detect `MACOSX_DEPLOYMENT_TARGET` and act accordingly (e.g. if `MACOSX_DEPLOYMENT_TARGET` is below 10.12, `getentropy` should not be considered available). The following symbols are affected: New in 10.13 (Python 3 only): utimensat HAVE_UTIMENSAT futimensHAVE_FUTIMENS New in 10.12: getentropy HAVE_GETENTROPY (Python 2/3) clock_gettime HAVE_CLOCK_GETTIME (Python 2/3) clock_settime HAVE_CLOCK_SETTIME (Python 3) clock_getresHAVE_CLOCK_GETRES (Python 3) New in 10.10 (Python 3 only): fstatat HAVE_FSTATAT faccessat HAVE_FACCESSAT fchmodatHAVE_FCHMODAT fchownatHAVE_FCHOWNAT linkat HAVE_LINKAT fdopendir HAVE_FDOPENDIR mkdirat HAVE_MKDIRAT renameatHAVE_RENAMEAT unlinkatHAVE_UNLINKAT readlinkat HAVE_READLINKAT symlinkat HAVE_SYMLINKAT openat HAVE_OPENAT -- messages: 301432 nosy: Maxime Belanger priority: normal severity: normal status: open title: `configure` script incorrectly detects symbols as available on Mac w/ Xcode 8+ versions: Python 2.7, Python 3.4, Python 3.5, Python 3.6 ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue31359> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27596] Build failure with Xcode 8 beta on OSX 10.11
Maxime Belanger added the comment: Glad to hear it! Thanks for the quick followup :). -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue27596> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27596] Build failure with Xcode 8 beta on OSX 10.11
Maxime Belanger added the comment: We're hitting this issue with Python 2.7 (which we deploy on Mac OS X 10.6-10.12). We've worked around it by manually patching `pyconfig.h` to un-define `HAVE_GETENTROPY` before `make`ing. Is a patch is in the works to support weak-linking this symbol (and the other new Sierra additions)? -- nosy: +Maxime Belanger ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue27596> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23534] `test_longdouble` fails on Mac when using system libffi (version 3.1)
New submission from Maxime Belanger: Greetings, We're compiling a custom version of Python 2.7.9 for the Mac (building on OS X 10.9 with Xcode 6.1), and we're instructing Python to use a vanilla copy of `libffi` (that we've also compiled ourselves) using the `--with-system-ffi` flag. We're running the regression test suite for sanity purposes and we've noticed this error in `test_ctypes`'s `test_longdouble` method: == FAIL: test_longdouble (ctypes.test.test_callbacks.Callbacks) -- Traceback (most recent call last): File ../Python.framework/Versions/2.7/lib/python2.7/ctypes/test/test_callbacks.py, line 88, in test_longdouble self.check_type(c_longdouble, 3.14) File ../Python.framework/Versions/2.7/lib/python2.7/ctypes/test/test_callbacks.py, line 24, in check_type self.assertEqual(result, arg) AssertionError: 0.0 != 3.14 I'm reasonably convinced this is caused by the version of `libffi` we're using. It seems that if the `--with-system-ffi` flag isn't set, an internal version of `libffi` is used instead. However, given that version 3.1 is said to be the version Python 2.7.9 is tracking, this feels like a bug to me. Also, it seems that trying to hit this code path in production code could trigger a crash. -- components: Tests, ctypes messages: 236729 nosy: Maxime Belanger priority: normal severity: normal status: open title: `test_longdouble` fails on Mac when using system libffi (version 3.1) type: crash versions: Python 2.7 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23534 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19884] Importing readline produces erroneous output
Maxime Belanger added the comment: I think the version check (`readline._READLINE_VERSION 0x0600`) is incorrect, as the test still fails for me on Mac OS X 10.9 with readline version 6.2 (0x602). Upgrading to 6.3 (0x603) fixes it, though. -- nosy: +Maxime Belanger ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19884 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
