[Python-Dev] [RELEASED] Python 3.5.4 is now available
On behalf of the Python development community and the Python 3.5 release team, I'm pleased to announce the availability of Python 3.5.4. Python 3.5.4 is the final 3.5 release in "bug fix" mode. The Python 3.5 branch has now transitioned into "security fixes mode"; all future improvements in the 3.5 branch will be security fixes only, and no conventional bug fixes will be merged. You can find Python 3.5.4 here: https://www.python.org/downloads/release/python-354/ Python 3.4.7 final will be released later today. Happy Pythoning, //arry/ ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Python 3.5 has now transitioned to "security fixes only" mode
The Python 3.5 branch has now entered "security fixes only" mode. No more bugfixes will be accepted into the 3.5 branch. In keeping with our modern workflow, I have changed the permissions on the 3.5 branch on Github so that only release managers can accept PRs into the branch. Please add me to your 3.5 security fix PRs, as I'm the one responsible for accepting them. (This was already true for the 3.4 branch, too.) I neglected to mention it in the release announcement, but this transition also means no more binary installers for 3.5 releases. This signals the end of my interactions with macOS Platform Expert Ned Deily and Windows Platform Expert Steve Dower when making releases. I just wanted to mention what a pleasure it has been working with those two gentlemen for the two Python releases I've RM'd for. They've made me look good every time. Cheers, //arry/ ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Interrupt thread.join() with Ctrl-C / KeyboardInterrupt on Windows
Hi all, Is it possible that thread.join() cannot be interrupted on Windows, while it can be on Linux? Would this be a bug, or is it by design? import threading, time def wait(): time.sleep(1000) t = threading.Thread(target=wait) t.start() t.join() # Press Control-C now. It stops on Linux, while it hangs on Windows. Tested on Python 3.6. Thanks, Jonathan ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PyThreadState_GET() returns NULL from within PyImport_GetModuleDict()
On 07Aug2017 2231, Patrick Rutkowski wrote: So, it seems to be the case that picking a mismatched python binary causes the crash, __but only with python3, not with python36__. This makes me wonder what the differences is between the two in the first place. I was getting the crash to begin with because I was linking my Debug build with the release build python3.lib, since I thought it shouldn't matter. My problem is fixed now, but if anyone could sheld light on the details of why exactly it happened then I would certainy be interested. I'd love to be able to, but honestly I have basically no idea. There is only one implementation of PyThreadState_Get() (which is what the PyThreadState_GET macro should map to), and that should never return NULL without terminating the entire process. My best guess is that the API forwarding being done by python3.dll is failing somehow and is turning into a function that returns 0. Since you are embedding rather than building an extension module, I'd suggest linking directly to python36[_d].dll, since you will presumably be including the Python runtime with your application (this is exactly the use case for the embeddable package, in case you hadn't seen that). The python3[_d].dll is most useful when building a .pyd that may be used with multiple versions of Python 3. It has some limitations and many bugs though, which I don't know we can ever fully resolve, but all of which can be avoided in your case by not using it :) Hope that helps, Steve ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] [ANN] Daily Windows builds of Python 3.x
Hi all As part of a deal with Zach Ware at PyCon, I agreed that if he removed the Subversion dependency from our builds, I would set up daily Windows builds of Python. Zach did an excellent job, and so I am now following through on my half of the deal :) For a while I've been uploading the official releases to nuget.org. These packages can be installed with nuget.exe (latest version always available at https://aka.ms/nugetclidl), which is quickly becoming a standard tool in Microsoft's build toolsets. It's very much a CI-focused package manager, rather than a user-focused one, and CI on Windows was previously an area where it was difficult to use Python. See the official feed at https://www.nuget.org/packages/python, and related packages pythonx86, python2 and python2x86. For people looking for an official "no installer" version of Python for Windows, this is it. And since all the infrastructure was there already, I decided to publish daily builds in a similar way to myget.org: https://www.myget.org/feed/python/package/nuget/pythondaily To install the latest daily build, run nuget.exe with this command: nuget.exe pythondaily -Source https://www.myget.org/F/python/api/v3/index.json (Note that if you already have a "pythondaily" package in that directory, nuget will consider the requirement satisfied. As I said, it's meant for reproducible CI builds rather than users who want to update things in the least amount of keystrokes :) ) The sys.version string contains the short commit hash. Please include this string when reporting bugs in these builds. Also, only the amd64 Release build is available pre-built. >>> sys.version '3.7.0a0 (remotes/origin/master:733d0f63c, Aug 8 2017, 15:56:14) [MSC v.1900 64 bit (AMD64)]' Hopefully this is valuable for people who want to include daily builds in their own test runs or validate recent bug fixes. Cheers, Steve ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [ANN] Daily Windows builds of Python 3.x
Thank you! I recall that we discussed that, but I understood that you was too busy to implement the idea. No, you didn't forget and you made it! ;-) Victor 2017-08-08 18:21 GMT+02:00 Steve Dower : > Hi all > > As part of a deal with Zach Ware at PyCon, I agreed that if he removed the > Subversion dependency from our builds, I would set up daily Windows builds > of Python. Zach did an excellent job, and so I am now following through on > my half of the deal :) > > For a while I've been uploading the official releases to nuget.org. These > packages can be installed with nuget.exe (latest version always available at > https://aka.ms/nugetclidl), which is quickly becoming a standard tool in > Microsoft's build toolsets. It's very much a CI-focused package manager, > rather than a user-focused one, and CI on Windows was previously an area > where it was difficult to use Python. > > See the official feed at https://www.nuget.org/packages/python, and related > packages pythonx86, python2 and python2x86. > > For people looking for an official "no installer" version of Python for > Windows, this is it. > > > And since all the infrastructure was there already, I decided to publish > daily builds in a similar way to myget.org: > > https://www.myget.org/feed/python/package/nuget/pythondaily > > To install the latest daily build, run nuget.exe with this command: > > nuget.exe pythondaily -Source > https://www.myget.org/F/python/api/v3/index.json > > (Note that if you already have a "pythondaily" package in that directory, > nuget will consider the requirement satisfied. As I said, it's meant for > reproducible CI builds rather than users who want to update things in the > least amount of keystrokes :) ) > > The sys.version string contains the short commit hash. Please include this > string when reporting bugs in these builds. Also, only the amd64 Release > build is available pre-built. > > >>> sys.version > '3.7.0a0 (remotes/origin/master:733d0f63c, Aug 8 2017, 15:56:14) [MSC > v.1900 64 bit (AMD64)]' > > Hopefully this is valuable for people who want to include daily builds in > their own test runs or validate recent bug fixes. > > Cheers, > Steve > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [ANN] Daily Windows builds of Python 3.x
On 8 August 2017 at 17:21, Steve Dower wrote: > For a while I've been uploading the official releases to nuget.org. These > packages can be installed with nuget.exe (latest version always available at > https://aka.ms/nugetclidl), which is quickly becoming a standard tool in > Microsoft's build toolsets. It's very much a CI-focused package manager, > rather than a user-focused one, and CI on Windows was previously an area > where it was difficult to use Python. > > See the official feed at https://www.nuget.org/packages/python, and related > packages pythonx86, python2 and python2x86. > > For people looking for an official "no installer" version of Python for > Windows, this is it. I've been aware of these builds for a while, but wasn't 100% sure of the status of them. It would be really useful if they could be publicised more widely - if for no other reason than to steer people towards these rather than the embedded distribution when these are more appropriate. But regardless of that minor point, the availability of these builds is really nice. > And since all the infrastructure was there already, I decided to publish > daily builds in a similar way to myget.org: > > https://www.myget.org/feed/python/package/nuget/pythondaily > > To install the latest daily build, run nuget.exe with this command: > > nuget.exe pythondaily -Source > https://www.myget.org/F/python/api/v3/index.json > > (Note that if you already have a "pythondaily" package in that directory, > nuget will consider the requirement satisfied. As I said, it's meant for > reproducible CI builds rather than users who want to update things in the > least amount of keystrokes :) ) > > The sys.version string contains the short commit hash. Please include this > string when reporting bugs in these builds. Also, only the amd64 Release > build is available pre-built. > > >>> sys.version > '3.7.0a0 (remotes/origin/master:733d0f63c, Aug 8 2017, 15:56:14) [MSC > v.1900 64 bit (AMD64)]' > > Hopefully this is valuable for people who want to include daily builds in > their own test runs or validate recent bug fixes. Nice! I can imagine these being a really useful resource for people wanting to (say) test against the development version in their Appveyor builds. Thanks for putting the effort into producing these :-) Paul ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Interrupt thread.join() with Ctrl-C / KeyboardInterrupt on Windows
On Tue, Aug 8, 2017 at 2:54 AM, Jonathan Slenders wrote: > Hi all, > > Is it possible that thread.join() cannot be interrupted on Windows, while it > can be on Linux? > Would this be a bug, or is it by design? > > > import threading, time > def wait(): > time.sleep(1000) > t = threading.Thread(target=wait) > t.start() > t.join() # Press Control-C now. It stops on Linux, while it hangs on > Windows. This comes down to a difference in how the Linux and Windows low-level APIs handle control-C and blocking functions: on Linux, the default is that any low-level blocking function can be interrupted by a control-C or any other random signal, and it's the calling code's job to check for this and restart it if necessary. This is annoying because it means that every low-level function call inside the Python interpreter has to have a bunch of boilerplate to detect this and retry, but OTOH it means that control-C automatically works in (almost) all cases. On Windows, they made the opposite decision: low-level blocking functions are never automatically interrupted by control-C. It's a reasonable design choice. The advantage is that sloppily written programs tend to work better -- on Linux you kind of *have* to put a retry loop around *every* low level call or your program will suffer weird random bugs, and on Windows you don't. But for carefully written programs like CPython this is actually pretty annoying, because if you *do* want to wake up on a control-C, then on Windows that has to be laboriously implemented on a case-by-case basis for each blocking function, and often this requires some kind of cleverness or is actually impossible, depending on what function you want to interrupt. At least on Linux the retry loop is always the same. The end result is that on Windows, control-C almost never works to wake up a blocked Python process, with a few special exceptions where someone did the work to implement this. On Python 2 the only functions that have this implemented are time.sleep() and multiprocessing.Semaphore.acquire; on Python 3 there are a few more (you can grep the source for _PyOS_SigintEvent to find them), but Thread.join isn't one of them. It looks like Thread.join ultimately ends up blocking in Python/thread_nt.h:EnterNonRecursiveMutex, which has a maze of #ifdefs behind it -- I think there are 3 different implementation you might end up with, depending on how CPython was built? Two of them seem to ultimately block in WaitForSingleObject, which would be easy to adapt to handle control-C. Unfortunately I think the implementation that actually gets used on modern systems is the one that blocks in SleepConditionVariableSRW, and I don't see any easy way for a control-C to interrupt that. But maybe I'm missing something -- I'm not a Windows expert. -n -- Nathaniel J. Smith -- https://vorpus.org ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Interrupt thread.join() with Ctrl-C / KeyboardInterrupt on Windows
Thank you Nathaniel for the response! Really interesting and helpful. 2017-08-08 20:51 GMT+02:00 Nathaniel Smith : > On Tue, Aug 8, 2017 at 2:54 AM, Jonathan Slenders > wrote: > > Hi all, > > > > Is it possible that thread.join() cannot be interrupted on Windows, > while it > > can be on Linux? > > Would this be a bug, or is it by design? > > > > > > import threading, time > > def wait(): > > time.sleep(1000) > > t = threading.Thread(target=wait) > > t.start() > > t.join() # Press Control-C now. It stops on Linux, while it hangs on > > Windows. > > This comes down to a difference in how the Linux and Windows low-level > APIs handle control-C and blocking functions: on Linux, the default is > that any low-level blocking function can be interrupted by a control-C > or any other random signal, and it's the calling code's job to check > for this and restart it if necessary. This is annoying because it > means that every low-level function call inside the Python interpreter > has to have a bunch of boilerplate to detect this and retry, but OTOH > it means that control-C automatically works in (almost) all cases. On > Windows, they made the opposite decision: low-level blocking functions > are never automatically interrupted by control-C. It's a reasonable > design choice. The advantage is that sloppily written programs tend to > work better -- on Linux you kind of *have* to put a retry loop around > *every* low level call or your program will suffer weird random bugs, > and on Windows you don't. > > But for carefully written programs like CPython this is actually > pretty annoying, because if you *do* want to wake up on a control-C, > then on Windows that has to be laboriously implemented on a > case-by-case basis for each blocking function, and often this requires > some kind of cleverness or is actually impossible, depending on what > function you want to interrupt. At least on Linux the retry loop is > always the same. > > The end result is that on Windows, control-C almost never works to > wake up a blocked Python process, with a few special exceptions where > someone did the work to implement this. On Python 2 the only functions > that have this implemented are time.sleep() and > multiprocessing.Semaphore.acquire; on Python 3 there are a few more > (you can grep the source for _PyOS_SigintEvent to find them), but > Thread.join isn't one of them. > > It looks like Thread.join ultimately ends up blocking in > Python/thread_nt.h:EnterNonRecursiveMutex, which has a maze of #ifdefs > behind it -- I think there are 3 different implementation you might > end up with, depending on how CPython was built? Two of them seem to > ultimately block in WaitForSingleObject, which would be easy to adapt > to handle control-C. Unfortunately I think the implementation that > actually gets used on modern systems is the one that blocks in > SleepConditionVariableSRW, and I don't see any easy way for a > control-C to interrupt that. But maybe I'm missing something -- I'm > not a Windows expert. > > -n > > -- > Nathaniel J. Smith -- https://vorpus.org > ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Interrupt thread.join() with Ctrl-C / KeyboardInterrupt on Windows
On 08Aug2017 1151, Nathaniel Smith wrote: It looks like Thread.join ultimately ends up blocking in Python/thread_nt.h:EnterNonRecursiveMutex, which has a maze of #ifdefs behind it -- I think there are 3 different implementation you might end up with, depending on how CPython was built? Two of them seem to ultimately block in WaitForSingleObject, which would be easy to adapt to handle control-C. Unfortunately I think the implementation that actually gets used on modern systems is the one that blocks in SleepConditionVariableSRW, and I don't see any easy way for a control-C to interrupt that. But maybe I'm missing something -- I'm not a Windows expert. I'd have to dig back through the recent attempts at changing this, but I believe the SleepConditionVariableSRW path is unused for all versions of Windows. A couple of people (including myself) attempted to enable that code path, but it has some subtle issues that were causing test failures, so we abandoned all the attempts. Though ISTR that someone put in more effort than most of us, but I don't think we've merged it (and if we have, it'd only be in 3.7 at this stage). Cheers, Steve ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Interrupt thread.join() with Ctrl-C / KeyboardInterrupt on Windows
On Tue, Aug 8, 2017 at 2:29 PM, Steve Dower wrote: > On 08Aug2017 1151, Nathaniel Smith wrote: >> >> It looks like Thread.join ultimately ends up blocking in >> Python/thread_nt.h:EnterNonRecursiveMutex, which has a maze of #ifdefs >> behind it -- I think there are 3 different implementation you might >> end up with, depending on how CPython was built? Two of them seem to >> ultimately block in WaitForSingleObject, which would be easy to adapt >> to handle control-C. Unfortunately I think the implementation that >> actually gets used on modern systems is the one that blocks in >> SleepConditionVariableSRW, and I don't see any easy way for a >> control-C to interrupt that. But maybe I'm missing something -- I'm >> not a Windows expert. > > > I'd have to dig back through the recent attempts at changing this, but I > believe the SleepConditionVariableSRW path is unused for all versions of > Windows. > > A couple of people (including myself) attempted to enable that code path, > but it has some subtle issues that were causing test failures, so we > abandoned all the attempts. Though ISTR that someone put in more effort than > most of us, but I don't think we've merged it (and if we have, it'd only be > in 3.7 at this stage). Ah, you're right -- the comments say it's used on Windows 7 and later, but the code disagrees. Silly me for trusting the comments :-). So it looks like it would actually be fairly straightforward to add control-C interruption support. -n -- Nathaniel J. Smith -- https://vorpus.org ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [RELEASED] Python 3.5.4 is now available
I updated some websites and services for the 3.5.4 release: * Status of Python branches in the devguide: https://devguide.python.org/#status-of-python-branches * Python security vulnerabilities: http://python-security.readthedocs.io/vulnerabilities.html * I removed all Python 3.5 buildbots: https://github.com/python/buildmaster-config/pull/17 Zachary Ware told me that 3.5.4 now comes before 3.6.2 on https://www.python.org/downloads/ but I don't think that it's a real issue :-) *Maybe* we could group releases per Python branch? Or sort by Python branches. Victor 2017-08-08 12:58 GMT+02:00 Larry Hastings : > > On behalf of the Python development community and the Python 3.5 release > team, I'm pleased to announce the availability of Python 3.5.4. > > Python 3.5.4 is the final 3.5 release in "bug fix" mode. The Python 3.5 > branch has now transitioned into "security fixes mode"; all future > improvements in the 3.5 branch will be security fixes only, and no > conventional bug fixes will be merged. > > > You can find Python 3.5.4 here: > > https://www.python.org/downloads/release/python-354/ > > > Python 3.4.7 final will be released later today. > > > Happy Pythoning, > > > /arry > > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com > ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Interrupt thread.join() with Ctrl-C / KeyboardInterrupt on Windows
On 08Aug2017 1512, Nathaniel Smith wrote: On Tue, Aug 8, 2017 at 2:29 PM, Steve Dower wrote: On 08Aug2017 1151, Nathaniel Smith wrote: It looks like Thread.join ultimately ends up blocking in Python/thread_nt.h:EnterNonRecursiveMutex, which has a maze of #ifdefs behind it -- I think there are 3 different implementation you might end up with, depending on how CPython was built? Two of them seem to ultimately block in WaitForSingleObject, which would be easy to adapt to handle control-C. Unfortunately I think the implementation that actually gets used on modern systems is the one that blocks in SleepConditionVariableSRW, and I don't see any easy way for a control-C to interrupt that. But maybe I'm missing something -- I'm not a Windows expert. I'd have to dig back through the recent attempts at changing this, but I believe the SleepConditionVariableSRW path is unused for all versions of Windows. A couple of people (including myself) attempted to enable that code path, but it has some subtle issues that were causing test failures, so we abandoned all the attempts. Though ISTR that someone put in more effort than most of us, but I don't think we've merged it (and if we have, it'd only be in 3.7 at this stage). Ah, you're right -- the comments say it's used on Windows 7 and later, but the code disagrees. Silly me for trusting the comments :-). So it looks like it would actually be fairly straightforward to add control-C interruption support. Except we're still hypothesising that the native condition variables will be faster than our emulation. I think until we prove or disprove that with a correct implementation, I'd rather not make a promise that Ctrl+C will work in situations where we depend on it. That's not to say that it isn't possible to continue fixing Ctrl+C handling in targeted locations. But I don't want to guarantee that an exception case like that will always work given there's a chance it may prevent us getting a performance benefit in the normal case. (I'm trying to advise caution, rather than saying it'll never happen.) Cheers, Steve ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] first post introduction and question regarding lto
On 8 August 2017 at 10:12, Gregory P. Smith wrote: > I don't know whether it is beneficial or not - but having the capability to > build LTO without PGO seems reasonable. I can review any pull requests > altering configure.ac and Makefile.pre.in to make such a change. Being able to separate them seems useful even if it's just from the performance research perspective of comparing "PGO only", "LTO only" and "PGO+LTO". The LTO only numbers may be of questionable relevance to us as CPython developers, but I can definitely see them being of interest to the folks working on C compiler toolchains. Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [ANN] Daily Windows builds of Python 3.x
On 9 August 2017 at 03:59, Paul Moore wrote: > On 8 August 2017 at 17:21, Steve Dower wrote: >> For a while I've been uploading the official releases to nuget.org. These >> packages can be installed with nuget.exe (latest version always available at >> https://aka.ms/nugetclidl), which is quickly becoming a standard tool in >> Microsoft's build toolsets. It's very much a CI-focused package manager, >> rather than a user-focused one, and CI on Windows was previously an area >> where it was difficult to use Python. >> >> See the official feed at https://www.nuget.org/packages/python, and related >> packages pythonx86, python2 and python2x86. >> >> For people looking for an official "no installer" version of Python for >> Windows, this is it. > > I've been aware of these builds for a while, but wasn't 100% sure of > the status of them. It would be really useful if they could be > publicised more widely - if for no other reason than to steer people > towards these rather than the embedded distribution when these are > more appropriate. The trade-offs between the various options for managing Python runtimes on Windows would likely make sense as a packaging.python.org discussion: https://packaging.python.org/discussions/ The woefully incomplete discussion on application deployment (https://packaging.python.org/discussions/deploying-python-applications/) could then be updated to reference that rather than having to cover it inline. Cheers, Nick. P.S. Thanks to https://github.com/pypa/python-packaging-user-guide/issues/317, we have 3 clearly distinct categories of docs in PyPUG these days: tutorials, where we deliberately only present one option to avoid overwhelming readers with too much information, guides, where we're still opinionated, but acknowledge alternatives, and discussions, where the overall tone is "the right option depends on your goals" -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
