Re: [Python-Dev] Late Python 3.7.1 changes to fix the C locale coercion (PEP 538) implementation
On Wed, 19 Sep 2018 at 22:07, Victor Stinner wrote: > > > IMHO the implementation is really a secondary concern here, the main > > question is: what is the correct behavior? > > > > Nick: > > > > * Do we agree that we need to provide a way to disable C locale > > coercion (PEP 538) even when -E is used? > > * Do you agree that Py_Initialize() and Py_Main() must not enable the > > C locale coercion (PEP 538)? > > > > I understood that your reply is yes for the second question, since you > > insist to push your change which also prevent Py_Initialize() and > > Py_Main() to enable C locale coercion. > > Hum, I'm not sure if I explained properly my opinion on these questions. > > I consider that Python 3.7.0 introduced a regression compared to > Python 3.6: it changes the LC_CTYPE locale for Python and all child > processes and it's not possible to opt-out for that when using -E > command line option. This *wasn't* broken in the original PEP 538 implementation - it was only broken when you ignored the PEP and tried to make everything work the same way PEP 540 did, including moving the coercion out of the Python CLI and into the runtime library APIs. I still think the locale coercion handling in Python 3.7.x is broken, but adding MORE code is NOT the right answer: going back to the original (correct) implementation is. So changing it back to the way the PEP is supposed to work is fine, making everything more complicated for no good reason whatsoever is not fine. What changed is the fact I decided it wasn't worth holding up 3.7.1 over (and it certainly isn't worth adding a new -X option in a point release). Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Late Python 3.7.1 changes to fix the C locale coercion (PEP 538) implementation
On Thu, 20 Sep 2018 at 06:48, Ned Deily wrote: > > On Sep 19, 2018, at 15:08, Victor Stinner wrote: > > Le mardi 18 septembre 2018, Victor Stinner a écrit : > > > Hi Unicode and locales lovers, > > > > > > tl; dr Nick, Ned, INADA-san: I modified 3.7.1 to add a new "-X > > > coerce_c_locale=value" option and make sure that the C locale coercion > > > cannot be when Python in embedded: are you ok with these changes? > > > > Nick asked me to revert, which means that no, he is not ok with these > > changes. > > > > I reverted my change in 3.7. > > Thank you, Victor! > > Nick, with regard to this does the current state of the 3.7 branch look > acceptable now for a 3.7.1? It's still broken relative to the PEP in the following respects: - Py_Initialize() coerces the C locale to C.UTF-8, even though it's not supposed to - Py_Main() coerces the C locale to C.UTF-8, even though it's not supposed to - PYTHONCOERCECLOCALE=0 doesn't work if -E or -I are passed on the command line (but it's supposed to) - PYTHONCOERCECLOCALE=warn doesn't work if -E or -I are passed on the command line (it's nominally supposed to do this too, but I'm less concerned about this one) The problem with Victor's patch is that instead of reverting to the as-designed-and-accepted PEP the way my PR (mostly) does, it instead introduces a whole new command line option (which then needs to be documented and tested), and still coerces *far* too late (not until Py_Initialise is already running, after who knows how much code in the embedding application has already executed). I don't have the time required to push through Victor's insistence that -E and -I are sacrosanct and must always be respected (despite PEP 538 explicitly saying that they won't be where PYTHONCOERCECLOCALE is concerned), and so we can't *possibly* change back to having the locale coercion work the way I originally implemented it, so I wrote the 3.7.x series off as a lost cause, and decided to devote my energies to getting things back to the way they were supposed to be for 3.8+. Regards, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Late Python 3.7.1 changes to fix the C locale coercion (PEP 538) implementation
On Thu, 20 Sep 2018 at 20:20, Nick Coghlan wrote: > > On Thu, 20 Sep 2018 at 06:48, Ned Deily wrote: > > > > On Sep 19, 2018, at 15:08, Victor Stinner wrote: > > > Le mardi 18 septembre 2018, Victor Stinner a écrit : > > > > Hi Unicode and locales lovers, > > > > > > > > tl; dr Nick, Ned, INADA-san: I modified 3.7.1 to add a new "-X > > > > coerce_c_locale=value" option and make sure that the C locale coercion > > > > cannot be when Python in embedded: are you ok with these changes? > > > > > > Nick asked me to revert, which means that no, he is not ok with these > > > changes. > > > > > > I reverted my change in 3.7. > > > > Thank you, Victor! > > > > Nick, with regard to this does the current state of the 3.7 branch look > > acceptable now for a 3.7.1? > > It's still broken relative to the PEP in the following respects: > > - Py_Initialize() coerces the C locale to C.UTF-8, even though it's > not supposed to > - Py_Main() coerces the C locale to C.UTF-8, even though it's not supposed to > - PYTHONCOERCECLOCALE=0 doesn't work if -E or -I are passed on the > command line (but it's supposed to) > - PYTHONCOERCECLOCALE=warn doesn't work if -E or -I are passed on the > command line (it's nominally supposed to do this too, but I'm less > concerned about this one) It's worth noting that even though the PYTHONCOERCECLOCALE=0 off switch doesn't currently work as described in PEP 538 when passing -E or -I, setting "LC_ALL=C" does (since that's handled by the C library, independently of any CPython command line flags). Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Nearly - all tests PASS for AIX
On Wed, 19 Sep 2018 at 00:00, Michael wrote: > On 17/09/2018 12:50, Michael wrote: > > Dear all, > > > > The last two months I have spent nearly all my free time to cleanup "a > > frustration" - from my side - the long list of failing tests for AIX > > (there were nearly 20 when I started). > > == Tests result: SUCCESS == > > 393 tests OK. Nice! > 1 test altered the execution environment: > test_threading > > 25 tests skipped: > test_dbm_gnu test_devpoll test_epoll test_gdb test_idle > test_kqueue test_lzma test_msilib test_ossaudiodev test_readline > test_spwd test_sqlite test_startfile test_tcl test_tix test_tk > test_ttk_guionly test_ttk_textonly test_turtle test_unicode_file > test_unicode_file_functions test_winconsoleio test_winreg > test_winsound test_zipfile64 > > Total duration: 13 min 30 sec > Tests result: SUCCESS > > May I put this up as a PR - not for merging - but to see how it > performs, or does not perform, with the Travis Ci, etc. tests? That seems like a reasonable approach to me - it will also allow folks to give the changes a quick skim and provide suggestions for splitting it up into more easily reviewed PRs. Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] debugging test_importlib.test_bad_traverse - script status is SUCCESS - but FAIL is expected.
On Tue, 18 Sep 2018 at 19:52, Michael wrote: > > On 17/09/2018 09:39, Michael wrote: > > I read the discussion related to issue32374. That seems to be sure that > > other events that could > > cause the test to fail (i.e., the program executes successfully) are > > caught early, and/or ignored > > so that the program fails - and the test succeeds. > > After reading below, I would appreciate knowing whether to ask that > issue32374 be reopened and the test adjusted so that the test is > "SkipIf" AIX? Or, something else? I'll work on something else, but I do > not want to guess the current intent of this test module. Reviewing https://bugs.python.org/issue32374, the purpose of the test case is to check that failing to handle the m_state == NULL case will always segfault (because the import machinery always checks that m_traverse is valid after the create stage), rather than only segfaulting sometimes (based on whether or not a cyclic gc run triggers at an inopportune moment). Since the AIX case won't segfault in either the deliberately triggered traversal *or* in a GC-induced traversal, skipping the test case on AIX seems fine (with the note that null pointer accesses are just zero on AIX, not a segfault). Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] SEC: Spectre variant 2: GCC: -mindirect-branch=thunk -mindirect-branch-register
On Tue, Sep 18, 2018 at 8:38 AM INADA Naoki wrote: > I think this topic should split to two topics: (1) Guard Python > process from Spectre/Meltdown > attack from other process, (2) Prohibit Python code attack other > processes by using > Spectre/Meltdown. (3) Guard Python from performance degradation by overly aggressive Spectre "mitigation". ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] SEC: Spectre variant 2: GCC: -mindirect-branch=thunk -mindirect-branch-register
On Thursday, September 20, 2018, Stefan Ring wrote: > On Tue, Sep 18, 2018 at 8:38 AM INADA Naoki > wrote: > > > I think this topic should split to two topics: (1) Guard Python > > process from Spectre/Meltdown > > attack from other process, (2) Prohibit Python code attack other > > processes by using > > Spectre/Meltdown. > > (3) Guard Python from performance degradation by overly aggressive > Spectre "mitigation". > Spectre has the potential of having a greater impact on cloud providers than Meltdown. Whereas Meltdown allows unauthorized applications to read from privileged memory to obtain sensitive data from processes running on the same cloud server, Spectre can allow malicious programs to induce a hypervisor to transmit the data to a guest system running on top of it. - Private SSL certs - Cached keys and passwords in non-zeroed RAM - [...] https://en.wikipedia.org/wiki/Spectre_(security_vulnerability) I really shouldn't need to apologise for bringing this up here. Here's one: https://github.com/Eugnis/spectre-attack/blob/master/Source.c Is this too slow in CPython with: - Coroutines (asyncio (tulip)) - PyPy JIT * - Numba JIT * - C Extensions * - Cython * * Not anyone here's problem. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Nearly - all tests PASS for AIX
On 20/09/2018 15:18, Nick Coghlan wrote: > That seems like a reasonable approach to me - it will also allow folks > to give the changes a quick skim and provide suggestions for splitting > it up into more easily reviewed PRs. I already have them as individual PR's (8 Open) - https://github.com/python/cpython/pulls?q=is%3Aopen+is%3Apr+author%3Aaixtools+sort%3Aupdated-desc But I'll add the combined one to get it through grinder and see if there are unexpected surprises. Michael > > Cheers, > Nick. signature.asc Description: OpenPGP digital signature ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Store startup modules as C structures for 20%+ startup speed improvement?
On Wed, Sep 19, 2018 at 12:32 AM, Stefan Behnel wrote: > Also, one thing that would be interesting to find out is whether constant > Python data structures can actually be pre-allocated in the data segment > (and I mean their object structs) . Then things like tuples of strings > (argument lists and what not) could be loaded and the objects quickly > initialised (although, is that even necessary?), rather than having to heap > allocate and create them. Probably something that we should try out in > Cython. > I might not be fully understanding the scope of your question but this patch does allocate constant data structures in the data segment. We could be more aggressive with that but we limit our scope to what is presented to the un-marshaling code. This may be relevant to Cython, as well. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Store startup modules as C structures for 20%+ startup speed improvement?
Carl Shapiro schrieb am 20.09.2018 um 20:21: > On Wed, Sep 19, 2018 at 12:32 AM, Stefan Behnel wrote: > >> Also, one thing that would be interesting to find out is whether constant >> Python data structures can actually be pre-allocated in the data segment >> (and I mean their object structs) . Then things like tuples of strings >> (argument lists and what not) could be loaded and the objects quickly >> initialised (although, is that even necessary?), rather than having to heap >> allocate and create them. Probably something that we should try out in >> Cython. > > I might not be fully understanding the scope of your question but this > patch does allocate constant data structures in the data segment. We could > be more aggressive with that but we limit our scope to what is presented to > the un-marshaling code. Ah, thanks, yes, it works recursively, also for tuples and code objects. Took me a while to figure out how to open the "frozemodules.c" file, but looking at that makes it clear. Yes, that's what I meant. > This may be relevant to Cython, as well. Totally. This might actually be more relevant for Cython than for CPython in the end, because it wouldn't be limited to the stdlib and its core modules. It's a bit more difficult for us, because this probably won't work easily across Python releases (2.[67] and 3.[45678] for now) and also definitely not for PyPy, but that just means some multiplication of the generated code, and we have the dynamic part of it already. Supporting that for Unicode strings will be fun, I'm sure. :) Stefan ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Store startup modules as C structures for 20%+ startup speed improvement?
Larry Hastings schrieb am 14.09.2018 um 23:27: > What the patch does: it takes all the Python modules that are loaded as > part of interpreter startup and deserializes the marshalled .pyc file into > precreated objects stored as static C data. What about the small integers cache? The C serialisation generates several PyLong objects that would normally reside in the cache. Is this handled somewhere? I guess the cache could entirely be loaded from the data segment. And the same would have to be done for interned strings. Basically anything that CPython only wants to have one instance of. That would severely limit the application of this optimisation to external modules, though. I don't see a way how they could load their data structures from the data segment without duplicating all sorts of "singletons". Stefan ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] SEC: Spectre variant 2: GCC: -mindirect-branch=thunk -mindirect-branch-register
On Thu, Sep 20, 2018 at 2:10 PM Wes Turner wrote: > > On Thursday, September 20, 2018, Stefan Ring wrote: >> >> On Tue, Sep 18, 2018 at 8:38 AM INADA Naoki wrote: >> >> > I think this topic should split to two topics: (1) Guard Python >> > process from Spectre/Meltdown >> > attack from other process, (2) Prohibit Python code attack other >> > processes by using >> > Spectre/Meltdown. >> >> (3) Guard Python from performance degradation by overly aggressive >> Spectre "mitigation". > > > > Spectre has the potential of having a greater impact on cloud providers > > than Meltdown. Whereas Meltdown allows unauthorized applications to read > > from privileged memory to obtain sensitive data from processes running on > > the same cloud server, Spectre can allow malicious programs to induce a > > hypervisor to transmit the data to a guest system running on top of it. > - Private SSL certs > - Cached keys and passwords in non-zeroed RAM > - [...] > > https://en.wikipedia.org/wiki/Spectre_(security_vulnerability) It's true that the attacks should worry cloud providers. Doesn't that mean that companies like Amazon, Microsoft (Steve), and Docker should have done analyses on CPython's vulnerability to these exploits? Has/should/can anyone officially representing Python contact the companies and ask them? When I followed your quote to find the context, I found it uses, as its source, a Forbes article. The source cited by THAT article is Daniel Gruss, who was one of the researchers. Should someone from the PSF contact the researchers? Steve says he spoke to some of them to judge whether the proposed compiler flags would help, and decided against it. Absent of expert input, here's my non-expert take: That quote requires an OS-level fix. A Python program without the proper permissions can't do such things unless there is a vulnerability with the OS, and it is extremely unlikely for anyone to update Python for Spectre but not update the OS (and they'd be screwed in any case). And even if there is a vulnerability in the OS, maybe the way to exploit it is by using arbitrary Python execution (which you need before you can TRY to use Spectre) on this Python interpreter. You can then write a new binary file and run THAT, and it will be fast enough. That's not something you can fix about CPython. Also, (again with my understanding) the problem of Spectre and Meltdown are that you can escape sandboxes and the like, such as the user/kernel divide, or a software sandbox like that provided by a JavaScript VM. For CPython to be "vulnerable" to these attacks, it needs to have some kind of sandbox or protection to break out of. Instead, we sometimes have sandboxes AROUND CPython (like Jupyter) or WITHIN CPython. I don't see how it makes sense to talk about a sandbox escape FOR CPython (yet). Your original post linked to a discussion about Linux using those build flags. Linux is a kernel, and has such protections that can be bypassed, so it has something to worry about. Malicious code can be native code, which (to my understanding) will be fast enough to exploit the cache miss time. Here's Google's article about the retpoline and why it helps: https://support.google.com/faqs/answer/7625886 As of yet, you have quoted passages that have little relevance to interpreter devs, especially non-JIT interpreters, and you have linked to entire articles for non-experts with little relevance to interpreter devs. This doesn't show that you have any better of an understanding than I have, which is less than the understanding that some of the Python devs have, and much less than what Steve has. In short, it LOOKS like you don't know what you're talking about. If you have a different and deeper understanding of the problem, then you need to show it, and say why there is a problem for CPython specifically. Or find someone who can do that for you. > Here's one: > https://github.com/Eugnis/spectre-attack/blob/master/Source.c > > Is this too slow in CPython with: > - Coroutines (asyncio (tulip)) > - PyPy JIT * > - Numba JIT * > - C Extensions * > - Cython * > > * Not anyone here's problem. C extensions are obviously fast enough. I think most of the other starred examples are fast enough, but it's probably more subtle than I think and requires further analysis by their devs. I also think there's something important I'm still missing about what's required and what it can do. I don't see what coroutines have to do with it. Coroutines are still Python code, and they're subject to the GIL. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
