Re: [Python-Dev] Encoding of PyFrameObject members
To get at the Python thread state from a signal handler (using 2.7 as a reference here; but i don't believe 3.4 has changed this part much) you need to modify the interpreter to expose pystate.c's "autoTLSkey" and thread.c's "struct key" as well as "keyhead" and "keymutex". >From there, in your signal handler you must try to acquire the newly exposed keymutex and do nothing if you were unable to get it. If you did acquire it (rare not to), you can walk the keyhead list looking for autoTLSkey to find the current valid thread state. I had an intern (hi Elena!) write a signal sampling based low overhead Python CPU profiler based on that last summer. I believe there are still bugs to shaken out (if they are even possible to fix... Armin's comments are true: signal handler code is super limited). I am stating this here because I want someone to pester me at PyCon if I haven't released our work as a proof of concept by then. The important take away: From what I could figure out, you need to modify the CPython interpreter to be more amenable to such introspection. A downside of a signal based profiler: *ALL* of the EINTR mishandling bugs within the Python interpreter, stdlib, and your own code will show up in your application. So until those are fixed (hooray for Antoine's PEP!), it may not be practical for use on production processes which is sort of the entire point of a low overhead sampling profiler... I'd like to get a buildbot setup that runs the testsuite while a continual barrage of signals are being generated. We really don't stress test that stuff (as evidence by the EINTR mishandling issues that are rampant) as non-fatal signals are so rare for most things... until they aren't. As a side note and encouragement: I wonder what PyPy could do for dynamically enabled and disabled low overhead CPU profiling. (take that as a hint that I want someone else to get extremely creative!) -gps On Sat Feb 07 2015 at 1:34:26 PM Greg Ewing wrote: > Maciej Fijalkowski wrote: > > However, you can't access thread > > locals from signal handlers (since in some cases it mallocs, thread > > locals are built lazily if you're inside the .so, e.g. if python is > > built with --shared) > > You might be able to use Py_AddPendingCall to schedule > what you want done outside the context of the signal > handler. > > The call will be made by the main thread, though, > so if you need to access the frame of whatever thread > was running when the signal occured, you will have > to track down its PyThreadState somehow and get the > frame from there. Not sure what would be involved > in doing that. > > -- > Greg > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > greg%40krypto.org > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] installing python 2.7.9 on a Mac
In article <201502070539.t175dmcj003...@fido.openend.se>, Laura Creighton wrote: > webmaster just got mail from a novice who is trying to learn Python in > an introductory class. She got a "The version of Tcl/Tk (8.5.7) in > use may be unstable" message. > > I think that the download page should have a link. > If you get > download and install . Any reason we cannot do that? The truncated "use may be unstable" message above is displayed when you run IDLE. The full message displayed already contains a link to the python.org web page that explains the most up-to-date recommendations for Tcl/Tk use with python.org installers on OS X and includes links to download options. That same link is included in the Welcome message and README text when running the installer for Python. It is also already linked to on the download pages for each release. Until we start shipping our own version of Tcl/Tk for OS X (which has been in the cards), I don't think we can make it much easier. -- Ned Deily, n...@acm.org ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Encoding of PyFrameObject members
Maciej Fijalkowski wrote: However, you can't access thread locals from signal handlers (since in some cases it mallocs, thread locals are built lazily if you're inside the .so, e.g. if python is built with --shared) You might be able to use Py_AddPendingCall to schedule what you want done outside the context of the signal handler. The call will be made by the main thread, though, so if you need to access the frame of whatever thread was running when the signal occured, you will have to track down its PyThreadState somehow and get the frame from there. Not sure what would be involved in doing that. -- Greg ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Encoding of PyFrameObject members
On 02/06/2015 11:48 PM, Francis Giraldeau wrote: > 2015-02-06 6:04 GMT-05:00 Armin Rigo: > > Hi, > > On 6 February 2015 at 08:24, Maciej Fijalkowski mailto:fij...@gmail.com>> wrote: > > I don't think it's safe to assume f_code is properly filled by the > > time you might read it, depending a bit where you find the frame > > object. Are you sure it's not full of garbage? > > > Yes, before discussing how to do the utf8 decoding, we should realize > that it is really unsafe code starting from the line before. From a > signal handler you're only supposed to read data that was written to > "volatile" fields. So even PyEval_GetFrame(), which is done by > reading the thread state's "frame" field, is not safe: this is not a > volatile. This means that the compiler is free to do crazy things > like *first* write into this field and *then* initialize the actual > content of the frame. The uninitialized content may be garbage, not > just NULLs. > > > Thanks for these comments. Of course accessing frames withing a signal handler is racy. I confirm that code encoded in non-ascii is not accessible from the uft8 buffer pointer. However, a call > to PyUnicode_AsUTF8() encodes the data and caches it in the unicode object. Later access returns the byte buffer without memory allocation and re-encoding. > > I think it is possible to solve both safety problems by registering a handler with PyPyEval_SetProfile(). On function entry, the handler will call PyUnicode_AsUTF8() on the required frame members to > make sure the utf8 encoded string is available. Then, we increment the refcount of the frame and assign it to a thread local pointer. On function return, the refcount is decremented. These operations > occurs in the normal context and they are not racy. The signal handler will use the thread local frame pointer instead of calling PyEval_GetFrame(). Does that sounds good? You could call Py_AddPendingCall() from your signal handler and access the frame members from the function scheduled by Py_AddPendingCall(). Xavier ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Encoding of PyFrameObject members
On Sat, Feb 7, 2015 at 12:48 AM, Francis Giraldeau wrote: > 2015-02-06 6:04 GMT-05:00 Armin Rigo : > >> Hi, >> >> On 6 February 2015 at 08:24, Maciej Fijalkowski wrote: >> > I don't think it's safe to assume f_code is properly filled by the >> > time you might read it, depending a bit where you find the frame >> > object. Are you sure it's not full of garbage? >> >> >> Yes, before discussing how to do the utf8 decoding, we should realize >> that it is really unsafe code starting from the line before. From a >> signal handler you're only supposed to read data that was written to >> "volatile" fields. So even PyEval_GetFrame(), which is done by >> reading the thread state's "frame" field, is not safe: this is not a >> volatile. This means that the compiler is free to do crazy things >> like *first* write into this field and *then* initialize the actual >> content of the frame. The uninitialized content may be garbage, not >> just NULLs. > > > Thanks for these comments. Of course accessing frames withing a signal > handler is racy. I confirm that code encoded in non-ascii is not accessible > from the uft8 buffer pointer. However, a call to PyUnicode_AsUTF8() encodes > the data and caches it in the unicode object. Later access returns the byte > buffer without memory allocation and re-encoding. > > I think it is possible to solve both safety problems by registering a > handler with PyPyEval_SetProfile(). On function entry, the handler will call > PyUnicode_AsUTF8() on the required frame members to make sure the utf8 > encoded string is available. Then, we increment the refcount of the frame > and assign it to a thread local pointer. On function return, the refcount is > decremented. These operations occurs in the normal context and they are not > racy. The signal handler will use the thread local frame pointer instead of > calling PyEval_GetFrame(). Does that sounds good? > > Thanks again for your feedback! > > Francis You still didn't explain what are you trying to achieve nor adressed armins questions about volatile. However, you can't access thread locals from signal handlers (since in some cases it mallocs, thread locals are built lazily if you're inside the .so, e.g. if python is built with --shared) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
