Re: The GIL and PyEval_RestoreThread
The thread variable I'm passing in is the one I originally got from calling
Py_NewInterpreter. I'd assumed that I didn't need to particularly track the
one I get back from SaveThread since it should always be the one I restored
previously (which does seem to be the case).
> It looks like you're resuming the same thread twice. As it's already
resumed the second time, no wonder it's not blocking!
That isn't how I read the docs though? It says "If the lock has been
created, the current thread must not have acquired it, otherwise deadlock
ensues." That suggests to me that it should try to acquire the GIL again
and wait until it can (although possibly also that it's not an expected use
and Python thread states are expected to be more 1:1 with C threads).
On Wed, Sep 27, 2023 at 3:53 AM MRAB via Python-list
wrote:
> On 2023-09-26 14:20, Peter Ebden via Python-list wrote:
> > Hi all,
> >
> > I've been working on embedding Python and have an interesting case around
> > locking with PyEval_RestoreThread which wasn't quite doing what I expect,
> > hoping someone can explain what I should expect here.
> >
> > I have a little example (I'm running this in parallel from two different
> > threads; I have some more C code for that but I don't think it's super
> > interesting):
> >
> > void run_python(PyThreadState* thread) {
> >LOG("Restoring thread %p...", thread);
> >PyEval_RestoreThread(thread);
> >LOG("Restored thread %p", thread);
> >PyRun_SimpleString("import time; print('sleeping'); time.sleep(3.0)");
> >LOG("Saving thread...");
> >PyThreadState* saved_thread = PyEval_SaveThread();
> >LOG("Saved thread %p", saved_thread);
> > }
> >
> > This produces output like
> > 11:46:48.110058893: Restoring thread 0xabc480...
> > 11:46:48.110121656: Restored thread 0xabc480
> > 11:46:48.110166060: Restoring thread 0xabc480...
> > sleeping
> > 11:46:48.110464194: Restored thread 0xabc480
> > sleeping
> > 11:46:51.111307541: Saving thread...
> > 11:46:51.111361075: Saved thread 0xabc480
> > 11:46:51.113116633: Saving thread...
> > 11:46:51.113177605: Saved thread 0xabc480
> >
> > The thing that surprises me is that both threads seem to be able to pass
> > PyEval_RestoreThread before either reaches the corresponding
> > PyEval_SaveThread call, which I wasn't expecting to happen; I assumed
> that
> > since RestoreThread acquires the GIL, that thread state would remain
> locked
> > until it's released.
> >
> > I understand that the system occasionally switches threads, which I guess
> > might well happen with that time.sleep() call, but I wasn't expecting the
> > same thread to become usable somewhere else. Maybe I am just confusing
> > things by approaching the same Python thread from multiple OS threads
> > concurrently and should be managing my own locking around that?
> >
> Storing the result of PyEval_SaveThread in a local variable looks wrong
> to me.
>
> In the source for the regex module, I release the GIL with
> PyEval_SaveThread and save its result. Then, when I want to claim the
> GIL, I pass that saved value to PyEval_RestoreThread.
>
> You seem to be releasing the GIL and discarding the result, so which
> thread are you resuming when you call PyEval_RestoreThread?
>
> It looks like you're resuming the same thread twice. As it's already
> resumed the second time, no wonder it's not blocking!
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>
--
Thought Machine Group Limited, a company registered in England & Wales.
Registered number: 4277.
Registered Office: 5 New Street Square,
London EC4A 3TW
<https://maps.google.com/?q=5+New+Street+Square,+London+EC4A+3TW=gmail=g>.
The content of this email is confidential and intended for the recipient
specified in message only. It is strictly forbidden to share any part of
this message with any third party, without a written consent of the sender.
If you received this message by mistake, please reply to this message and
follow with its deletion, so that we can ensure such a mistake does not
occur in the future.
--
https://mail.python.org/mailman/listinfo/python-list
The GIL and PyEval_RestoreThread
Hi all,
I've been working on embedding Python and have an interesting case around
locking with PyEval_RestoreThread which wasn't quite doing what I expect,
hoping someone can explain what I should expect here.
I have a little example (I'm running this in parallel from two different
threads; I have some more C code for that but I don't think it's super
interesting):
void run_python(PyThreadState* thread) {
LOG("Restoring thread %p...", thread);
PyEval_RestoreThread(thread);
LOG("Restored thread %p", thread);
PyRun_SimpleString("import time; print('sleeping'); time.sleep(3.0)");
LOG("Saving thread...");
PyThreadState* saved_thread = PyEval_SaveThread();
LOG("Saved thread %p", saved_thread);
}
This produces output like
11:46:48.110058893: Restoring thread 0xabc480...
11:46:48.110121656: Restored thread 0xabc480
11:46:48.110166060: Restoring thread 0xabc480...
sleeping
11:46:48.110464194: Restored thread 0xabc480
sleeping
11:46:51.111307541: Saving thread...
11:46:51.111361075: Saved thread 0xabc480
11:46:51.113116633: Saving thread...
11:46:51.113177605: Saved thread 0xabc480
The thing that surprises me is that both threads seem to be able to pass
PyEval_RestoreThread before either reaches the corresponding
PyEval_SaveThread call, which I wasn't expecting to happen; I assumed that
since RestoreThread acquires the GIL, that thread state would remain locked
until it's released.
I understand that the system occasionally switches threads, which I guess
might well happen with that time.sleep() call, but I wasn't expecting the
same thread to become usable somewhere else. Maybe I am just confusing
things by approaching the same Python thread from multiple OS threads
concurrently and should be managing my own locking around that?
Thanks in advance,
Peter
--
Thought Machine Group Limited, a company registered in England & Wales.
Registered number: 4277.
Registered Office: 5 New Street Square,
London EC4A 3TW
.
The content of this email is confidential and intended for the recipient
specified in message only. It is strictly forbidden to share any part of
this message with any third party, without a written consent of the sender.
If you received this message by mistake, please reply to this message and
follow with its deletion, so that we can ensure such a mistake does not
occur in the future.
--
https://mail.python.org/mailman/listinfo/python-list
[issue34722] Non-deterministic bytecode generation
Change by Peter Ebden : -- keywords: +patch pull_requests: +8885 stage: -> patch review ___ Python tracker <https://bugs.python.org/issue34722> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34722] Non-deterministic bytecode generation
Peter Ebden added the comment: Thanks for the pointer, I'll have a bit more of a dig into it (although Serhiy makes a good point too...). -- ___ Python tracker <https://bugs.python.org/issue34722> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34722] Non-deterministic bytecode generation
New submission from Peter Ebden :
We've found that the following code produces non-deterministic bytecode,
even post PEP-552:
def test(x):
if x in {'ONE', 'TWO', 'THREE'}:
pass
It's not too hard to test it:
$ python3.7 -m compileall --invalidation-mode=unchecked-hash test.py
Compiling 'test.py'...
$ sha1sum __pycache__/test.cpython-37.pyc
61e5682ca95e8707b4ef2a79f6454dafd800 __pycache__/test.cpython-37.pyc
$ rm __pycache__/test.cpython-37.pyc
$ python3.7 -m compileall --invalidation-mode=unchecked-hash test.py
Compiling 'test.py'...
$ sha1sum __pycache__/test.cpython-37.pyc
222a06621b491879e5317b34e9dd715bacd89b7d __pycache__/test.cpython-37.pyc
It looks like the peephole optimiser is converting the LOAD_CONST instructions
for the set into a single LOAD_CONST for a frozenset which then serialises in
nondeterministic order. One can hence work around it by setting PYTHONHASHSEED
to a known value.
I'm happy to help out with this if needed, although I don't have a lot of
familiarity with the relevant code.
--
components: Interpreter Core
messages: 325644
nosy: Peter Ebden
priority: normal
severity: normal
status: open
title: Non-deterministic bytecode generation
versions: Python 3.7
___
Python tracker
<https://bugs.python.org/issue34722>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29094] Regression in zipfile writing in 2.7.13
New submission from Peter Ebden:
In Python 2.7.13, using zipfile.ZipFile to write into a file with some initial
preamble produces a zip file that cannot be read again by some zip
implementations.
Our use case is using pex (https://github.com/pantsbuild/pex) which writes a
zip that begins with a shebang, and later attempting to manipulate that using
Go's standard archive/zip package. In 2.7.12 that works OK, but in 2.7.13 the
.pex file is rejected on reading. Linux's command-line unzip tool will read the
archive, but issues a warning ("4 extra bytes at beginning or within zipfile")
which wasn't present previously.
zipfile.ZipFile does read the files OK.
I assume this is related to https://bugs.python.org/issue26293 since that's the
most obvious zipfile change in 2.7.13. It's pretty easy to reproduce using the
example in that issue:
from zipfile import ZipFile
with open('a.zip', 'wb') as base:
base.write(b'old\n')
with ZipFile(base, 'a') as myzip:
myzip.write('eggs.txt')
unzip -t a.zip
Archive: a.zip
warning [a.zip]: 4 extra bytes at beginning or within zipfile
(attempting to process anyway)
...
--
components: Library (Lib)
messages: 284172
nosy: Peter Ebden
priority: normal
severity: normal
status: open
title: Regression in zipfile writing in 2.7.13
versions: Python 2.7
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue29094>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
