subject:"Parallel python \+ \?\?"



Alban Browaeys  added the comment:

I managed to reproduce the bug.py crash with main branch up to commit 
12c0012cf97d21bc637056983ede0eaf4c0d9c33 .

--
Added file: 
https://bugs.python.org/file50700/bug.py_asyncio_cpustressed-crash.log

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)



Alban Browaeys  added the comment:

bisect of main for bug.py with each local python builds get me to commit 
b127e70a8a682fe869c22ce04c379bd85a00db67 "bpo-46070: Fix asyncio initialization 
guard (GH-30423)" as the one that fixed bug.py most of the time.

At times I can make bug.py segfault be it on python 3.9, 3.10 or main branch. 
It is pretty hard (I can have a batch of 200 runs without an issue) but seems 
easier to reproduce with a CPU stressed, then I can have two segfaults in a 
batch of 50 runs.

Bash:
for i in {1..50}; do ./python  ../python-crash-kodi/bug.py ; done
or sh:
for i in `seq 1 50`; do ./python  ../python-crash-kodi/bug.py ; done

with:
stress -c `nproc --all` at the same time.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)



Alban Browaeys  added the comment:

While bisecting main branch I did not only get segfaults but also exceptions, 
namely:

$ ./python  ../python-crash-kodi/sqlite3_crash.py 
Exception ignored deletion of interned string failed:
Traceback (most recent call last):
  File 
"/home/prahal/Projects/WIP/libreelec/cpython_bisect/Lib/sqlite3/dbapi2.py", 
line 81, in register_adapters_and_converters
register_adapter(datetime.datetime, adapt_datetime)
KeyError: 'isoformat'
Exception ignored deletion of interned string failed:
Traceback (most recent call last):
  File 
"/home/prahal/Projects/WIP/libreelec/cpython_bisect/Lib/sqlite3/dbapi2.py", 
line 83, in register_adapters_and_converters
register_converter("timestamp", convert_timestamp)
KeyError: 'timepart_full'
Exception ignored deletion of interned string failed:
Traceback (most recent call last):
  File 
"/home/prahal/Projects/WIP/libreelec/cpython_bisect/Lib/sqlite3/dbapi2.py", 
line 83, in register_adapters_and_converters
register_converter("timestamp", convert_timestamp)
KeyError: 'day'
Exception ignored deletion of interned string failed:
Traceback (most recent call last):
  File 
"/home/prahal/Projects/WIP/libreelec/cpython_bisect/Lib/sqlite3/dbapi2.py", 
line 83, in register_adapters_and_converters
register_converter("timestamp", convert_timestamp)
KeyError: 'month'
Exception ignored deletion of interned string failed:
Traceback (most recent call last):
  File 
"/home/prahal/Projects/WIP/libreelec/cpython_bisect/Lib/sqlite3/dbapi2.py", 
line 83, in register_adapters_and_converters
register_converter("timestamp", convert_timestamp)
KeyError: 'year'
Exception ignored deletion of interned string failed:
Traceback (most recent call last):
  File 
"/home/prahal/Projects/WIP/libreelec/cpython_bisect/Lib/sqlite3/dbapi2.py", 
line 83, in register_adapters_and_converters
register_converter("timestamp", convert_timestamp)
KeyError: 'timepart'
Exception ignored deletion of interned string failed:
Traceback (most recent call last):
  File 
"/home/prahal/Projects/WIP/libreelec/cpython_bisect/Lib/sqlite3/dbapi2.py", 
line 83, in register_adapters_and_converters
register_converter("timestamp", convert_timestamp)
KeyError: 'datepart'
Exception ignored deletion of interned string failed:
Traceback (most recent call last):
  File "", line 688, in _load_unlocked
KeyError: 'convert_timestamp'
Exception ignored deletion of interned string failed:
Traceback (most recent call last):
  File "", line 688, in _load_unlocked
KeyError: 'convert_date'
Exception ignored deletion of interned string failed:
Traceback (most recent call last):
  File "", line 688, in _load_unlocked
KeyError: 'adapt_datetime'
Exception ignored deletion of interned string failed:
Traceback (most recent call last):
  File "", line 688, in _load_unlocked
KeyError: 'adapt_date'


The 3.10 branch bisect pointed to one commit that fix the crash after 3.10.1 
which is 72c260cf0c71eb01eb13100b751e9d5007d00b70 [3.10] bpo-46006: Revert 
"bpo-40521: Per-interpreter interned strings (GH-20085)" (GH-30422) (GH-30425) 
which makes sense regarding main branch logs. It is commit 
35d6540c904ef07b8602ff014e520603f84b5886 in the main branch.

What remains to be seen is why "bpo-46070: _PyGC_Fini() untracks objects 
(GH-30577)" looks fine in the main branch (though it has no effect on the 
import crash) and not in 3.9 and 3.10 branch.
Mind in the main branch "bpo-46006: Revert "bpo-40521: Per-interpreter interned 
strings (GH-20085)" (GH-30422)" was already applied when "bpo-46070: 
_PyGC_Fini() untracks objects (GH-30577)" went in so it was also unrelated to 
the fix of the initial report.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)



Alban Browaeys  added the comment:

3.10 branch is fixed if I revert e6bb17fe29713368e1fd93d9ac9611017c4f570c 
bpo-46070: _PyGC_Fini() untracks objects (GH-30577). Be it if I revert it atop 
current head 9006b4471cc4d74d0cbf782d841a330b0f46a3d0 or if I test the commit 
before e6bb17fe29713368e1fd93d9ac9611017c4f570c was merged.

As this made no sense with regards to this bug report history that this fix 
broke the branch, I tried v3.10.1 which lacks this fix. Vanilla is broken too. 
Also applying the "_PyGC_Fini untracks objects" upon 3.10.1 does not fix the 
test case crash.

I am puzzled. Will try to bisect the commit that fixed the testcase in the 3.10 
branch before "_PyGC_Fini untracks objects" was merged and after 3.10.1.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)



Alban Browaeys  added the comment:

I did 3.9 branch bisect and commit 52937c26adc35350ca0402070160cf6dc838f359 
bpo-46070: _PyGC_Fini() untracks objects (GH-30577) (GH-30580) is the one that 
broke 3.9 . While with my main branch builds this commit is fine, it is not for 
the 3.9 branches.

Proceeding with 3.10 branch bisect of first bad commit and redoing main branch 
first good commit.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)



Alban Browaeys  added the comment:

By "It is fixed in main branch commit 12c0012cf97d21bc637056983ede0eaf4c0d9c33 
." I mean that this commit is good not that this commit fixes the issue.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)



Alban Browaeys  added the comment:

sqlite3_crash.py does not crashes on python 3.9 below and equal 3.9.9 and 
python main branch 12c0012cf97d21bc637056983ede0eaf4c0d9c33. I confirm it 
crashes on python 3.9.10, 3.9.11, 3.10 branch commit 
9006b4471cc4d74d0cbf782d841a330b0f46a3d0 .
It is fixed in main branch commit 12c0012cf97d21bc637056983ede0eaf4c0d9c33 .

Currently bisecting both 3.9.9 to 3.9.10 and 3.10 to 3.11 main branch for bad 
to good.

The patches in this bug report are already merged in the 3.10 branch which 
crash.

I cannot reproduce win_py399_crash_reproducer.py which I used as a basis for 
this test case.
The backtrace is the same as the ones from the crashes of the kodi addons (me 
Jellyfin Kodi addon), which is the initial report .
This looks like importing sqlite3 in threads plays badly.

I can reproduce on aarch64 (Odroid C2) LibreElec and builds of cpython on 
Debian stable x86_64 (the extensive testing of the broken interpreters is done 
on x86_64 Debian stable bullseye with a cpython clone and running from 
builddir).

--
nosy: +prahal
Added file: https://bugs.python.org/file50699/sqlite3_crash.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)

2022-01-20 Thread STINNER Victor



Change by STINNER Victor :


--
Removed message: https://bugs.python.org/msg411050

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)

2022-01-20 Thread STINNER Victor



STINNER Victor  added the comment:

Another measure using the command:

PYTHONHASHSEED=0 ./python -X showrefcount -c pass

I had to run the command 20 times to get a stable value, I don't know why.

main branch: [21981 refs, 5716 blocks]
PR: [21887 refs, 5667 blocks]

=> the PR removes 94 references and 49 memory blocks.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)

2022-01-20 Thread jokot3



Change by jokot3 :


--
nosy: +jokot3

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)



STINNER Victor  added the comment:

I created bpo-46368: "faulthandler: add the ability to dump all interpreters, 
not only the current interpreter".

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)



STINNER Victor  added the comment:

pyobject_ob_interp.patch: Quick & dirty patch that I wrote to add 
PyObject.ob_interp, store in which interpreter an object has been created.

--
Added file: https://bugs.python.org/file50560/pyobject_ob_interp.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)



STINNER Victor  added the comment:

Victor:
> (*) I made the GC state per-interpreter: commit 
> 7247407c35330f3f6292f1d40606b7ba6afd5700 (Nov 20, 2019)

Eric Snow:
> FYI, this was done by me in an earlier comment which we ended up
reverting.  Later you basically un.reverted that.

Well, I recall that your change had to be reverted 2 or 3 times because there 
were many crashes on FreeBSD, and no one understood why it crashed. The root 
cause was bugs related to the GIL and daemon threads. It took me a while (and 
multiple commits) to identify and fix all of them:
https://vstinner.github.io/gil-bugfixes-daemon-threads-python39.html

I decided to split your work into smaller changes to better debug these 
crashes. bpo-36854 contains a few changes, but these changes are based on work 
that I pushed earlier.

For example, there was a tricky bug related to clearing a Python thread state:
https://github.com/python/cpython/commit/9da7430675ceaeae5abeb9c9f7cd552b71b3a93a

Also, once the GC was made per interpreter, we started to discover more and 
more tricky reference leaks:
https://vstinner.github.io/subinterpreter-leaks.html

I spent a significant time to reorder code of Py_Finalize() and 
Py_EndInterpreter() to clear objects earlier or in a different order. Recently, 
I made sure that the free lists can no longer be used after they are cleared. 
It took some notes at:
https://pythondev.readthedocs.io/finalization.html

One of the hardest fix was the commit 9ad58acbe8b90b4d0f2d2e139e38bb5aa32b7fb6 
of bpo-19466. To make this change, first I had to fix a very old bug of 
PyThreadState_Clear() with commit 5804f878e779712e803be927ca8a6df389d82cdf 
(bpo-20526).

Well, it was a long journey and it's not done yet :-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)



STINNER Victor  added the comment:

It would be nice to add some tests.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)



STINNER Victor  added the comment:


New changeset 52937c26adc35350ca0402070160cf6dc838f359 by Victor Stinner in 
branch '3.9':
bpo-46070: _PyGC_Fini() untracks objects (GH-30577) (GH-30580)
https://github.com/python/cpython/commit/52937c26adc35350ca0402070160cf6dc838f359


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)

2022-01-13 Thread miss-islington



miss-islington  added the comment:


New changeset e6bb17fe29713368e1fd93d9ac9611017c4f570c by Miss Islington (bot) 
in branch '3.10':
bpo-46070: _PyGC_Fini() untracks objects (GH-30577)
https://github.com/python/cpython/commit/e6bb17fe29713368e1fd93d9ac9611017c4f570c


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)



STINNER Victor  added the comment:

I tested manually my fix GH-30580 using:

* (1) attached  win_py399_crash_reproducer.py
* (2) https://bugs.python.org/issue46070#msg410447 mthod

Without my fix, I can easily reproduce the crash with (1) and (2).

With my fix, I can no longer reproduce the crash with (1) or (2).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)



Change by STINNER Victor :


--
pull_requests: +28779
pull_request: https://github.com/python/cpython/pull/30580

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)

2022-01-13 Thread miss-islington



Change by miss-islington :


--
pull_requests: +28778
pull_request: https://github.com/python/cpython/pull/30579

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)

2022-01-13 Thread miss-islington



Change by miss-islington :


--
pull_requests: +28777
pull_request: https://github.com/python/cpython/pull/30578

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)



STINNER Victor  added the comment:


New changeset 1a4d1c1c9b08e75e88aeac90901920938f649832 by Victor Stinner in 
branch 'main':
bpo-46070: _PyGC_Fini() untracks objects (GH-30577)
https://github.com/python/cpython/commit/1a4d1c1c9b08e75e88aeac90901920938f649832


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)

2022-01-13 Thread Eric Snow



Eric Snow  added the comment:

> (*) I made the GC state per-interpreter: commit 
> 7247407c35330f3f6292f1d40606b7ba6afd5700 (Nov 20, 2019)

FYI, this was done by me in an earlier comment which we ended up
reverting.  Later you basically un.reverted that.

> The bug is that a C function object (_sre.compile) is created in an 
> interpreter, tracked by the GC list of this interpreter, and then it is 
> destroye and untracked in another interpreter.

FWIW, at one point I had a branch that supported sharing read-only
Py_Buffer data.  When the receiving interpreter was done with it I'd
call Py_AddPendingCall() to schedule the cleanup in the "owner"
interpreter.  However, this only worked because I kept track of the
owner.  Adding that pointer to every object wouldn't be feasible but I
suppose there are other things we could do that wouldn't be super
inefficient, like don't worry about it for the main interpreter, use a
hash table (Victor's idea), borrow some of the bits of the PyObject
head to store a flag or even an index into an array (if there are only
a few interpreters), or even make the allocator per-interpreter and
then extrapolate the interpreter based on the object's address.

Regardless, it is still much simpler to make all objects per-interpreter.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)



STINNER Victor  added the comment:

This issue has a complex history.

(*) I made the GC state per-interpreter: commit 
7247407c35330f3f6292f1d40606b7ba6afd5700 (Nov 20, 2019)

(*) This change triggered a _PyImport_FixupExtensionObject() bug in 
sub-interpreter, I fixed it with commit 
82c83bd907409c287a5bd0d0f4598f2c0538f34d (Nov 22, 2019)

(*) My _PyImport_FixupExtensionObject() fix introduced bpo-44050 regression, it 
was fixed by commit b9bb74871b27d9226df2dd3fce9d42bda8b43c2b (Oct 5, 2021)

(*) A race condition in the _asyncio extension has been identified and fixed by 
the commit b127e70a8a682fe869c22ce04c379bd85a00db67 (Jan 7, 2021)

(*) I identified a race condition introduced by the per-interpreter GC state 
cahnge: I proposed GH-30577 to fix it.


So far, the GC race condition has only been reproduced on Windows with Python 
3.9 and the _sre exception. On Python 3.10 and newer, it's harder to reproduce 
the crash using stdlib extensions since many of them have been ported to the 
multi-phase initializatioin API.

The GC race condition involves dangling pointers and depends on the memory 
allocator and when GC collections are triggered.

The bug is that a C function object (_sre.compile) is created in an 
interpreter, tracked by the GC list of this interpreter, and then it is 
destroye and untracked in another interpreter. The problem is that the object 
is untracked after the GC list has been destroyed and so "prev" and "next" 
objects of the PyGC_Head structure *can* become dangling pointers.

It's unclear to me what are the "prev" and "next" objects of the C function 
causing the crash (_sre.compile). At least, it seems like it's also used by 
more than one interpreter: it should *not* be done, see bpo-40533.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)



STINNER Victor  added the comment:

Oh. I managed to write a simple fix which doesn't require to revert the whole 
"per-interpreter GC" change: GH-30577.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)



Change by STINNER Victor :


--
pull_requests: +28776
pull_request: https://github.com/python/cpython/pull/30577

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)



STINNER Victor  added the comment:

When the crash occurs, the _sre.compile function is not destroyed in the 
interpreter which created the function.



The crash is related to _sre.compile method. This method is created in 
PyInit__sre() called by "import _sre".

On Windows, the _sre module is not imported at startup. So it's imported first 
in a subinterpreter.

In Python 3.9, the _sre module doesn't use the multiphase initialization API 
and PyModuleDef.m_size = -1. When the module is imported, 
_PyImport_FixupExtensionObject() copies the module dictionary into 
PyModuleDef.m_copy.

In Py_Finalize() and Py_EndInterpreter(), _PyImport_Cleanup() does two things:

* (1) set _sre.__dict__['compile'] to None -> kill the first reference to the 
function
* (2) call _PyInterpreterState_ClearModules() which does 
Py_CLEAR(def->m_base.m_copy), clear the cached copy of the _sre module dict -> 
kill the second reference

I modified Python to add an "ob_interp" member to PyObject to log in which 
interpreter an object is created. I also modified meth_dealloc() to log when 
_sre.compile function is deleted.

Extract of the reformatted output to see what's going on:
---
(...)

(1)
fixup: COPY _sre ModuleDef copy: def=7FFF19209810 interp=01EC1846F2A0

(2)
import: UPDATE(_sre ModuleDef copy): interp=01EC184AB790

(3)
_PyImport_Cleanup: interp=01EC1846F2A0
_PyInterpreterState_ClearModules: PY_CLEAR _sre ModuleDef m_copy: 
def=7FFF19209810 interp=01EC1846F2A0

(4)
_PyImport_Cleanup: interp=01EC184AB790
meth_dealloc(compile): m->ob_interp=01EC1846F2A0, 
interp=01EC184AB790

Windows fatal exception: access violation
(...)
---

Steps:

* (1)

  * interpreter #1 (01EC1846F2A0) creates the _sre.compile function
  * interpreter #1 (01EC1846F2A0) copies _sre module dict into 
PyModuleDef.m_copy
  * at this point, _sre.compile should have 2 references

* (2)

  * interpreter #2 (01EC184AB790) imports _sre: it creates a new module 
object and copies the function from PyModuleDef.m_copy
  * at this point, _sre.compile should have 3 references

* (3)

  * interpreter #1 exit: Py_EndInterpreter() calls _PyImport_Cleanup()
  * at this point, _sre.compile should have 1 reference

* (4)

  * interpreter #2 exit: Py_EndInterpreter() calls _PyImport_Cleanup()
  * the last reference to _sre.compile is deleted: 0 reference
  * meth_dealloc() is called

The first problem is that the function was created in the interpreter #1 but 
deleted in the interpreter #2.

The second problem is that the function is tracked by the GC and it is part of 
the GC list of the interpreter #1. When the interpreter #2 destroys the 
function, the GC list of interpreter #1 is already freed: PyGC_Head contains 
dangling pointers.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)



STINNER Victor  added the comment:

I wrote 3 scripts to reproduce the bug in a more reliable way. So I just have 
to type "bisect" and it runs the test 12 times.

(1) bisect.bat:
---
@"C:\vstinner\python\3.9\PCbuild\amd64\python_d.exe" bisect.py
---


(2) bisect.py:
---
import subprocess
import os
import sys

BISECT = False

def run(*args):
print("+ ", ' '.join(args))
env = dict(os.environ)
env['PYTHONFAULTHANDLER'] = '1'
proc = subprocess.run(args, env=env)
exitcode = proc.returncode
if exitcode:
print()
print(f"COMMAND FAILED: {exitcode}")
if BISECT:
print()
print("type: git bisect bad")
sys.exit(exitcode)

python = sys.executable
#script = "win_py399_crash_reproducer.py"
script = "bug.py"
nrun = 12
for i in range(1, nrun+1):
print(f"Run #{i}/{nrun}")
if i % 2:
run(python, script)
else:
run(python, "-X", "dev", script)

if BISECT:
print()
print("Not reproduced")
print()
run("git", "checkout", ".")
run("git", "bisect", "good")
---


(3) win_py399_crash_reproducer.py (import "_sre"):
---
# When this program is run on windows using python 3.9.9 it crashes about 50%
# of the time.

import _testcapi
import threading

code = """
import _sre
print("exit subinterpreter")
"""

def doIt():
_testcapi.run_in_subinterp(code)

tt=[]

for i in range(16):
t = threading.Thread(target=doIt)
t.start()
tt.append(t)

for t in tt:
t.join()
print("exit main")
---


Example:
---
vstinner@DESKTOP-DK7VBIL C:\vstinner\python\3.9>bisect
Run #1/12
+  C:\vstinner\python\3.9\PCbuild\amd64\python_d.exe bug.py
Run #2/12
+  C:\vstinner\python\3.9\PCbuild\amd64\python_d.exe -X dev bug.py
Run #3/12
+  C:\vstinner\python\3.9\PCbuild\amd64\python_d.exe bug.py
Windows fatal exception: access violation
(...)
Current thread 0x0420 (most recent call first):
  File "C:\vstinner\python\3.9\bug.py", line 13 in doIt
  File "C:\vstinner\python\3.9\lib\threading.py", line 910 in run
  File "C:\vstinner\python\3.9\lib\threading.py", line 973 in _bootstrap_inner
  File "C:\vstinner\python\3.9\lib\threading.py", line 930 in _bootstrap
(...)
COMMAND FAILED: 3221225477
---

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)



STINNER Victor  added the comment:

I modified PR 30565 (3.10) and PR 30566 (3.9) to fix the ABI. I added 
_PyGC_GetState() which always use PyInterpreterState.gc of the main interpreter.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)

2022-01-12 Thread Eric Snow



Eric Snow  added the comment:

> adding a new "gc" member in the _PyRuntimeState structure also causes the ABI 
> CI check to fail.

What if you move it to the end of the struct?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)



STINNER Victor  added the comment:

I prepared 3 pull requests to revert the commit 
7247407c35330f3f6292f1d40606b7ba6afd5700:

* PR 30564: main branch
* PR 30565: 3.10 branch
* PR 30566: 3.9 branch

The problem is that the "Check if the ABI has changed" CI job fails in 3.9 and 
3.10 branches.

I recently had the issue for a different revert in bpo-46006: I decided to keep 
the "removed" member, and mark it as "unused". See the commit 
72c260cf0c71eb01eb13100b751e9d5007d00b70 in the 3.10 branch:

struct _Py_unicode_state {
(...)

-PyObject *interned;

+// Unused member kept for ABI backward compatibility with Python 3.10.0:
+// see bpo-46006.
+PyObject *unused_interned;

(...)
}


I can keep the "gc" member in PyInterpreterState, but adding a new "gc" member 
in the _PyRuntimeState structure also causes the ABI CI check to fail.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)



Change by STINNER Victor :


--
pull_requests: +28769
pull_request: https://github.com/python/cpython/pull/30566

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)



Change by STINNER Victor :


--
pull_requests: +28768
pull_request: https://github.com/python/cpython/pull/30565

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)



Change by STINNER Victor :


--
pull_requests: +28767
pull_request: https://github.com/python/cpython/pull/30564

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)

2022-01-07 Thread STINNER Victor



STINNER Victor  added the comment:

This bug is hard to reproduce for different reasons:

* It occurs randomly: I need between 1 and 50 attempts to reproduce the bug 
using win_py399_crash_reproducer.py

* So far, the bug was only reproduced on Windows.

* I failed to reproduce the crash on Linux. I tried PYTHONMALLOC=malloc and 
PYTHONMALLOC=malloc_debug with and without 
LD_PRELOAD=/usr/lib64/libjemalloc.so.2 (jemalloc memory allocator).

* The _sre extension has been converted to multi-phase init in Python 3.10. 
"import _sre" is no longer enough to reproduce the crash on Python 3.10 and 
newer.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)

2022-01-07 Thread STINNER Victor



Change by STINNER Victor :


--
title: [subinterpreters] asyncio crash when importing _asyncio in 
subinterpreter (Python 3.8 regression) -> [subinterpreters] crash when 
importing _sre in subinterpreters in parallel (Python 3.9 regression)

___
Python tracker 
<https://bugs.python.org/issue46070>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Parallel python in the cloud

2014-05-24 Thread Charles Gagnon

We were happily using PiCloud for several long calculations and we very happy 
with with it. With their realtime cores, we could take really large 
calculations set and run through fairly quickly.

Now that PiCloud is going away, we ran a few tests on Mutlyvac but so far, we 
are struggling to accomplish the same thing we had on PiCloud.

I have several pieces of my puzzle but can't seem to be able to put it 
together. I've seen and tried StarCluster and also various parallel python 
options but all options seem challenging to put together.

The goal is to mimic PiCloud, ie. loop through a function:

def some_NP_func(x, y):
   ...
   return z

some_cloud.call(some_NP_func, a1, a2)

Which computes the function on the cloud. We use this often in for loops with 
arrays of arguments. The other scenario is:

some_cloud.map(some_NP_intense_func, [...], [...])

Which iterates through and returns results. We need to run a lot of this in 
batch from a scheduler so I always try to avoid interactive environment (how 
does iPython parallel work in batch?).

What is the preferred approach or method right now for heavy parallel 
computation like this?


Regards,
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Parallel python in the cloud

2014-05-24 Thread Robert Kern


On 2014-05-24 07:46, Charles Gagnon wrote:

We were happily using PiCloud for several long calculations and we very happy 
with with it. With their realtime cores, we could take really large 
calculations set and run through fairly quickly.

Now that PiCloud is going away, we ran a few tests on Mutlyvac but so far, we 
are struggling to accomplish the same thing we had on PiCloud.

I have several pieces of my puzzle but can't seem to be able to put it 
together. I've seen and tried StarCluster and also various parallel python options but 
all options seem challenging to put together.

The goal is to mimic PiCloud, ie. loop through a function:

def some_NP_func(x, y):
...
return z

some_cloud.call(some_NP_func, a1, a2)

Which computes the function on the cloud. We use this often in for loops with 
arrays of arguments. The other scenario is:

some_cloud.map(some_NP_intense_func, [...], [...])

Which iterates through and returns results. We need to run a lot of this in 
batch from a scheduler so I always try to avoid interactive environment (how 
does iPython parallel work in batch?).


IPython parallel works just fine in batch. As far as your client code (i.e. 
what you wrote above) is concerned, it's just another library. E.g.


https://github.com/ipython/ipython/blob/master/examples/Parallel%20Computing/nwmerge.py
https://github.com/ipython/ipython/blob/master/examples/Parallel%20Computing/itermapresult.py

etc.

--
Robert Kern

I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth.
  -- Umberto Eco

--
https://mail.python.org/mailman/listinfo/python-list

Re: Parallel Python x.y.A and x.y.B installations on a single Windows machine

2013-11-27 Thread Jurko Gospodnetić


  Hi.

On 25.11.2013. 14:42, Jurko Gospodnetić wrote:

   So far all tests seem to indicate that things work out fine if we
install to some dummy target folder, copy the target folder to some
version specific location  uninstall. That leaves us with a working
Python folder sans the start menu and registry items, both of which we
do not need for this. Everything I've played around with so far seems to
use the correct Python data depending on the interpreter executable
invoked, whether or not there is a regular Windows installation
somewhere on the same machine.

   We can use the script suggested by Ned Batchelder to temporarily
change the 'current installation' if needed for some external installer
package to correctly recognize where to install its content.

   I'm still playing around with this, and will let you know how it goes.


  Just wanted to let you know that the usage I described above seems to 
work in all the cases I tried out.


  I added some batch scripts for running a specific Python interpreter 
as a convenience and everything works 'naturally' in our development 
environment.


  Packages can be easily installed to a specific targeted environment 
using for example:

  py243 -m easy_install pip
  py332 -m pip install pytest
[not mentioning tweaks needed for specific ancient Python versions]

  Thank you all for all the suggestions.

  Best regards,
Jurko Gospodnetić


--
https://mail.python.org/mailman/listinfo/python-list

Parallel Python x.y.A and x.y.B installations on a single Windows machine


  Hi all.

  I was wondering what is the best way to install multiple Python 
installations on a single Windows machine.


  Regular Windows installer works great as long as all your 
installations have a separate major.minor version identifier. However, 
if you want to have let's say 2.4.3  2.4.4 installed at the same time 
it does not seem to work.


  I have not been able to find any prepackaged Python installation or 
really any solution to this. Most of the advice seems to boil down to 
'do not use such versions together, use only the latest'.


  We would like to run automated tests on one of our projects (packaged 
as a Python library) with different Python versions, and since our code 
contains workarounds for several problems with specific Python patch 
versions, we'd really like to be able to run the tests with those 
specific versions and with as little fuss as possible.


  Looking at what the Python installer does, the only problematic part 
for working around this manually seems to be the registry entries under 
'Software\Python\PythonCore\M.m' where 'M.n' is the major.minor version 
identifier. If Python interpreter expects to always find its entries 
there, then I guess there is no way to do what we need without building 
customized Python executables. Is there a way to force a specific Python 
interpreter to not read in this information, read it from an .ini file 
or something similar?


  Many thanks.

  Best regards,
Jurko Gospodnetić

--
https://mail.python.org/mailman/listinfo/python-list

Re: Parallel Python x.y.A and x.y.B installations on a single Windows machine

2013-11-25 Thread Ned Batchelder

On Monday, November 25, 2013 7:32:30 AM UTC-5, Jurko Gospodnetić wrote:
Hi all.

I was wondering what is the best way to install multiple Python
installations on a single Windows machine.

Regular Windows installer works great as long as all your
installations have a separate major.minor version identifier. However,
if you want to have let's say 2.4.3 2.4.4 installed at the same time
it does not seem to work.

I have not been able to find any prepackaged Python installation or
really any solution to this. Most of the advice seems to boil down to
'do not use such versions together, use only the latest'.

We would like to run automated tests on one of our projects (packaged
as a Python library) with different Python versions, and since our code
contains workarounds for several problems with specific Python patch
versions, we'd really like to be able to run the tests with those
specific versions and with as little fuss as possible.

Looking at what the Python installer does, the only problematic part
for working around this manually seems to be the registry entries under
'Software\Python\PythonCore\M.m' where 'M.n' is the major.minor version
identifier. If Python interpreter expects to always find its entries
there, then I guess there is no way to do what we need without building
customized Python executables. Is there a way to force a specific Python
interpreter to not read in this information, read it from an .ini file
or something similar?

Many thanks.

Best regards,
Jurko Gospodnetiďż˝

IIRC, Python itself doesn't read those registry entries, except when installing
pre-compiled .msi or .exe kits. Once you have Python installed, you can move
the directory someplace else, then install another version of Python.

If you need to use many different Pythons of the same version, this script
helps manage the registry:
http://nedbatchelder.com/blog/201007/installing_python_packages_from_windows_installers_into.html

--Ned.
--
https://mail.python.org/mailman/listinfo/python-list

Re: Parallel Python x.y.A and x.y.B installations on a single Windows machine

2013-11-25 Thread Chris Angelico

On Mon, Nov 25, 2013 at 11:32 PM, Jurko Gospodnetić
jurko.gospodne...@pke.hr wrote:
 Most of the advice seems to boil down to 'do not use such versions together,
 use only the latest'.

   We would like to run automated tests on one of our projects (packaged as a
 Python library) with different Python versions, and since our code contains
 workarounds for several problems with specific Python patch versions, we'd
 really like to be able to run the tests with those specific versions and
 with as little fuss as possible.

What this says to me is that you're doing something very unusual here
- most people won't be doing that. So maybe you need an unusual
solution.

Is it possible to set up virtualization to help you out? Create a
virtual machine in something like VirtualBox, then clone it for every
Python patch you want to support (you could have one VM that handles
all the .0 releases and another that handles all the .1 releases, or
you could have a separate VM for every Python you want to test). You
could then have a centralized master that each VM registers itself
with, and it feeds out jobs to them. Assuming your tests can be fully
automated, this could work out fairly efficiently - each VM has a
script that establishes a socket connection to the master, the master
hands out a job, the VMs run the test suite, the master collects up a
series of Pass/Fail reports. You could run everything on a single
physical computer, even.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Parallel Python x.y.A and x.y.B installations on a single Windows machine

2013-11-25 Thread Albert-Jan Roskam

On Mon, 11/25/13, Jurko Gospodnetić jurko.gospodne...@pke.hr wrote:

 Subject: Parallel Python x.y.A and x.y.B installations on a single Windows 
machine
 To: python-list@python.org
 Date: Monday, November 25, 2013, 1:32 PM

   Hi all.

   I was wondering what is the best way to install
 multiple Python installations on a single Windows machine.

   Regular Windows installer works great as long as all
 your installations have a separate major.minor version
 identifier. However, if you want to have let's say 2.4.3
  2.4.4 installed at the same time it does not seem to
 work.

   I have not been able to find any prepackaged Python
 installation or really any solution to this. Most of the
 advice seems to boil down to 'do not use such versions
 together, use only the latest'.

   We would like to run automated tests on one of our
 projects (packaged as a Python library) with different
 Python versions, and since our code contains workarounds for
 several problems with specific Python patch versions, we'd
 really like to be able to run the tests with those specific
 versions and with as little fuss as possible.

   Looking at what the Python installer does, the only
 problematic part for working around this manually seems to
 be the registry entries under
 'Software\Python\PythonCore\M.m' where 'M.n' is the
 major.minor version identifier. If Python interpreter
 expects to always find its entries there, then I guess there
 is no way to do what we need without building customized
 Python executables. Is there a way to force a specific
 Python interpreter to not read in this information, read it
 from an .ini file or something similar?

HI Jurko,

Check out the following packages: virtualenv, virtualenvwrapper, tox
virtualenv + wrapper make it very easy to switch from one python version to 
another. Stricly speaking you don't need virtualenvwrapper, but it makes 
working with virtualenv a whole lot easier.Tox also uses virtualenv. You can 
configure it to sdist your package under different python versions. Also, you 
can make it run nosetests for each python version and/or implementation (pypy 
and jython are supported)

Albert-Jan 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Parallel Python x.y.A and x.y.B installations on a single Windows machine


  Hi.

On 25.11.2013. 14:04, Chris Angelico wrote:

Is it possible to set up virtualization to help you out? Create a
virtual machine in something like VirtualBox, then clone it for every
Python patch you want to support (you could have one VM that handles
all the .0 releases and another that handles all the .1 releases, or
you could have a separate VM for every Python you want to test).
...


  Thank you for the suggestion.

  Yup, we could do that, but at first glance it really smells like an 
overkill. Not to mention the potential licensing issues with Windows and 
an unlimited number of Windows installations. :-)


  So far all tests seem to indicate that things work out fine if we 
install to some dummy target folder, copy the target folder to some 
version specific location  uninstall. That leaves us with a working 
Python folder sans the start menu and registry items, both of which we 
do not need for this. Everything I've played around with so far seems to 
use the correct Python data depending on the interpreter executable 
invoked, whether or not there is a regular Windows installation 
somewhere on the same machine.


  We can use the script suggested by Ned Batchelder to temporarily 
change the 'current installation' if needed for some external installer 
package to correctly recognize where to install its content.


  I'm still playing around with this, and will let you know how it goes.

  Thank you again for replying!

  Best regards,
Jurko Gospodnetić


--
https://mail.python.org/mailman/listinfo/python-list

Re: Parallel Python x.y.A and x.y.B installations on a single Windows machine

2013-11-25 Thread Chris Angelico

On Tue, Nov 26, 2013 at 12:42 AM, Jurko Gospodnetić
jurko.gospodne...@pke.hr wrote:
   Yup, we could do that, but at first glance it really smells like an
 overkill. Not to mention the potential licensing issues with Windows and an
 unlimited number of Windows installations. :-)

Ah, heh... didn't think of that. When I spin up arbitrary numbers of
VMs, they're always Linux, so licensing doesn't come into it :)

   So far all tests seem to indicate that things work out fine if we install
 to some dummy target folder, copy the target folder to some version specific
 location  uninstall. That leaves us with a working Python folder sans the
 start menu and registry items, both of which we do not need for this.
 Everything I've played around with so far seems to use the correct Python
 data depending on the interpreter executable invoked, whether or not there
 is a regular Windows installation somewhere on the same machine.

Okay! That sounds good. Underkill is better than overkill if you can
get away with it!

Good luck. You'll need it, if you're trying to support Python 2.4 and
all newer versions AND manage issues across patch releases...

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Parallel Python x.y.A and x.y.B installations on a single Windows machine


  Hi.

On 25.11.2013. 13:46, Ned Batchelder wrote:

IIRC, Python itself doesn't read those registry entries, except when installing 
pre-compiled .msi or .exe kits.  Once you have Python installed, you can move 
the directory someplace else, then install another version of Python.

If you need to use many different Pythons of the same version, this script 
helps manage the registry: 
http://nedbatchelder.com/blog/201007/installing_python_packages_from_windows_installers_into.html


  Thank you for the information!

  As I mentioned in another reply, so far I think we can use this 
script to temporarily change the 'current installation' if needed for 
some external installer package to correctly recognize where to install 
its content.


  bike-sheddingIf we do use it, I'll most likely modify it to first 
make a backup copy of the original registry key and use that later on to 
restore the original registry state instead of reconstructing its 
content to what the script assumes it should be./bike-shedding


  Best regards,
Jurko Gospodnetić


--
https://mail.python.org/mailman/listinfo/python-list

Re: Parallel Python x.y.A and x.y.B installations on a single Windows machine


  Hi.

On 25.11.2013. 14:20, Albert-Jan Roskam wrote:

Check out the following packages: virtualenv, virtualenvwrapper, tox
virtualenv + wrapper make it very easy to switch from one python
version to another. Stricly speaking you don't need
virtualenvwrapper, but it makes working with virtualenv a whole lot
easier.Tox also uses virtualenv. You can configure it to sdist your
package under different python versions. Also, you can make it run
nosetests for each python version and/or implementation (pypy and
jython are supported)


  I'll look into using virtualenv and possibly tox once I get into 
issues with mismatched installed Python package versions, but for now 
I'm dealing with installing different Python interpreter versions and, 
unless I'm overlooking something here, virtualenv does not help with 
that. :-(


  Thanks for the suggestion though, I'm definitely going to read up on 
those packages soon. :-)


  Best regards,
Jurko Gospodnetić


--
https://mail.python.org/mailman/listinfo/python-list

Re: Parallel Python x.y.A and x.y.B installations on a single Windows machine

2013-11-25 Thread Albert-Jan Roskam

On Mon, 11/25/13, Jurko Gospodnetić jurko.gospodne...@pke.hr wrote:

 Subject: Re: Parallel Python x.y.A and x.y.B installations on a single Windows 
machine
 To: python-list@python.org
 Date: Monday, November 25, 2013, 2:57 PM

   Hi.

 On 25.11.2013. 14:20, Albert-Jan Roskam wrote:
  Check out the following packages: virtualenv,
 virtualenvwrapper, tox
  virtualenv + wrapper make it very easy to switch from
 one python
  version to another. Stricly speaking you don't need
  virtualenvwrapper, but it makes working with virtualenv
 a whole lot
  easier.Tox also uses virtualenv. You can configure it
 to sdist your
  package under different python versions. Also, you can
 make it run
  nosetests for each python version and/or implementation
 (pypy and
  jython are supported)

   I'll look into using virtualenv and possibly tox once
 I get into issues with mismatched installed Python package
 versions, but for now I'm dealing with installing different
 Python interpreter versions and, unless I'm overlooking
 something here, virtualenv does not help with that. :-(

  Are you sure? 
http://stackoverflow.com/questions/1534210/use-different-python-version-with-virtualenv

Below is a little terminal session.  I often switch between python 3.3 and 
python 2.7. My virtualenv for python 3.3 is called python33. workon is a 
virtualenv wrapper command. And check out the envlist in tox.ini on 
http://tox.readthedocs.org/en/latest/example/basic.html

antonia@antonia-HP-2133 ~ $ workon python3.3
ERROR: Environment 'python3.3' does not exist. Create it with 'mkvirtualenv 
python3.3'.
antonia@antonia-HP-2133 ~ $ workon python33
(python33)antonia@antonia-HP-2133 ~ $ python
Python 3.3.2 (default, Sep  1 2013, 22:59:57) 
[GCC 4.7.2] on linux
Type help, copyright, credits or license for more information.
 quit()
(python33)antonia@antonia-HP-2133 ~ $ deactivate
antonia@antonia-HP-2133 ~ $ python
Python 2.7.3 (default, Sep 26 2013, 16:38:10) 
[GCC 4.7.2] on linux2
Type help, copyright, credits or license for more information.
 quit()

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Parallel Python x.y.A and x.y.B installations on a single Windows machine

2013-11-25 Thread Chris Angelico

On Tue, Nov 26, 2013 at 1:15 AM, Albert-Jan Roskam fo...@yahoo.com wrote:
 Below is a little terminal session.  I often switch between python 3.3 and 
 python 2.7. My virtualenv for python 3.3 is called python33. workon is a 
 virtualenv wrapper command. And check out the envlist in tox.ini on 
 http://tox.readthedocs.org/en/latest/example/basic.html

That's two different minor versions, though. Can you have 3.3.1 and
3.3.2 installed, by that method?

Incidentally, if this were on Linux, I would just build the different
versions in different directories, and then run them without
installing. But the OP seems to have a solution that works, and I
think it'll be the simplest.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Parallel Python x.y.A and x.y.B installations on a single Windows machine


  Hi.

On 25.11.2013. 15:15, Albert-Jan Roskam wrote:

   Are you sure? 
http://stackoverflow.com/questions/1534210/use-different-python-version-with-virtualenv


  Yup, I'm pretty sure by now (based on reading the docs, not trying it 
out though).


  Virtualenv allows you to set up different environments, each of them 
having a separate Python folder structure and each possibly connected to 
a separate Python interpreter executable. However, it does not solve the 
problem of getting those separate Python interpreter executables 
installed in the first place, which is the problem I was attacking. :-)


  Still playing around with my multiple installations setup here. Will 
let you know how it goes...


  So far, one important thing I noticed is that you need to run all 
your installations 'for the current user only', or otherwise it moves at 
least one DLL file (python24.dll) into a Windows system folder and then 
the next installation deletes it from there, and overwrites it with its 
own. :-( But I can live with that... :-)


  Best regards,
Jurko Gospodnetić


--
https://mail.python.org/mailman/listinfo/python-list

Re: Parallel Python x.y.A and x.y.B installations on a single Windows machine

2013-11-25 Thread Terry Reedy


On 11/25/2013 8:42 AM, Jurko Gospodnetić wrote:


   So far all tests seem to indicate that things work out fine if we
install to some dummy target folder, copy the target folder to some
version specific location  uninstall.


If the dummy folder had 3.3.0, you should not need to uninstall to 
install 3.3.1 on top of it. But it is easy and probably safest.



That leaves us with a working
Python folder sans the start menu and registry items, both of which we
do not need for this. Everything I've played around with so far seems to
use the correct Python data depending on the interpreter executable
invoked, whether or not there is a regular Windows installation
somewhere on the same machine.


Just a reminder: you can run one file or set of files with multiple 
Pythons by putting 'project.pth' containing the same 'path-to-project' 
in the Lib/site-packages of each Python directory. I do this to test one 
file with 2.7 and 3.3 (and just added 3.4) without copying the file.


--
Terry Jan Reedy


--
https://mail.python.org/mailman/listinfo/python-list

Re: Parallel Python x.y.A and x.y.B installations on a single Windows machine


  Hi.

On 25.11.2013. 17:38, Terry Reedy wrote:

   So far all tests seem to indicate that things work out fine if we
install to some dummy target folder, copy the target folder to some
version specific location  uninstall.


If the dummy folder had 3.3.0, you should not need to uninstall to
install 3.3.1 on top of it. But it is easy and probably safest.


  Without the uninstall step you get stuck with invalid registry and 
start menu items refering to an invalid path until you install another 
matching major.minor.X version.




Just a reminder: you can run one file or set of files with multiple
Pythons by putting 'project.pth' containing the same 'path-to-project'
in the Lib/site-packages of each Python directory. I do this to test one
file with 2.7 and 3.3 (and just added 3.4) without copying the file.


  Thanks for the tip. That might come in useful. At the moment I just 
run the pytest framework using different python interpreters, without 
having to install the package at all (possibly first running 'setup.py 
build' to get the sources converted to Python 3 format).


  Best regards,
Jurko Gospodnetić


--
https://mail.python.org/mailman/listinfo/python-list

Parallel python + ??

2008-06-11 Thread Thor

Hi,

I am running a program using Parallel Python and I wonder if there is a
way/module to know in which CPU/core the process is running in. Is that
possible?

Ángel
--
http://mail.python.org/mailman/listinfo/python-list

Re: Parallel python + ??

2008-06-11 Thread Gerhard Häring


Thor wrote:

Hi,

I am running a program using Parallel Python and I wonder if there is a
way/module to know in which CPU/core the process is running in. Is that
possible?


This is of course OS-specific. On Linux, you can parse the proc filesystem:

 open(/proc/%i/stat % os.getpid()).read().split()[39]

You can use the taskset utility to query or set CPU affinity on Linux.

-- Gerhard

--
http://mail.python.org/mailman/listinfo/python-list

Re: Parallel python + ??

2008-06-11 Thread Thor

Gerhard Häring wrote:

 This is of course OS-specific. On Linux, you can parse the proc
 filesystem:
 
   open(/proc/%i/stat % os.getpid()).read().split()[39]
 
 You can use the taskset utility to query or set CPU affinity on Linux.
 
It is going to be in Linux (mainly) I was thinking about something like
this:

import Module

def process(self):
  print I am running on processor, Module.cpu,core, Module.core
  


Checking the raskset right now...:) Thanks.
--
http://mail.python.org/mailman/listinfo/python-list

RE: Parallel Python environments..

2007-11-07 Thread Thorsten Kampe

* bruce (Tue, 6 Nov 2007 13:43:10 -0800)
 if i have python 2.4.3 installed, it gets placed in the python2.4 dir.. if i
 don't do anything different, and install python 2.4.2, it too will get
 placed in the python2.4 tree... which is not what i want.
 
 i'm running rhel4/5...

So you're using rpm as a packet manager. I suggest you RTFM to see if 
there are options for slots or dual installations.
 
 so.. i still need to know what to do/change in order to be able to run
 multiple versions of python, and to switch back/forth between the versions.

Unpack the Python rpm to ~/bin or compile Python yourself. And be more 
specific about what you mean with switching back/forth. 

On the other hand - as Gabriel pointed out: there is almost a 100% 
certainty that the problem you want to solve by having Python 2.4.2 
*and* 2.4.3 simultaneously exists only in your head or cannot be 
solved this way.

Thorsten
-- 
http://mail.python.org/mailman/listinfo/python-list

Parallel Python environments..

Hi..

If I wanted to be able to build/test/use parallel python versions, what
would I need to do/set (paths/libs/etc...) and where would I need to place
the 2nd python version, so as not to screw up my initial python dev env.

I'd like to be able to switch back/forth between the different versions if
possible. I know it should be, but I haven't been able to find what I'm
looking for via the 'net...

Any sites/pointers describing the process would be helpuful. In particular,
any changfes to the bashrc/profile/etc... files to allow me to accomplish
this would be helpful.

thanks

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Parallel Python environments..

2007-11-06 Thread Thorsten Kampe

* bruce (Tue, 6 Nov 2007 07:13:43 -0800)
 If I wanted to be able to build/test/use parallel python versions, what
 would I need to do/set (paths/libs/etc...)

nothing

 and where would I need to place the 2nd python version, so as not to
 screw up my initial python dev env.

Anywhere you like (probably ~/bin would be best)
 
 Any sites/pointers describing the process would be helpuful. In particular,
 any changfes to the bashrc/profile/etc... files to allow me to accomplish
 this would be helpful.

Nothing like that. Just change the shebang.

Thorsten
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Parallel Python environments..

2007-11-06 Thread [EMAIL PROTECTED]

In Gentoo Linux you can select between installed python version using
python-config script.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Parallel Python environments..

2007-11-06 Thread Diez B. Roggisch

bruce wrote:

 Hi..
 
 If I wanted to be able to build/test/use parallel python versions, what
 would I need to do/set (paths/libs/etc...) and where would I need to place
 the 2nd python version, so as not to screw up my initial python dev env.
 
 I'd like to be able to switch back/forth between the different versions if
 possible. I know it should be, but I haven't been able to find what I'm
 looking for via the 'net...
 
 Any sites/pointers describing the process would be helpuful. In
 particular, any changfes to the bashrc/profile/etc... files to allow me to
 accomplish this would be helpful.

Installation of several python versions is easy on at least Windows 
unixish platforms. For the latter, ususally your package-management offers
several versions. If your's doesn't or you use windows, just install as
required by python itself.

Only what gets chosen as default python version in case of e.g. *.py-files
clicking in windows explorer depends on the python version installed as
last and must be changed by the OS's means for it if other behavior is
desired.

Diez
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Parallel Python environments..

2007-11-06 Thread Gabriel Genellina

En Tue, 06 Nov 2007 18:43:10 -0300, bruce [EMAIL PROTECTED]  
escribió:

 if i have python 2.4.3 installed, it gets placed in the python2.4 dir..  
 if i
 don't do anything different, and install python 2.4.2, it too will get
 placed in the python2.4 tree... which is not what i want.

Any reason you want to keep 2.4.2 *and* 2.4.3 separate? The latter is only  
a bugfix over the 2.4 version - anything working on 2.4.2 should work on  
2.4.3. And 2.4.4, the latest bugfix on that series. Binaries, shared  
libraries, extensions, etc. targetted to 2.4x should work with 2.4.4
You may want to have separate directories for 2.4 and 2.5, yes; binaries,  
shared libraries and extensions do NOT work across versions changing the  
SECOND digit. But changes on the THIRD digit should not have compatibility  
problems.

-- 
Gabriel Genellina

-- 
http://mail.python.org/mailman/listinfo/python-list

RE: Parallel Python environments..

thorsten...

if i have python 2.4.3 installed, it gets placed in the python2.4 dir.. if i
don't do anything different, and install python 2.4.2, it too will get
placed in the python2.4 tree... which is not what i want.

i'm running rhel4/5...

so.. i still need to know what to do/change in order to be able to run
multiple versions of python, and to switch back/forth between the versions.

thanks


-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] Behalf
Of Thorsten Kampe
Sent: Tuesday, November 06, 2007 8:19 AM
To: python-list@python.org
Subject: Re: Parallel Python environments..


* bruce (Tue, 6 Nov 2007 07:13:43 -0800)
 If I wanted to be able to build/test/use parallel python versions, what
 would I need to do/set (paths/libs/etc...)

nothing

 and where would I need to place the 2nd python version, so as not to
 screw up my initial python dev env.

Anywhere you like (probably ~/bin would be best)

 Any sites/pointers describing the process would be helpuful. In
particular,
 any changfes to the bashrc/profile/etc... files to allow me to accomplish
 this would be helpful.

Nothing like that. Just change the shebang.

Thorsten
--
http://mail.python.org/mailman/listinfo/python-list

-- 
http://mail.python.org/mailman/listinfo/python-list

RE: Parallel Python environments..

i'm running rhel...

so there isn't a python-config script as far as i know..


-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] Behalf
Of [EMAIL PROTECTED]
Sent: Tuesday, November 06, 2007 8:26 AM
To: python-list@python.org
Subject: Re: Parallel Python environments..


In Gentoo Linux you can select between installed python version using
python-config script.

-- 
http://mail.python.org/mailman/listinfo/python-list
-- 
http://mail.python.org/mailman/listinfo/python-list

RE: Parallel Python environments..