Re: [Python-Dev] Weak Dictionary Iteration Behavior in Python 3
Hi, Josiah Carlson gmail.com> writes: > i = list(d.keys()) Obviously that doesn't solve the problem. list() consumes the generator one after another, objects can still die when the list is created. Imagine the following example which uses threads:: from time import sleep from weakref import WeakKeyDictionary from threading import Thread class Foo(object): pass d = WeakKeyDictionary() l = [] for x in range(10): obj = Foo() d[obj] = None l.append(obj) del obj def iterater(): for item in list(d.keys()): pass Thread(target=iterater).start() while True: del l[0] Regards, Armin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Weak Dictionary Iteration Behavior in Python 3
Hi, Adam Olsen gmail.com> writes: > IMO, this is a deeper problem than suggested. As far as I know, > python does not (and should not) make promises as to when it'll > collect object. We should expect weakrefs to be cleared at random > points, and code defensively. It doesn't promise when objects are collected and that's not the problem here. The main problem is that the old solution for weakrefs relayed on the fact that .keys() was considered atomic. I don't say it has to become again, but the weak dictionaries have to somehow counter that problem. They could for example only remove items from the internal dict if no dict view to that dict is alive. Speaking of atom keys() / values() / items() operations: I guess we will see more of those problems in threaded situations when people start to convert code over to Python. I've seen quite a few situations where code relays on keys() holding the interpreter lock. Regards, Armin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Weak Dictionary Iteration Behavior in Python 3
Armin Ronacher wrote: > Speaking of atom keys() / values() / items() operations: I guess we will > see more of those problems in threaded situations when people start to > convert code over to Python. I've seen quite a few situations where code > relays on keys() holding the interpreter lock. list(iter) doesn't re-enter the eval loop if the iterator is written in C and hence won't let go of the GIL. As I believe the dict views in 3.0 are all C classes, I'd like to see a failing test case along these lines that doesn't involve a dict subclass and code written in Python. Note that I'm not saying that I know for certain that there aren't any problems with list(d.keys()) that aren't seen with the direct list conversion in 2.x - I'm just saying it may not be as easy to provoke such problems as it initially appears due to the way the list constructor interacts with the supplied iterable, so it is going to take an actual failing test case to convince me. Cheers, Nick. -- Nick Coghlan | [EMAIL PROTECTED] | Brisbane, Australia --- http://www.boredomandlaziness.org ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] Python Regression Test Failures basics (1)
On Sun, Sep 14, 2008 at 4:07 AM, Nick Coghlan <[EMAIL PROTECTED]> wrote: > Neal Norwitz wrote: >> test_epoll skipped -- kernel doesn't support epoll() > ... >> test_ioctl skipped -- Unable to open /dev/tty > ... >> test_multiprocessing skipped -- OSError raises on RLock creation, see > issue 3111! > ... >> test test_normalization failed -- Traceback (most recent call last): >> File "/tmp/python-test/local/lib/python2.6/test/test_normalization.py", >> line 90, in test_main >> self.failUnless(X == NFC(X) == NFD(X) == NFKC(X) == NFKD(X), c) >> AssertionError: 6918 > ... >> 326 tests OK. >> 1 test failed: >> test_normalization > ... >> 3 skips unexpected on linux2: >>test_epoll test_multiprocessing test_ioctl > > Neal, > > What environment are you using to run the debug-mode regression tests? > The above four tests run without any problems for me, but I'm just > running them in a normal Kubuntu desktop shell. Something in Neal's build which has made a difference before is that the tests are run after a "make install". > > Cheers, > Nick. > > P.S. I'm in the process of creating a --with-pydebug build to see if > that makes any difference. > > -- > Nick Coghlan | [EMAIL PROTECTED] | Brisbane, Australia > --- >http://www.boredomandlaziness.org > ___ > Python-Dev mailing list > [email protected] > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/musiccomposition%40gmail.com > -- Cheers, Benjamin Peterson "There's no place like 127.0.0.1." ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Weak Dictionary Iteration Behavior in Python 3
Nick Coghlan schrieb: > Armin Ronacher wrote: >> Speaking of atom keys() / values() / items() operations: I guess we will >> see more of those problems in threaded situations when people start to >> convert code over to Python. I've seen quite a few situations where code >> relays on keys() holding the interpreter lock. > > list(iter) doesn't re-enter the eval loop if the iterator is written in > C and hence won't let go of the GIL. As I believe the dict views in 3.0 > are all C classes, I'd like to see a failing test case along these lines > that doesn't involve a dict subclass and code written in Python. WeakKeyDictionary.keys() iterates over its .data dictionary's keys() instead of list(data.keys()). Therefore, the user of WeakKeyDictionary has no chance to do list(keys()) without running Python code. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Weak Dictionary Iteration Behavior in Python 3
Georg Brandl wrote: > Nick Coghlan schrieb: >> Armin Ronacher wrote: >>> Speaking of atom keys() / values() / items() operations: I guess we will >>> see more of those problems in threaded situations when people start to >>> convert code over to Python. I've seen quite a few situations where code >>> relays on keys() holding the interpreter lock. >> list(iter) doesn't re-enter the eval loop if the iterator is written in >> C and hence won't let go of the GIL. As I believe the dict views in 3.0 >> are all C classes, I'd like to see a failing test case along these lines >> that doesn't involve a dict subclass and code written in Python. > > WeakKeyDictionary.keys() iterates over its .data dictionary's keys() instead > of list(data.keys()). Therefore, the user of WeakKeyDictionary has no chance > to do list(keys()) without running Python code. Ouch, there's the dict subclass with Python code that I was asking for before I got concerned (I forgot that the weakref module is only partially implemented in C). I agree this is a problem, but I'm struggling to think of possible solutions that aren't either horrendously complicated or completely contrary to the idea of view-based dicts. Cheers, Nick. -- Nick Coghlan | [EMAIL PROTECTED] | Brisbane, Australia --- http://www.boredomandlaziness.org ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] Python Regression Test Failures basics (1)
On Sun, Sep 14, 2008 at 5:24 AM, Benjamin Peterson <[EMAIL PROTECTED]> wrote: > On Sun, Sep 14, 2008 at 4:07 AM, Nick Coghlan <[EMAIL PROTECTED]> wrote: >> Neal Norwitz wrote: >>> test_epoll skipped -- kernel doesn't support epoll() >> ... >>> test_ioctl skipped -- Unable to open /dev/tty >> ... >>> test_multiprocessing skipped -- OSError raises on RLock creation, see >> issue 3111! >> ... >>> test test_normalization failed -- Traceback (most recent call last): >>> File "/tmp/python-test/local/lib/python2.6/test/test_normalization.py", >>> line 90, in test_main >>> self.failUnless(X == NFC(X) == NFD(X) == NFKC(X) == NFKD(X), c) >>> AssertionError: 6918 >> ... >>> 326 tests OK. >>> 1 test failed: >>> test_normalization >> ... >>> 3 skips unexpected on linux2: >>>test_epoll test_multiprocessing test_ioctl >> >> Neal, >> >> What environment are you using to run the debug-mode regression tests? >> The above four tests run without any problems for me, but I'm just >> running them in a normal Kubuntu desktop shell. > > Something in Neal's build which has made a difference before is that > the tests are run after a "make install". Benjamin is correct. See Misc/build.sh for the script that generates this. See http://docs.python.org/dev/results/ for details about the most recent run. n ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] ',' precedence in documentation
C. Titus Brown wrote:
over on the pygr list, we were recently discussing a mildly confusing
edit I made:
assert 'seq1' in self.db, self.db.keys()
This was interpreted by someone as being
assert 'seq1' in (self.db, self.db.keys())
which is very different from the actual meaning,
assert ('seq1' in self.db), self.db.keys()
where 'self.db.keys()' serves as the message to be printed out when the
assertion fails. Apart from questions of why I was apparently out to
confuse people with this edit, the question of ',' precedence came up.
Looking at
http://docs.python.org/ref/summary.html
there's no mention of ',' binding and tuple creation vs the other
operators.
A bare "," is part of the "expression list" syntax; it's not an operator:
http://docs.python.org/ref/exprlists.html
You have to look at the statement descriptions to find out whether a
statement wants an expression or an expression list (e.g. "return" takes
an expression list, while "assert" takes two expressions, not two
expression lists).
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] ',' precedence in documentation
On Sun, Sep 14, 2008 at 12:47 PM, Fredrik Lundh <[EMAIL PROTECTED]> wrote:
> C. Titus Brown wrote:
>
>> over on the pygr list, we were recently discussing a mildly confusing
>> edit I made:
>>
>> assert 'seq1' in self.db, self.db.keys()
>> This was interpreted by someone as being
>>
>> assert 'seq1' in (self.db, self.db.keys())
>>
>> which is very different from the actual meaning,
>>
>> assert ('seq1' in self.db), self.db.keys()
>>
>> where 'self.db.keys()' serves as the message to be printed out when the
>> assertion fails. Apart from questions of why I was apparently out to
>> confuse people with this edit, the question of ',' precedence came up.
>> Looking at
>>
>>http://docs.python.org/ref/summary.html
>>
>> there's no mention of ',' binding and tuple creation vs the other
>> operators.
>
> A bare "," is part of the "expression list" syntax; it's not an operator:
>
>http://docs.python.org/ref/exprlists.html
>
> You have to look at the statement descriptions to find out whether a
> statement wants an expression or an expression list (e.g. "return" takes an
> expression list, while "assert" takes two expressions, not two expression
> lists).
Note that in any case, the 'in' operator binds more tightly than the
comma: e.g. f(x in y, z) means f((x in y), z). The only seeming
exception to this rule is the 'for' statement (and list comprehensions
and generator expressions) where 'in' is not an expression operator
but part of the statement syntax.
So if someone thought
assert 'seq1' in self.db, self.db.keys()
meant
assert 'seq1' in (self.db, self.db.keys())
they'd be in for a nasty surprise trying the same trick in an 'if'
statement, like this:
if 'seq1' in self.db, self.db.keys(): ...
Fortunately that raises a syntax error. The really bad surprises come
from things like
assert x in A, x in B
which I've seen assumed to mean
assert (x in A) and (x in B)
in claimed analogy to the use of the comma in the print statement.
I think in general Python has erred on the side of having too many
different syntactical uses for commas. We killed a few in 3.0 with the
introduction of "except E as v" and the demotion of print to a
function call. Perhaps we should aim to kill "assert B, S" as well?
--
--Guido van Rossum (home page: http://www.python.org/~guido/)
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] ',' precedence in documentation
On Sun, Sep 14, 2008 at 3:00 PM, Guido van Rossum <[EMAIL PROTECTED]> wrote: > > I think in general Python has erred on the side of having too many > different syntactical uses for commas. We killed a few in 3.0 with the > introduction of "except E as v" and the demotion of print to a > function call. Perhaps we should aim to kill "assert B, S" as well? And replace it with what? > > -- > --Guido van Rossum (home page: http://www.python.org/~guido/) > ___ > Python-Dev mailing list > [email protected] > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/musiccomposition%40gmail.com > -- Cheers, Benjamin Peterson "There's no place like 127.0.0.1." ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
