Re: [Python-Dev] PEP 343 - next steps
Barry Warsaw wrote: > On Fri, 2005-06-10 at 16:23, Guido van Rossum wrote: > >>While there's still some activity in the Wiki, nothing (to me) sounds >>like signs of serious disagreement or truly better alternatives. So I >>think I'd like to move forward towards acceptance soon (before >>EuroPython). > > > Well, I finally read PEP 343 in its entirety. Having not really had the > time to follow this thread, I have to say "well done". I like the PEP a > lot and can see many places in my own code where it would make things > more readable. > > I have one comment, and I may be too late or may be repeating other > suggestions (no, I haven't poured through the wiki page yet). I > actually don't think "with" reads as well and would propose "when". The idea behind 'with' is that the block is executed while holding (i.e. 'with') the resource. I think the '-ing' form of the example templates is a holdover from the days when the PEP used the 'do' keyword - I find that past tense or noun forms tend to read better than present tense for custom built with templates named after the action that occurs on entry: # Past tense with locked(my_lock): with opened(my_file, mode): with redirected_stdout(my_stream): # Noun forms with my_lock: with open(my_file, mode) as f: with extra_precision(): with decimal.getcontext() as ctx: The place where the '-ing' form still makes some sense is when the template is named after the action that will occur at the *end* of the block: with closing(some_gen()) as g: with releasing(some_rsrc()) as r: Cheers, Nick. -- Nick Coghlan | [EMAIL PROTECTED] | Brisbane, Australia --- http://boredomandlaziness.blogspot.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 343 - next steps
Nick Coghlan wrote: > Nick Coghlan wrote: > >>Then all the two new opcodes (e.g. ALLOW_ASYNC, BLOCK_ASYNC) would >>have to do is set the state of tstate->allow_async_exc appropriately. > > > Actually, to allow the use of 'with' statements inside functions > called via EXPR and the call to __enter__, it would be necessary to > support nesting of calls to BLOCK_ASYNC and ALLOW_ASYNC, so a better > approach would be to make the field "tstate->block_async_exc", > incrementing it in BLOCK_ASYNC, and decrementing it in ALLOW_ASYNC. And, replying to myself yet again. . . Such an approach would obviously break in the face of an exception that occurs between the BLOCK_ASYNC and ALLOW_ASYNC opcodes (async exceptions would remain blocked in the thread). This can be dealt with by storing the question of whether or not async exceptions are permitted as part of the *frame* state, rather than the thread state. That is, make the field access be "f->allow_async_events", and have ALLOW_ASYNC and BLOCK_ASYNC set or clear that flag, respectively. The error handling code after the main opcode switch statement can then simply set "f->allow_async_exc" back to true in order to ensure that handling of async exceptions is resumed correctly in the event of try/finally or try/except blocks causing further execution within the frame. (Given that only expressions are possible on the resource acquisition line, dealing with the WHY_EXPRESSION case seems to be all that would be needed). Regards, Nick. P.S. My suggested semantics and the above implementation outline should be up on the Wiki shortly. -- Nick Coghlan | [EMAIL PROTECTED] | Brisbane, Australia --- http://boredomandlaziness.blogspot.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] hashlib - faster md5/sha, adds sha256/512 support
I have finished up the hashlib work I started on in feb/march for patch 1121611 and 935454 after some discussion on this list. The full patch including tests and module documentation has been posted in the sf patch 1121611 as hashlib-008. I believe it is done and ready and would like to commit it after a review. Let the reviewing begin! For easy viewing, here's what the module documentation looks like: http://electricrain.com/greg/hashlib-py25-doc/module-hashlib.html The code is in the hashlib-008.patch file: http://redirx.com/?3e19 hashlib incorporates both the 1121611 openssl hash support and the 935454 sha256+sha512 module support into a single hashlib module that picks the fastest implementation of each algorithm to use. OpenSSLs implementations of md5 and sha1 are nearly 2x as fast as the existing python builtin versions. The (now legacy) md5 and sha modules are turned into simple stubs that use hashlib. The actual sourceforge patch tracker entries: https://sourceforge.net/tracker/?func=detail&aid=1121611&group_id=5470&atid=305470 https://sourceforge.net/tracker/?func=detail&aid=935454&group_id=5470&atid=305470 Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 343 - next steps
Nick Coghlan wrote: > Either way, the code generation for the with statement could simply > emit BLOCK_ASYNC as the first opcode, then ALLOW_ASYNC as the opcode > immediate preceding SETUP_FINALLY. Correct behaviour in the face of > asynchronous events is then guaranteed, even for cases where the > resource is acquired in EXPR, or when __enter__ is written in Python. Coming up out of the implementation weeds, this can be described as follows: """ Atomic resource acquisition The resource acquisition line that starts the with statement is executed atomically. That is, the Python interpreter prevents asynchronous events and automatic context switches while the resource is being acquired. Context switches due to the current thread becoming blocked (e.g. by attempting to acquire a synchronisation lock) will still occur, avoiding deadlock. Automatic context switches or asynchronous events that would have occurred during evaluation of the resource acquisition line are deferred so that they occur immediately prior to the execution of the contained suite. This ensures that the acquired resource is still properly released in the face of an asynchronous exception such as KeyboardInterrupt. To avoid blocking exceptions for an extended period, 'expensive' calculations can be carried out on the line preceding the with statement, and stored in a local variable for use in the resource acquisition: tmp = some_expensive_calculation() with tmp as rsrc: # Perform operations with resource Note that, in this case, the actual resource acquisition should occur in tmp.__enter__, in order to enjoy the benefits of the atomic resource acquisition. """ Cheers, Nick. -- Nick Coghlan | [EMAIL PROTECTED] | Brisbane, Australia --- http://boredomandlaziness.blogspot.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 343 - next steps
Nick Coghlan wrote: > Then all the two new opcodes (e.g. ALLOW_ASYNC, BLOCK_ASYNC) would > have to do is set the state of tstate->allow_async_exc appropriately. Actually, to allow the use of 'with' statements inside functions called via EXPR and the call to __enter__, it would be necessary to support nesting of calls to BLOCK_ASYNC and ALLOW_ASYNC, so a better approach would be to make the field "tstate->block_async_exc", incrementing it in BLOCK_ASYNC, and decrementing it in ALLOW_ASYNC. Cheers, Nick. -- Nick Coghlan | [EMAIL PROTECTED] | Brisbane, Australia --- http://boredomandlaziness.blogspot.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 343 - next steps
Guido van Rossum wrote:
>>It also considers the possibility of using with statements in an RAII
>>style by acquiring the resource in __init__ or __new__ rather than
>>__enter__.
>
>
> Python isn't C++; you shouldn't do that.
>
>
>>This is a topic that got a brief mention during early discussion here,
>>but doesn't seem to be covered in the current version of PEP 343.
>
>
> I haven't been infected with the RAII meme, so I can't help you here.
I expected your reaction to be along these lines, and at one point
updated the Wiki to say basically this.
Then I went back to PEP 343, and had another look at the examples. The
one that sprang out at me was the generic 'closing()' template.
Being able to say "ensure method X of object Y is called when the
block exits" seems like a *really* useful feature for 'with' statements.
However, when using this approach, the resource is always going to be
acquired during evaluation of EXPR, since it needs to be passed an an
argument to the 'closing()' (or equivalent) template.
If the guarantee only applies to __enter__ though, then such templates
would have to be written 'correctly' using deferred evaluation of the
argument:
class closing(object):
def __init__(self, resource_factory)
self.factory = resource_factory
self.resource = None
def __enter__(self):
if self.resource is not None:
raise RuntimeException("Cannot nest use of template")
self.resource = self.factory()
return self.resource
def __exit__(self, *exc_info):
self.resource.close()
self.resource = None
Then usage gets a bit uglier, because a lambda is needed in order to
defer evaluation of the argument:
with closing(lambda: open("somefile.txt", "r")) as f:
# Do stuff
If, however, two new opcodes were added that blocked and unblocked
asynchronous exceptions in a given thread, then not only would it be
possible to implement 'closing()' in the natural way, then resource
handlers written in Python would 'do the right thing' to.
Specifically, what if we added a new boolean flag 'permit_async_exc'
to the thread state that is checked at the top of main loop in ceval.c
where the asynchronous events are handled?
The simplest trick would be to change the line "if (--_Py_Ticker < 0)"
to "if ((--_Py_Ticker < 0) && (_Py_Ticker = 0,
tstate->allow_async_exc))" (using the comma operator to keep
_Py_Ticker from overflowing in pathological cases).
Then all the two new opcodes (e.g. ALLOW_ASYNC, BLOCK_ASYNC) would
have to do is set the state of tstate->allow_async_exc appropriately.
Notice that _Py_Ticker keeps decrementing while the async_exceptions
are blocked, so that any queued events should be handled immediately
after execution of the BLOCK_ASYNC call (unless the next opcode is
SETUP_FINALLY, in which case that opcode will be executed first).
The setting of _Py_Ticker to zero, and the check of the tstate field
only occur when _Py_Ticker goes negative, so they only add to the cost
of executing the ceval loop when the check interval has expired.
Compared to the cost of Py_MakePendingCalls, and the check for thread
context switches, these costs should be negligible.
An alternative implementation would still allow thread switches to
happen, and prevent just the asynchronous events. Then the ALLOW_ASYNC
opcode could just force _Py_Ticker to zero in order to ensure prompt
evaluation of any pending events.
Either way, the code generation for the with statement could simply
emit BLOCK_ASYNC as the first opcode, then ALLOW_ASYNC as the opcode
immediate preceding SETUP_FINALLY. Correct behaviour in the face of
asynchronous events is then guaranteed, even for cases where the
resource is acquired in EXPR, or when __enter__ is written in Python.
Cheers,
Nick.
--
Nick Coghlan | [EMAIL PROTECTED] | Brisbane, Australia
---
http://boredomandlaziness.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 343 - next steps
On Fri, 2005-06-10 at 16:23, Guido van Rossum wrote:
> While there's still some activity in the Wiki, nothing (to me) sounds
> like signs of serious disagreement or truly better alternatives. So I
> think I'd like to move forward towards acceptance soon (before
> EuroPython).
Well, I finally read PEP 343 in its entirety. Having not really had the
time to follow this thread, I have to say "well done". I like the PEP a
lot and can see many places in my own code where it would make things
more readable.
I have one comment, and I may be too late or may be repeating other
suggestions (no, I haven't poured through the wiki page yet). I
actually don't think "with" reads as well and would propose "when". The
word I'd really like to use here is "while" but given that we already
have while loops and "as" is optional, I don't think that's a
possibility. If you made "as" required, it would be though, I think.
Look at the given examples:
with locking(mylock):
when locking(mylock):
with opening("/etc/passwd") as f:
when opening("/etc/passwd") as f:
with opening(filename, "w") as f:
with redirecting_stdout(f):
print "Hello world"
when opening(filename, "w") as f:
when redirecting_stdout(f):
print "Hello world"
with signal_blocking():
when signal_blocking():
> - throw() is a term taken from Java & C++. We can't call the method
> raise() -- but perhaps we can call it next_raising() or next_raise(),
> which emphasizes the similarity with next(). Thoughts? I'm not strong
> on this; I think throw() is fine too, especially since I expect that
> it will be used explicitly extremely rarely -- the only customer is
> the with_template decorator.
As others have mentioned "throw()" is fine.
> - Whether and how to keep a door open for a future extension to the
> syntax that allows multiple resources to be acquired in a single
> with-statement. Possible syntax could be
>
> (a)with EXPR1 [as VAR1], EXPR2 [as VAR2], EXPR3 [as VAR3], ...:
>
> or
>
> (b)with EXPR1, EXPR2, EXPR3, ... as VAR1, VAR2, VAR3, ...:
I agree that (a) is better and feel +1 on keeping the door open to it.
-Barry
signature.asc
Description: This is a digitally signed message part
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 343 - next steps
At 09:02 AM 6/11/2005 -0700, Guido van Rossum wrote:
>[Nick]
> > I also added a new question to the Wiki regarding what, if any,
> > guarantees will be made regarding when (or if) asynchronous interrupts
> > will be blocked. Given that the call to __enter__() can invoke
> > arbitrary Python code, is pushing the with statement setup inside a
> > single opcode actually sufficient?
>
>Not sufficient if __enter__ is written in Python; in that case,
>*nothing* can be done to make it sufficient.
I think PEP 343 would greatly benefit from some discussion of the
signalling issue. I've been wracking my brain a bit on it, and I'm having
trouble seeing where the problem is.
>The single opcode makes it possible to implement __enter__ in C to
>solve the issue in specific cases.
Or if __enter__ is a no-op, that would also solve the problem. Suitability
for 'with' could be keyed off of the __exit__ method instead. If there's
no __enter__, or the __enter__ is a default no-op __enter__ written in C,
then __exit__ is sufficient for many resources.
> > It also considers the possibility of using with statements in an RAII
> > style by acquiring the resource in __init__ or __new__ rather than
> > __enter__.
>
>Python isn't C++; you shouldn't do that.
Actually, it seems to me that it's standard procedure in today's Python:
objects allocate resources in __init__, and then have a 'close()' or some
similar method to release them. That's why so many people see this:
with open('foo') as f:
...
As a natural use of the 'with' statement, whether they have any familiarity
with C++ or not. (My own experiences with C++ are perhaps a decade out of
date, long before RAII was a meme in any event.)
It seems to me that there are two sensible patterns for using 'with':
1. __enter__ allocates or acquires a resource, __exit__ releases it
2. EXPR allocates or acquires a resource, __exit__ releases it (and
__enter__ doesn't execute any Python, because it's a no-op)
Pattern 1 is useful for transactions, locks, redirection, decimal contexts,
etc.
Pattern 2 is useful for resources that are acquired by creation anyway,
like files.
But maybe my understanding is flawed, because I don't *really* understand
the signal issue. I've been assuming that you're talking about the fact
that an exception can occur anywhere, due to e.g. KeyboardInterrupt or an
exception raised by some other signal handler. It seems to me that using
pattern 2, one could write 'closing()' safely
by doing this:
class closing(object):
def __init__(self, resource):
try:
self.resource = resource
except:
resource.close()
raise
def __exit__(self,*exc):
self.resource.close()
but maybe I'm still fundamentally "not getting" how the signal issue
manifests in practice. I suppose it's true that *another* exception could
then occur between the LOAD_ATTR and the CALL opcodes in the exception
handler here, but it's also true that this could happen with any Python
exception-handling code, and it could also happen in __exit__(). So I'm
not sure I see why this should influence the design of 'with' very much,
other than to point out that no Python code is safe from asynchronous
exceptions.
In any case, C-implemented types can handle this situation just fine, so a
'closing' written in C could easily be included with the implementation.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 343 - next steps
[Nick] > I also added a new question to the Wiki regarding what, if any, > guarantees will be made regarding when (or if) asynchronous interrupts > will be blocked. Given that the call to __enter__() can invoke > arbitrary Python code, is pushing the with statement setup inside a > single opcode actually sufficient? Not sufficient if __enter__ is written in Python; in that case, *nothing* can be done to make it sufficient. The single opcode makes it possible to implement __enter__ in C to solve the issue in specific cases. > It also considers the possibility of using with statements in an RAII > style by acquiring the resource in __init__ or __new__ rather than > __enter__. Python isn't C++; you shouldn't do that. > This is a topic that got a brief mention during early discussion here, > but doesn't seem to be covered in the current version of PEP 343. I haven't been infected with the RAII meme, so I can't help you here. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Thoughts on stdlib evolvement
> "Gustavo" == Gustavo Niemeyer <[EMAIL PROTECTED]> writes: Gustavo> An issue to consider about this is that maintainers (not Gustavo> talking about you or anyone else specifically) have Gustavo> different concepts of stability, and while it may seem Gustavo> perfectly ok to refactor external modules between two Gustavo> stable releases, doing so in the standard library would Gustavo> spread fear and "python is so untrustful" feelings. This simply hasn't been a problem in XEmacs's equivalent to the standard library. In fact, the library modules tend to be more stable than the core, for several reasons. One important one is that the modules we distribute are not necessarily the bleeding edge. In particular, we try to keep up (within a couple weeks) with bugfixes to stable lines, but often lag several months after a major refactoring. This multitrack approach does involve some extra administrative work, but the burden on the core team is minimal. Most modules are managed by volunteers who do not feel able to contribute to the core (and often not to the coding of "their" module), but are very pleased to be able to contribute in this way. They are usually far more competent than the core developers to judge when new features are sufficiently attractive and stable to warrant updating to a refactored version, too. They tend to be more conservative than the module's lead maintainer, too. I will grant that XEmacs deserves (and probably has) a lower trust rating than Python, but that is due to flaws in the _core_, not in the package management. -- School of Systems and Information Engineering http://turnbull.sk.tsukuba.ac.jp University of TsukubaTennodai 1-1-1 Tsukuba 305-8573 JAPAN Ask not how you can "do" free software business; ask what your business can "do for" free software. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 343 - next steps
Guido van Rossum wrote: > While there's still some activity in the Wiki, nothing (to me) sounds > like signs of serious disagreement or truly better alternatives. So I > think I'd like to move forward towards acceptance soon (before > EuroPython). I agree with keeping throw() as the exception injection method name, and the addition of required parentheses when using tuple unpacking when creating VAR. I also added a new question to the Wiki regarding what, if any, guarantees will be made regarding when (or if) asynchronous interrupts will be blocked. Given that the call to __enter__() can invoke arbitrary Python code, is pushing the with statement setup inside a single opcode actually sufficient? It also considers the possibility of using with statements in an RAII style by acquiring the resource in __init__ or __new__ rather than __enter__. This is a topic that got a brief mention during early discussion here, but doesn't seem to be covered in the current version of PEP 343. Regards, Nick. -- Nick Coghlan | [EMAIL PROTECTED] | Brisbane, Australia --- http://boredomandlaziness.blogspot.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 343 - next steps
Andrew Koenig wrote: > Of course, this usage shows that the syntax is unnecessary in this context, > but what I care about is > > def f(x as (a, b)): > # ... > > which has the advantage over the alternative > > def f(x): > (a, b) = x > # ... > > that if you call f with the wrong arguments, you get a clearer diagnostic > message. Doesn't tuple unpacking give you what you want here already? Py> def f((a, b)): ... print a, b ... Py> f(1, 2) Traceback (most recent call last): File "", line 1, in ? TypeError: f() takes exactly 1 argument (2 given) Py> f((1, 2)) 1 2 A couple of more complex examples: Py> def f((a, b), (c, d)): ... print a, b, c, d ... Py> f((1, 2), (3, 4)) 1 2 3 4 Py> def f((a, b, (c, d))): ... print a, b, c, d ... Py> f((1, 2, (3, 4))) 1 2 3 4 About the only downside is the need to rebuild the tuple if you actually need it. Cheers, Nick. -- Nick Coghlan | [EMAIL PROTECTED] | Brisbane, Australia --- http://boredomandlaziness.blogspot.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 343 - next steps
"Andrew Koenig" <[EMAIL PROTECTED]> wrote: > Of course, this usage shows that the syntax is unnecessary in this context, > but what I care about is > > def f(x as (a, b)): > # ... > > which has the advantage over the alternative > > def f(x): > (a, b) = x > # ... > > that if you call f with the wrong arguments, you get a clearer diagnostic > message. Thanks to the time machine, you can already do this: def f((a, b)): # ... -- Christian Tanzerhttp://www.c-tanzer.at/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 343 - next steps
> The issue is: if we allow VAR to be a > comma-separated list of variables now, that cuts off the extension to > (a) in the future; so the PEP would have to be amended to state that > VAR must be a single variable or a list of variables IN PARENTHESES. > Thoughts? I am not sure how relevant this is, but the syntax "x as (a, b)" sure looks to me like (a, b) is a tuple. Of course, I'm biased by ML, which has a feature that I wish Python had: fun f(x as (a, b)) = ... This says that f's argument must be a 2-element tuple. It binds x to the argument and also binds a and b to the argument's components. Of course the program could be written this way: fun f(x) = let val (a, b) = x in ... end but the "as" syntax is more succinct, direct, and convenient. If such a feature were implemented in Python, I would imagine it to allow usages such as x as (a, b) = (3, 4) which would be equivalent to x = (a, b) = (3, 4) Of course, this usage shows that the syntax is unnecessary in this context, but what I care about is def f(x as (a, b)): # ... which has the advantage over the alternative def f(x): (a, b) = x # ... that if you call f with the wrong arguments, you get a clearer diagnostic message. It's kind of an edge case, and I am not seriously pushing for its adoption, but I do think it worth pointing out that when I see "x as (a, b)", that's what it immediately brings to mind. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] b32encode and NUL bytes
On Fri, Jun 10, 2005 at 01:13:35PM -0500, Jeff Epler wrote:
> > Is this a feature? I do see b32encode padding the string with NULs first.
This is bug #1170331, which was fixed when I applied patch #1171487
earlier this week.
> This also seems suspect:
> >>> base64.b32encode("\0a")
> 'ABQQ'
> >>> base64.b32decode(_)
> 'a'
This case is also fixed in CVS.
--amk
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
