Re: [Python-Dev] lifting of prohibition against readlines inside a for line in file in Py3?
Terry Reedy wrote: I suspect your original query got lost in the shuffle. If you do not get an answer this time, file an issue on the tracker bugs.python.org but do not select whether it is a behavior or doc issue. At least, it will stay open until resolved. Filing a tracker issue is probably a good idea regardless. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Peephole Optimization
Hi, I was looking around possibilities of bytecode optimizations in cpython and was looking at some older bugs. One of them being issue#2499http://bugs.python.org/issue2499; the following line kind of confuses me and wasnt sure what exactly Raymond(et al) is planning, as i presume that bytecode optimizations are much _easier_ than optimizations in AST. Most of the peepholer is going to be migrated up the chain, after the AST is generated, but before the opcodes are generated. If there are some optimizations that can be done in the bytecodes, then 'where' would be the suggested place to incorporate the same; as i also see that some of the more rudimentary optimizations have been rejected in many of the patches. Regards, -V- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Peephole Optimization
On Thu, Feb 19, 2009 at 8:53 AM, Venkatraman S venka...@gmail.com wrote: Hi, Hi ... If there are some optimizations that can be done in the bytecodes, then 'where' would be the suggested place to incorporate the same; The way I modify function's bytecode now (... but I am open to further suggestions ... ;) is writing decorators ... but this is not valid for more general transformations (AFAICS ...). -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Peephole Optimization
[Venkatraman S] the following line kind of confuses me and wasnt sure what exactly Raymond(et al) is planning, I think the AST optimization work is being pursued by tlee. See his current results on the branch: tlee-ast-optimize/ I don't know if this work has stalled but it was off to a good start. as i presume that bytecode optimizations are much _easier_ than optimizations in AST. . . . Most of the peepholer is going to be migrated up the chain, after the AST is generated, but before the opcodes are generated. I don't think your presumption is true. For some transformations (especially constant folding), more is possible with AST than with bytecode. We won't know for sure until the work is done and there is a good, working patch. Right now, the peepholer is limited by basic blocks, by only rewriting opcode sequences that are same or shorter size, and has to do tricks like writing-out NOPs and the eliminating them while rewriting the jump targets. Essentially, it disassembles fragments, analyzes them, and reassembles them. Better to start before the assembly, do the AST rewrites, and then less the existing assembler do its thing. When it comes to optimizing long fragements and extended args, the peepholer punts and does nothing. All of this makes it tightly bound to a given set of opcodes and everything must be re-evaluated whenever someone proposes a new opcode or changes the meaning of the existing code. If there are some optimizations that can be done in the bytecodes, then 'where' would be the suggested place to incorporate the same; The where is being developed. Your help would be welcome. as i also see that some of the more rudimentary optimizations have been rejected in many of the patches. A number of patches have been accepted. The ones that were rejected were either too complicated, interfered with other optimizations, or did too much work for too little payoff (i.e. deadcode elimination which costs compilation time but doesn't actually speedup code). Raymond ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] lifting of prohibition against readlines inside a for line in file in Py3?
On Wed, 18 Feb 2009 at 20:31, Guido van Rossum wrote: On Wed, Feb 18, 2009 at 6:38 PM, rdmur...@bitdance.com wrote: On Wed, 18 Feb 2009 at 21:25, Antoine Pitrou wrote: Nick Coghlan ncoghlan at gmail.com writes: I *think* the 2.x system had an internal buffer that was used by the file iterator, but not by the file methods. With the new IO stack for 3.0, there is now a common buffer shared by all the file operations (including iteration). However, given that the lifting of the restriction is currently undocumented, I wouldn't want to see a commitment to keeping it lifted until we know that it won't cause any problems for the io-in-c rewrite for 3.1 (hopefully someone with more direct involvement with that rewrite will chime in, since they'll know a lot more about it than I do). As you said, there is no special buffering for the file iterator in 3.x, which means the restriction could be lifted (actually there is nothing relying on this restriction in the current code, except perhaps the telling flag in TextIOWrapper). Currently I have python (2.x) code that uses 'readline' instead of 'for x in myfile' in order to avoid the 'for' buffering (ie: be presented with the next line as soon as it is written to the other end of a pipe, instead of waiting for the buffer to fill). Does no special buffering mean that 'for' doesn't use a read-ahead buffer in p3k, or that readline does use such a buffer, because the latter could make my code break unexpectedly when porting to p3k. Have a look at the code in io.py (class TextIOWrapper): http://svn.python.org/view/python/branches/py3k/Lib/io.py?view=log I believe it doesn't force reading ahead more than necessary. If a single low-level read() returns enough data to satisfy the __next__() or readline() (or it can be satisfied from data already buffered) then it won't force reading more. Hmm. I'm not sure I'm reading the code right, but it looks from the docstrings like TextIOWrapper expects to read from a BufferedIOBase object, whose doc string contains this comment: If the argument is positive, and the underlying raw stream is not 'interactive', multiple raw reads may be issued to satisfy the byte count (unless EOF is reached first). But for interactive raw streams (XXX and for pipes?), at most one raw read will be issued, and a short result does not imply that EOF is imminent. Since the 'pipe' comment is an XXX, it is not clear that my use case is covered. However, the actual implementation of readinto seems to only call 'read' once, so as long as the 'read' of the subclass returns whatever bytes are available, then it looks good to me :) Since TextIOWrapper is careful to call 'read1' on the wrapped buffer object, and the one place that 'read1' has a docstring clearly indicates that it does at most one read and returns whatever data is ready, it seems that the _intent_ of the code is as you expressed. I'm a python programmer first, and my C is pretty rusty, so I'm not sure if I'm up to looking through the new C code to see how this got translated. I'm thinking that both my use case (and in my case 'for' should now work for me) and the OP's are the way it is intended to work, but documentation of this seems like it would be a good idea. Since the OP doesn't seem to have opened a ticket, I did so: http://bugs.python.org/issue5323. As I said there, I'm willing to work on doc and test patches if this is the behavior the io library is required to have in 3.x. --RDM ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] lifting of prohibition against readlines inside a for line in file in Py3?
Hello, rdmurray at bitdance.com writes: Since the 'pipe' comment is an XXX, it is not clear that my use case is covered. However, the actual implementation of readinto seems to only call 'read' once, so as long as the 'read' of the subclass returns whatever bytes are available, then it looks good to me :) The only thing I can say is that it should work as you expect it to. But it would be great if you could give it a try and share your observations with us. We haven't had many people report on real-world uses of the new IO library (although we could optimistically deduce from it that people aren't having any problems, except for the speed issue). Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] A suggestion: Do proto-PEPs in Google Docs
[Aside to Guido: Oops, I think I accidentally sent you a contentless reply. Sorry!] As a suggestion, I think this is relevant to everybody who might be writing a PEP, so I'm cross-posting to Python-Dev. Probably no discussion is needed, but Reply-To is set to Python-Ideas. On Python-Ideas, Guido van Rossum writes: On Thu, Feb 19, 2009 at 2:12 AM, Greg Ewing wrote: Fifth draft of the PEP. Re-worded a few things slightly to hopefully make the proposal a bit clearer up front. Wow, how I long for the days when we routinely put things like this under revision control so its easy to compare versions. FWIW, Google Docs is almost there. Working with Brett et al on early drafts of PEP 0374 was easy and pleasant, and Google Docs gives control of access to the document to the editor, not the Subversion admin. The ability to make comments that are not visible to non-editors was nice. Now that it's in Subversion it's much less convenient for me (a non-committer). I actually have to *decide* to work on it, rather than simply raising a browser window, hitting refresh and fixing a typo or two (then back to day job work). The main problem with Google Docs is that is records a revision automatically every so often (good) but doesn't prune the automatic commits (possibly hard to do efficiently) OR mark user saves specially (easy to do). This lack of marking important revisions makes the diff functionality kind of tedious. I don't know how automatic the conversion to reST was, but the PEP in Subversion is a quite accurate conversion of the Google Doc version. Overall, I recommend use of Google Docs for Python-Ideas level of PEP drafts. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] IO implementation: in C and Python?
As we prepare to merge the io-c branch, the question has come up [1] about the original Python implementation. Should it just be deleted in favor C version? The wish to maintain the two implementations together has been raised on the basis that Python is easier to experiment on and read (for other vm implementors). Thoughts? http://bugs.python.org/issue4565 -- Regards, Benjamin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] IO implementation: in C and Python?
On Thu, 19 Feb 2009 at 21:41, Benjamin Peterson wrote: As we prepare to merge the io-c branch, the question has come up [1] about the original Python implementation. Should it just be deleted in favor C version? The wish to maintain the two implementations together has been raised on the basis that Python is easier to experiment on and read (for other vm implementors). I'm personally +0 on this, but I note that it is easier to read not just for other vm implementors, but for users. Witness the question about the behavior of 'for' vs 'readline'. I'd have had a much harder time figuring out the behavior if I'd had to read the C code. That said, I'm not personally sure if maintaining both versions is worth it. Real python developers should make that decision :) --RDM ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] IO implementation: in C and Python?
On Thu, Feb 19, 2009 at 19:41, Benjamin Peterson benja...@python.orgwrote: As we prepare to merge the io-c branch, the question has come up [1] about the original Python implementation. Should it just be deleted in favor C version? The wish to maintain the two implementations together has been raised on the basis that Python is easier to experiment on and read (for other vm implementors). Probably not a surprise, but +1 from me for keeping the pure Python version around for the benefit of other VMs as well as a reference implementation. -Brett ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] A suggestion: Do proto-PEPs in Google Docs
On Thu, Feb 19, 2009 at 7:17 PM, Stephen J. Turnbull turnb...@sk.tsukuba.ac.jp wrote: On Python-Ideas, Guido van Rossum writes: On Thu, Feb 19, 2009 at 2:12 AM, Greg Ewing wrote: Fifth draft of the PEP. Re-worded a few things slightly to hopefully make the proposal a bit clearer up front. Wow, how I long for the days when we routinely put things like this under revision control so its easy to compare versions. FWIW, Google Docs is almost there. Working with Brett et al on early drafts of PEP 0374 was easy and pleasant, and Google Docs gives control of access to the document to the editor, not the Subversion admin. The ability to make comments that are not visible to non-editors was nice. Now that it's in Subversion it's much less convenient for me (a non-committer). I actually have to *decide* to work on it, rather than simply raising a browser window, hitting refresh and fixing a typo or two (then back to day job work). The main problem with Google Docs is that is records a revision automatically every so often (good) but doesn't prune the automatic commits (possibly hard to do efficiently) OR mark user saves specially (easy to do). This lack of marking important revisions makes the diff functionality kind of tedious. I don't know how automatic the conversion to reST was, but the PEP in Subversion is a quite accurate conversion of the Google Doc version. Overall, I recommend use of Google Docs for Python-Ideas level of PEP drafts. Rietveld would also be a good option: it offers more at-will revision control (rather than whenever Google Docs decides), allows you to attach comments to the revisions, and will give you nice diffs between PEP iterations. Collin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] IO implementation: in C and Python?
On Thu, Feb 19, 2009 at 8:38 PM, Brett Cannon br...@python.org wrote: On Thu, Feb 19, 2009 at 19:41, Benjamin Peterson benja...@python.org wrote: As we prepare to merge the io-c branch, the question has come up [1] about the original Python implementation. Should it just be deleted in favor C version? The wish to maintain the two implementations together has been raised on the basis that Python is easier to experiment on and read (for other vm implementors). Probably not a surprise, but +1 from me for keeping the pure Python version around for the benefit of other VMs as well as a reference implementation. You have been practice channeling me again, haven't you? I like the idea of having two (closely matching) implementations very much. In 2.x we did this on an ad-hoc basis, e.g. [c]StringIO, pickle/cPickle, heapq/_heapq. In 3.0 we've moved towards standardizing the approach -- the foo.py file first defines everything and then tries to import * from _foo on top of that. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] IO implementation: in C and Python?
On Thu, Feb 19, 2009 at 9:07 PM, Guido van Rossum gu...@python.org wrote: On Thu, Feb 19, 2009 at 8:38 PM, Brett Cannon br...@python.org wrote: On Thu, Feb 19, 2009 at 19:41, Benjamin Peterson benja...@python.org wrote: As we prepare to merge the io-c branch, the question has come up [1] about the original Python implementation. Should it just be deleted in favor C version? The wish to maintain the two implementations together has been raised on the basis that Python is easier to experiment on and read (for other vm implementors). Probably not a surprise, but +1 from me for keeping the pure Python version around for the benefit of other VMs as well as a reference implementation. You have been practice channeling me again, haven't you? I like the idea of having two (closely matching) implementations very much. Agreed. In particular, this helps any projects that are focused on improving the performance of pure-Python code: they can work on minimizing the delta between the Python and C versions. Collin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] IO implementation: in C and Python?
Guido van Rossum wrote: On Thu, Feb 19, 2009 at 8:38 PM, Brett Cannon br...@python.org wrote: On Thu, Feb 19, 2009 at 19:41, Benjamin Peterson benja...@python.org wrote: As we prepare to merge the io-c branch, the question has come up [1] about the original Python implementation. Should it just be deleted in favor C version? The wish to maintain the two implementations together has been raised on the basis that Python is easier to experiment on and read (for other vm implementors). Probably not a surprise, but +1 from me for keeping the pure Python version around for the benefit of other VMs as well as a reference implementation. You have been practice channeling me again, haven't you? I like the idea of having two (closely matching) implementations very much. In 2.x we did this on an ad-hoc basis, e.g. [c]StringIO, pickle/cPickle, heapq/_heapq. In 3.0 we've moved towards standardizing the approach -- the foo.py file first defines everything and then tries to import * from _foo on top of that. Currently, if I want to verify that (say) cFoo and Foo do the same thing, or compare their speed, it's easy because I can import the modules separately. Given the 3.0 approach, how would one access the Python versions without black magic or hacks? -- Steven ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] IO implementation: in C and Python?
On Thu, Feb 19, 2009 at 21:35, Steven D'Aprano st...@pearwood.info wrote: Guido van Rossum wrote: On Thu, Feb 19, 2009 at 8:38 PM, Brett Cannon br...@python.org wrote: On Thu, Feb 19, 2009 at 19:41, Benjamin Peterson benja...@python.org wrote: As we prepare to merge the io-c branch, the question has come up [1] about the original Python implementation. Should it just be deleted in favor C version? The wish to maintain the two implementations together has been raised on the basis that Python is easier to experiment on and read (for other vm implementors). Probably not a surprise, but +1 from me for keeping the pure Python version around for the benefit of other VMs as well as a reference implementation. You have been practice channeling me again, haven't you? I like the idea of having two (closely matching) implementations very much. In 2.x we did this on an ad-hoc basis, e.g. [c]StringIO, pickle/cPickle, heapq/_heapq. In 3.0 we've moved towards standardizing the approach -- the foo.py file first defines everything and then tries to import * from _foo on top of that. Currently, if I want to verify that (say) cFoo and Foo do the same thing, or compare their speed, it's easy because I can import the modules separately. Given the 3.0 approach, how would one access the Python versions without black magic or hacks? As of right now there is no standard practice, although coming up with a standard way of handling this would probably be a good thing as this will also help with the testing story. -Brett ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] IO implementation: in C and Python?
On Fri, Feb 20, 2009 at 12:35 AM, Steven D'Aprano st...@pearwood.info wrote: Currently, if I want to verify that (say) cFoo and Foo do the same thing, or compare their speed, it's easy because I can import the modules separately. Given the 3.0 approach, how would one access the Python versions without black magic or hacks? My prefered way to handle this is to keep the original Python implementations with a leading underscore (e.g., pickle._Pickler). I found this was the easiest way to test the C and Python implementations without resorting to import hacks. -- Alexandre ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
