Re: [Python-Dev] Py2.6 ideas
Martin v. Löwis wrote:
> Raymond Hettinger schrieb:
>> No need to go so widely off-track. The idea is to have an efficient type
>> that
>> is directly substitutable for tuples but is a bit more self-descriptive. I
>> like
>> to have the doctest result cast at NamedTuple('TestResults failed
>> attempted).
>> The repr of that result looks like TestResult(failed=0, attempted=15) but
>> is
>> still accessible as a tuple and passes easily into other functions that
>> expect a
>> tuple. This sort of thing would be handly for things like os.stat().
>> http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/500261
>
> I'd like to repeat Guido's question: Why does this still need to support
> the tuple interface (i.e. indexed access)?
So that it remains interoperable with existing libraries that expect a
tuple? Otherwise you'd be casting (and copying) every time you needed to
pass it to something that used indexed access.
Cheers,
Nick.
--
Nick Coghlan | [EMAIL PROTECTED] | Brisbane, Australia
---
http://www.boredomandlaziness.org
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] generic async io (was: microthreading vs. async io)
Greg Ewing <[EMAIL PROTECTED]> wrote: > > > An I/O operation passes a buffer, length, file and action and receives a > > token back. > > You seem to be using the word "threading" in a completely > different way than usual here, which may be causing some > confusion. Not really, though I may have been unclear again. Here is why that approach is best regarded as a threading concept: Perhaps the main current approach to using threads to implement asynchronous I/O operates by the main threads doing just that, and the I/O threads transferring the data synchronously. The reason that a token is needed is to avoid a synchronous data copy that blocks both threads. My general point is that all experience is that asynchronous I/O is best done by separating it completely from threads, and defining a proper asynchronous but NOT threaded interface. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: [EMAIL PROTECTED] Tel.: +44 1223 334761Fax: +44 1223 334679 ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New syntax for 'dynamic' attribute access
On Fri, Feb 16, 2007 at 12:40:54PM +1300, Greg Ewing wrote: > The quote is actually "a camel is a *racehorse* designed by a committee". > Camels are very good at surviving in the desert, but not so good at > winning a horse race (not camel race). Which is the point of the saying. That changes the meaning, but... have you ever tried to ride a horse designed by a group of Clever Individuals loosely connected by email? ;) I am afraid of even thinking of its ugliness and speed. (-: I think a committee is better than nothing, and I believe CP4E has been dropped from the agenda. Oleg. -- Oleg Broytmannhttp://phd.pp.ru/[EMAIL PROTECTED] Programmers don't die, they just GOSUB without RETURN. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Py2.6 ideas
Nick Coghlan schrieb: >> I'd like to repeat Guido's question: Why does this still need to >> support the tuple interface (i.e. indexed access)? > > So that it remains interoperable with existing libraries that expect a > tuple? Otherwise you'd be casting (and copying) every time you needed to > pass it to something that used indexed access. Can you give a few example, for libraries where this isn't already done? Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Py2.6 ideas
Martin v. Löwis wrote: > Nick Coghlan schrieb: >>> I'd like to repeat Guido's question: Why does this still need to >>> support the tuple interface (i.e. indexed access)? >> >> So that it remains interoperable with existing libraries that expect a >> tuple? Otherwise you'd be casting (and copying) every time you needed >> to pass it to something that used indexed access. > > Can you give a few example, for libraries where this isn't already done? I don't have any specific examples of that, no - that's why I phrased it as a question. However, another aspect that occurred to me is that inheriting from tuple has significant practical benefits in terms of speed and memory consumption, at which point it doesn't seem worthwhile to *remove* the indexing capability. I suppose you *could* write a completely new C-level record class, but given that Raymond's NamedTuple class gets good performance from a Python implementation, rewriting it in C seems like wasted effort. Regards, Nick. -- Nick Coghlan | [EMAIL PROTECTED] | Brisbane, Australia --- http://www.boredomandlaziness.org ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-3000] UserDict revamp
[Steve] > No complaints here. Not that you need my permission of course. ;-) Same here, obviously. [Raymond] > Thanks, I had already started working on this one. > Of course, everyone is welcome to contribute. Ok, you can count on that. -- EduardoOPadoan (eopadoan->altavix::com) Bookmarks: http://del.icio.us/edcrypt ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New syntax for 'dynamic' attribute access
>I think a committee is better than nothing, and I believe CP4E has > been dropped from the agenda. The general CP4E idea is part of the "General Pythonic Ideal", whatever it may be :P -- EduardoOPadoan (eopadoan->altavix::com) Bookmarks: http://del.icio.us/edcrypt ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Py2.6 ideas
Nick Coghlan schrieb: > However, another aspect that occurred to me is that inheriting from > tuple has significant practical benefits in terms of speed and memory > consumption, at which point it doesn't seem worthwhile to *remove* the > indexing capability. I'm not so sure that inheriting from tuple, and giving it named fields, has significant speed and memory benefits. In particular for the memory benefits, you can use __slots__ to achieve the same effects, and more efficiently so (because it you don't store the tuple length). As for speed, I would have to see measurements to be convinced it is faster. > I suppose you *could* write a completely new C-level record class, but > given that Raymond's NamedTuple class gets good performance from a > Python implementation, rewriting it in C seems like wasted effort. It wouldn't necessarily be rewriting: In the C API, you have already the PyStructSequence machinery (see posixmodule.c:stat_result_fields for an example). It's just that this machinery isn't available to Python code, yet, and no alternative convenience library is, either (other than using __slots__, which won't directly give indexed access). Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New syntax for 'dynamic' attribute access
Oleg Broytmann schrieb: >That changes the meaning, but... have you ever tried to ride a horse > designed by a group of Clever Individuals loosely connected by email? ;) I > am afraid of even thinking of its ugliness and speed. (-: >I think a committee is better than nothing, and I believe CP4E has > been dropped from the agenda. Ah, this passive voice again, and again the assumption that there is an agenda of python-dev. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New syntax for 'dynamic' attribute access
On Fri, Feb 16, 2007 at 01:42:54PM +0100, "Martin v. L?wis" wrote: > Ah, this passive voice again, and again the assumption that there is an > agenda of python-dev. Exactly opposite. There is no agenda, and thus there is no pojnt in trying to prevent new features in Python language based on the idea the features are not "4E". Python, IMHO, has gone far beyond "4E" already. Oleg. -- Oleg Broytmannhttp://phd.pp.ru/[EMAIL PROTECTED] Programmers don't die, they just GOSUB without RETURN. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] urllib2 EP + decr. startup time
Hello to all.
During more than two years i widely use urllib2 to write
commercial applications (almost for extracting data from web sites to
excel sheets)
and here is some enhanced enhanced for it:
1) Add support for 'HEAD' request (and maybe some other).
This needs small changes.
a)Add request_type = 'GET' to urllib2.Request class constructor.
b)Then put request_type value pass to http header, except Request has
data - in this case it's change to 'POST'.
The results of such request will be the same as in case of 'GET' request,
except zero size of body.
2)HTTH keep-alive opener. Almost complete realizations can be found
in urlgrabber (http://linux.duke.edu/projects/urlgrabber)(used by yum, so tested
well enough, i think). It's use urllib2 opener protocol and well integrated in
urllib2 structure. They need just little change to properly support
some headers.
3) Save HTTP exchange history. Now there is no suitable way to
obtain all sent and received headers. Received headers are saved only
for last response in redirection chain and sent headers are not saved at all.
I use run-time patching of httplib to intercept of the sent and received
data (may be i missed something?). Proposal is to add property
'history' to object returned from urllib2.urlopen - list
of objects which contain send/recv headers for all redirect chain.
4) Add possibilities to obtain underlying socket, used for recv http data.
Now it's impossible to work with http connection in async matter
(or i miss something again?).
If connection hangs then whole program hangs too and i don't known way
to fix this.
Of cause if you obtain such socket then you respond for compression and etc.
Now i use following code:
x = urrlib2.urlopen(.)
sock = x.fp._sock.fp._sock.
There only one problem, as i know, - chunked encoding. In case of
chunked encoding need to return socket-like object which
do all work to assemble chunks in original stream. I already use
such object for two years and it's ok.
5) And now for something completely different ;)).
This is just initial proposal and it needs enhancement. May be i
need put it to python-ideas list?
At last Goggle SOC there was one of problem to solve - the decrease
of interpreter's startup time.
'strace' command shows next: most of startup time the interpreter
try to find imported modules.
And most of them finished with 'not found' error, because of large
size of sys.path variable.
In future this time will be increase - setuptools adds many dirs to
search path
using pth files (to manage installed modules and eggs).
I propose to add something like .so caching which used in modern
*nix sytems to load
shared libraries.
a) Add to python interpreter --build-modules-index option. When python found
this opts it scans all dirs in paths and build dictionary
{module_name:module_path}.
Dict will be saved in external file (save only top-dir for packages
and path for one-file modules).
Also it saves in this file mtime for all pth files and dirs from
path and path variable.
b) When interpreter is started-up it, as usually, scans all path
dirs for pth files,
and using data saved in cache file check is new modules or search
dirs added or
old modified.
Then it read cache file and compares mtimes and path dirs. If it
isn't modified then
cache data used for fast module loading. If imported module didn't found in
cache - interpreter falls back to standard scheme.
Also it is necessary to add some options to control of using cache
like --disable-cache,
--clear-cache,disable cashing some dirs, etc.
---
K.Danilov aka KoDer
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Py2.6 ideas
At 01:38 PM 2/16/2007 +0100, Martin v. Löwis wrote: >Nick Coghlan schrieb: > > However, another aspect that occurred to me is that inheriting from > > tuple has significant practical benefits in terms of speed and memory > > consumption, at which point it doesn't seem worthwhile to *remove* the > > indexing capability. > >I'm not so sure that inheriting from tuple, and giving it named fields, >has significant speed and memory benefits. In particular for the memory >benefits, you can use __slots__ to achieve the same effects, and more >efficiently so (because it you don't store the tuple length). As for >speed, I would have to see measurements to be convinced it is faster. For an otherwise-pure Python implementation, the performance benefit of inheriting from a tuple is in having ready-made C implementations of hashing and comparison. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] urllib2 EP + decr. startup time
At 04:38 PM 2/16/2007 +0200, KoDer wrote:
>'strace' command shows next: most of startup time the interpreter
>try to find imported modules.
>And most of them finished with 'not found' error, because of large
>size of sys.path variable.
>In future this time will be increase - setuptools adds many dirs to
>search path
>using pth files (to manage installed modules and eggs).
Actually, under normal circumstances, most eggs installed are .zip files,
which the interpreter already caches the indexes of. Eggs installed as
directories should be increasing in rarity, except for in-development
packages installed via the "develop" command. Also, I plan to make
setuptools 0.7's "nest" packaging tool support managing packages the "old"
way, i.e., as a single flat directory structure, using manifest files to
support uninstallation and the like. So it should not really be the case
that this will keep increasing over time.
Also, are you aware that putting a zipped version of the standard library
on sys.path already speeds up startup considerably? Python since 2.3
automatically includes an appropriate entry in sys.path:
Python 2.3.4 (#53, May 25 2004, 21:17:02) [MSC v.1200 32 bit (Intel)] on win32
>>> import sys
>>> sys.path
['', 'C:\\WINDOWS\\system32\\python23.zip', '', 'c:\\Python23\\DLLs',
'c:\\Pytho
n23\\lib', 'c:\\Python23\\lib\\plat-win', 'c:\\Python23\\lib\\lib-tk',
'c:\\Pyth
on23']
Creating the zip file that's already listed in the default sys.path will
allow most startup imports to be handled without a lot of path checking.
>I propose to add something like .so caching which used in modern
>*nix sytems to load
>shared libraries.
>
>a) Add to python interpreter --build-modules-index option. When python
> found
>this opts it scans all dirs in paths and build dictionary
>{module_name:module_path}.
>Dict will be saved in external file (save only top-dir for packages
>and path for one-file modules).
>Also it saves in this file mtime for all pth files and dirs from
>path and path variable.
Unless you mean something more abstract by "dirs" than just filesystem
directories, this isn't going to help eggs or other zipped modules any,
compared to how they are now.
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] urllib2 EP + decr. startup time
2007/2/16, Phillip J. Eby <[EMAIL PROTECTED]>: > At 04:38 PM 2/16/2007 +0200, KoDer wrote: . > > > Also, are you aware that putting a zipped version of the standard library > on sys.path already speeds up startup considerably? Python since 2.3 > automatically includes an appropriate entry in sys.path: > zipped version has one weakness - you can't put .so(or dll) files inside. In my system 19 from 25 installed egg add directories ,not archives (because it's contain dll ). But even without egg directories >> ['', 'C:\\Python25\\Scripts', 'C:\\WINDOWS\\system32\\python25.zip', 'C:\\Python25\\DLLs', 'C:\\Python25\\lib', 'C:\\Python25\\lib\\plat-win', . 'C:\\Python25\\lib\\site-packages\\wx-2.8-msw-unicode'] len(sys.path) == 18 (without eggs) near 18 / 2 = 9 'file not found' errors for every first module import. So improvement of setuptools will help, but not solve this problem . -- K.Danilov aka KoDer ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] urllib2 EP + decr. startup time
At 07:29 PM 2/16/2007 +0200, KoDer wrote: >2007/2/16, Phillip J. Eby <[EMAIL PROTECTED]>: > > At 04:38 PM 2/16/2007 +0200, KoDer wrote: >. > > > > > > Also, are you aware that putting a zipped version of the standard library > > on sys.path already speeds up startup considerably? Python since 2.3 > > automatically includes an appropriate entry in sys.path: > > > >zipped version has one weakness - you can't put .so(or dll) files inside. >In my system 19 from 25 installed egg add directories ,not archives >(because it's contain dll ). There's something wrong there. You can put .so, .dll, or .pyd files in eggs - they get extracted to a cache directory automatically, but those directories *don't* get added to sys.path. The mere presence of such files is not enough to cause an egg to be installed unzipped. The package has to be using things like __file__, the package author has to have marked it zip-unsafe, or easy_install was invoked with a request to install unzipped. As long as none of these conditions apply, the egg should be installed zipped, with dynamic libraries automatically extracted on-demand to the $PYTHON_EGG_CACHE. >But even without egg directories >> >['', >'C:\\Python25\\Scripts', >'C:\\WINDOWS\\system32\\python25.zip', >'C:\\Python25\\DLLs', >'C:\\Python25\\lib', >'C:\\Python25\\lib\\plat-win', >. >'C:\\Python25\\lib\\site-packages\\wx-2.8-msw-unicode'] >len(sys.path) == 18 (without eggs) near 18 / 2 = 9 'file not found' errors > for every first module import. That depends on what the module is. In the path you've shown, having a python25.zip would allow only 2 failures before each stdlib import. (Note that an failed zipfile imports are roughly equivalent to a dictionary lookup in time - they don't access the filesystem once the zipfile index is loaded). >So improvement of setuptools will help, but not solve this problem . Right -- most of your problem will be solved by creating 'C:\\WINDOWS\\system32\\python25.zip', containing the contents of C:\\Python25\\lib\\. Hm. Interesting, actually. Does anybody know why it's looking for 'C:\\WINDOWS\\system32\\python25.zip'? That seems wrong to me. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Py2.6 ideas
On 2/16/07, Nick Coghlan <[EMAIL PROTECTED]> wrote:
> Martin v. Löwis wrote:
> > Raymond Hettinger schrieb:
> >> No need to go so widely off-track. The idea is to have an efficient type
> >> that
> >> is directly substitutable for tuples but is a bit more self-descriptive.
> >> I like
> >> to have the doctest result cast at NamedTuple('TestResults failed
> >> attempted).
> >> The repr of that result looks like TestResult(failed=0, attempted=15) but
> >> is
> >> still accessible as a tuple and passes easily into other functions that
> >> expect a
> >> tuple. This sort of thing would be handly for things like os.stat().
> >> http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/500261
> >
> > I'd like to repeat Guido's question: Why does this still need to support
> > the tuple interface (i.e. indexed access)?
>
> So that it remains interoperable with existing libraries that expect a
> tuple? Otherwise you'd be casting (and copying) every time you needed to
> pass it to something that used indexed access.
In the case of os.stat and friends I propose that in Py3k we drop the
tuple-ness completely; it's been dual-op since 2.2.
Maybe Raymond's proposed record type should have two versions: one
that's also a tuple, for compatibility, and one that's just a record.
The compatibility version should also support having named fields that
don't show up in the tuple view -- this has proved very useful for the
os.stat() result.
--
--Guido van Rossum (home page: http://www.python.org/~guido/)
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
> As others have reported: if you have a gtk reactor and a wx reactor to > support, in addition to a "regular" Win32 reactor (e.g. PythonWin), > your approach still won't work. They all do GetMessage behind the > scenes (sp?), yet you would run them in multiple threads, so they get > their own messages. Plus they are not thread-safe. I'm not sure what your point is here. Let me see if I can clarify things, at least from my side. I suggest it is possible to implement a PerfectReactor. I define this PerfectReactor as a singleton top-level reactor which runs in the main thread of a Python application. All incoming events would be serialized to this single reactor, which would dispatch them to their appropriate listeners. When asked to monitor disparate event systems, the PerfectReactor would act as a concentrator of multiple other reactors, with various ugly things going on hidden in the implementation. The goal would be that modular libraries of event listeners could be written in an agnostic manner, simply requesting to listen to whatever events they need to, and the main application wouldn't have to know in advance what reactors to configure or how to configure them. You seem to assert it is not possible to implement such a PerfectReactor, and list counter-examples. So far so good. If you're saying "it is impossible for any implementation to simultaneously support more than one of the Win32, gtk, and wx reactors": fine, this hypothetical PerfectReactor can't do the impossible, but that's not an argument against a PerfectReactor. If it *is* possible to support more than one of these simultaneously through precise configuration, I assert that a PerfectReactor could support that configuration too, and the goal would be to have it automatically configure itself. You point out the gtk reactor and the wx reactor must be run in the main thread, because they run their own message pumps. What I had in mind was that, when you asked to start listening to Win32 events, the Win32 reactor would tell the PerfectReactor "I must be run in the main thread", and it would (re-) configure that to be the main thread's listener; the notification mechanism for other threads saying "there's a new event in your queue" would switch to using Win32 messages. If the gtk and wx reactors had the same restriction, then whichever was first introduced to the PerfectReactor first would win, and introducing subsequent reactors would throw an exception. (Again, can't do the impossible.) However, if gtk and wx expose enough of the plumbing of their message pumps, this central Win32 message pump could identify messages intended for the gtk and wx reactor message pumps and inject them. The gtk and wx reactors wouldn't call GetMessage, but they could call the rest of their message loops. Therefore they could all be run in the main thread. (And yes, "behind the scenes" is correct.) > if the PerfectReactor is to support Tcl/Tk, it needs to run the Tcl > even loop in the Tcl thread, which is normally the main thread (in > Tkinter, it's the thread where you originally created the tkapp > object). OTOH, PerfectReactor may also need to run some Win32 event > loop in the same thread. Again, I'm not sure of your point. If you're saying "it is impossible to run multithreaded Tcl and any message pump (Win32, gtk, wx) in the same application", then fine, PerfectReactor cannot do the impossible. If it is only possible to do with clever configuration, then I maintain a PerfectReactor could self-configure to support this configuration at runtime. Cheers, /larry/ ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Py2.6 ideas
[Martin v. Löwis] >Why does this still need to support the > tuple interface (i.e. indexed access)? I used named tuples whereever I need a tuple but the number and meaning of the fields starts to tax my memory. For doctests, I return a named tuple like TestResults(failed=0, attempted=15). That named tuple can still be unpacked after a function call: f,a=results. And it can be unpacked in a function call: f(*results). It can be handed to functions that expect a tuple: 'Missed %d out of %d tests' % results. Also, the named tuple used with indexed access has the same high performance as a regular tuple; however, if an error occurs, its repr is shown in a more readable form. Likewise, when constructing the NamedTuple, an editor's tooltips reminds you of what goes in each field. Those properties have proved useful to me when modeling option contracts where each contract has to track the remaining time, interest rate, option type, underlying security, and strike price. The same applies to model results: delta, gamma, vega, theta, rho. This could also be done with attribute access, but it would be much slower and much more verbose when unpacking the model's results: d, g, v, t, r = model(somecontract) vs. m = model(somecontract) d, g, v, t, r = m.delta, m.gamma, m.vega, m.theta, m.rho > I'm not (anymore) sure that you are aware that the os.stat result *already* > has named fields, in addition to the indexed access. Of course, that specific example was solved long ago. We did not however expose a general purpose mechanism applicable where similar issues arise for other tuples. Raymond ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Lack of sequential decompression in the zipfile module
Though I am an avid Python programmer, I've never forayed into the area of developing Python itself, so I'm not exactly sure how all this works. I was confused (and somewhat disturbed) to discover recently that the zipfile module offers only one-shot decompression of files, accessible only via the read() method. It is my understanding that the module will handle files of up to 4 GB in size, and the idea of decompressing 4 GB directly into memory makes me a little queasy. Other related modules (zlib, tarfile, gzip, bzip2) all offer sequential decompression, but this does not seem to be the case for zipfile (even though the underlying zlib makes it easy to do). Since I was writing a script to work with potentially very large zipped files, I took it upon myself to write an extract() method for zipfile, which is essentially an adaption of the read() method modeled after tarfile's extract(). I feel that this is something that should really be provided in the zipfile module to make it more usable. I'm wondering if this has been discussed before, or if anyone has ever viewed this as a problem. I can post the code I wrote as a patch, though I'm not sure if my file IO handling is as robust as it needs to be for the stdlib. I'd appreciate any insight into the issue or direction on where I might proceed from here so as to fix what I see as a significant problem. Thanks, Derek ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] urllib2 EP + decr. startup time
On 16/02/07, Phillip J. Eby <[EMAIL PROTECTED]> wrote: > Hm. Interesting, actually. Does anybody know why it's looking for > 'C:\\WINDOWS\\system32\\python25.zip'? That seems wrong to me. It looks alongside python25.dll, which is installed in windows\system32 by default. If you then ask why the DLL goes in the system directory, I believe this is to help with Python COM objects, which don't otherwise have the Python directory on their PATH (and so wouldn't find python25.dll if it were there). Paul. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Lack of sequential decompression in the zipfile module
On 2/16/07, Derek Shockey <[EMAIL PROTECTED]> wrote: > Though I am an avid Python programmer, I've never forayed into the area of > developing Python itself, so I'm not exactly sure how all this works. > > I was confused (and somewhat disturbed) to discover recently that the > zipfile module offers only one-shot decompression of files, accessible only > via the read() method. It is my understanding that the module will handle > files of up to 4 GB in size, and the idea of decompressing 4 GB directly > into memory makes me a little queasy. Other related modules (zlib, tarfile, > gzip, bzip2) all offer sequential decompression, but this does not seem to > be the case for zipfile (even though the underlying zlib makes it easy to > do). > > Since I was writing a script to work with potentially very large zipped > files, I took it upon myself to write an extract() method for zipfile, which > is essentially an adaption of the read() method modeled after tarfile's > extract(). I feel that this is something that should really be provided in > the zipfile module to make it more usable. I'm wondering if this has been > discussed before, Not that I know of, but searching Google would better answer that question. > or if anyone has ever viewed this as a problem. Not that I know of. > I can post > the code I wrote as a patch, though I'm not sure if my file IO handling is > as robust as it needs to be for the stdlib. I'd appreciate any insight into > the issue or direction on where I might proceed from here so as to fix what > I see as a significant problem. > Best way is to post it as a patch to the SF tracker for Python (http://sourceforge.net/patch/?group_id=5470). Then hopefully someone will eventually get to it and have a look. Just please understand that it might be a while as it requires someone to take an interest in your patch to put the time and effort to make sure it up to including. To help your chances of getting it included, make sure you do the following: 1. Make it match PEP 7/8 style guidelines. 2. Have unit tests. 3. Have proper documentation. It is okay if it is not in LaTeX if you don't already know the language. -Brett > Thanks, > Derek > > ___ > Python-Dev mailing list > [email protected] > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/brett%40python.org > > ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Py2.6 ideas
> Maybe Raymond's proposed record type should have two versions: one
> that's also a tuple, for compatibility, and one that's just a record.
FWIW, ML unifies tuples and records by defining a tuple to be a record whose
component names are all consecutive integers starting with 1.
For example, in ML, the literal { name = "ark", state = "NJ" } represents a
record with type { name: string, state: string }. The identifiers "name"
and "state" are bound during compilation, ML being a statically typed
language.
In ML, one extracts a component named foo by applying a function named #foo.
So, for example, the value of
#state { name = "ark", state = "NJ" }
is "NJ", and trying to evaluate
#foo { name = "ark", state = "NJ" }
results in a compilation error because of type-checking failure.
Component names can be either identifiers or integers. So, for example,
{ name = "spells", 1 = "xyzzy", 2 = "plugh" }
is a record of type {1: string, 2: string, name: string }.
So here is the point. If the component names of a record are all positive
integers with no gaps, the record is *also* a tuple. So, for example
{ 2 = "plugh", 1 = "xyzzy" }
has exactly the same meaning--including the same type--as
{ "xyzzy", "plugh" }
In both cases, the compiler normalizes the display, both of the value (i.e.
it prints {"xyzzy", "plugh"} instead of { 2 = "plugh", 1 = "xyzzy" }, and it
prints the type as
string * string
instead of (the equivalent)
{ 1: string, 2: string }
So in ML, tuple types aren't really anything special -- they're just
abbreviations for elements of a particular subset of record types.
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Py2.6 ideas
Raymond Hettinger schrieb: > d, g, v, t, r = model(somecontract) I find that line quite unreadable, and find it likely that I would not be able to remember the standard order of the fields. You almost "had me" with the two fields example, but this makes me think "-1" again. Is it really that you need all these values in the following computation? For stat, this was never the case: you would only need some field normaly (especially when the more esoteric, platform-dependent fields got added). If you absolutely want tuple unpacking on a record-like object, I'd suggest to support explicit tuple conversion, like d, g, v, t, r = model(somecontract).to_tuple() Or, if you absolutely cannot stand the explicit tuple creation, add def __getitem__(self, index): return getattr(self, self.__slots__[index]) # or is it self.[self.__slots__[index]] :-? No need to inherit from tuple for that. > Of course, that specific example was solved long ago. We did not > however expose a general purpose mechanism applicable where > similar issues arise for other tuples. As you've explained now, your use case is not similar. For os.stat, it's a means of transition and backwards compatibility. For your code, it seems you want it a permanent feature in your code. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Alternative Python VM
It could be interesting. """ - pyvm is * 2 times * faster than Python 2.4. In the source code there is a collection of benchmarks which includes 65 python scripts collected from the internet. At average if Python 2.4 needs 24 hours to do some job, pyvm can do it in 12 hours. - pyvm is a virtual machine that can run Python 2.3/2.4 bytecode. There is a compiler written in python (the 'pyc' program) which is executed by the vm to compile source code to bytecode. It is very easy to do advanced optimizations in python and the compiler produces bytecode of high quality (speed and size). """ http://students.ceid.upatras.gr/~sxanth/pyvm/ Sokolov Yura (funny_falcon) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
[EMAIL PROTECTED] wrote: > When you boil it down, Twisted's event loop is just a notification for > "a connection was made", "some data was received on a connection", "a > connection was closed", and a few APIs to listen or initiate different > kinds of connections, start timed calls, and communicate with threads. > All of the platform details of how data is delivered to the > connections are abstracted away. How do you propose we would make a > less "specific" event mechanism? But that is exactly the problem I have with Twisted. For HTTP it creates its own set of notifications instead of structuring the code similar to SocketServer (UDP and TCP), BaseHTTPServer, SimpleHTTPServer etc which are well understood in the python community and e.g. used by medusa and asyncore. Having to completely restructure one's own code is a high price. Giving control away into a big framework that calls my own code for not so easy to understand reasons (for a twisted noob) does not give me a warm feeling. It's o.k. for complex applications like web servers but for small networking applications I'd like to have a chance to understand what's going on. Asyncore is so simple that it's easy to follow when I let it do the select() for me. That said, I conclude that the protocol implementations are superb but unfortunately to tightly coupled to the Twisted philosophy, sitting in the middle, trying to orchestrate instead of being easy to integrate. Joachim ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New syntax for 'dynamic' attribute access
* Ben North <[EMAIL PROTECTED]> [2007-02-11 23:45:05 +]:
> Dynamic attribute access is currently possible using the "getattr"
> and "setattr" builtins. The present PEP suggests a new syntax to
> make such access easier, allowing the coder for example to write
>
> x.('foo_%d' % n) += 1
>
> z = y.('foo_%d' % n).('bar_%s' % s)
>
> instead of
>
> attr_name = 'foo_%d' % n
> setattr(x, attr_name, getattr(x, attr_name) + 1)
>
> z = getattr(getattr(y, 'foo_%d' % n), 'bar_%s' % s)
Clipper (and other members of the "xBase" family) have something vaguely
similar, and from working with that syntax, my feeling is that this is
too "subtle" given the mixing of levels going on here (a "literal" name
vs. an expression evaluating to a name). That is, it's too easy to not
properly notice the presence / absence of the () and become confused
between foo.bar and foo.(bar) or similar.
(For the curious, the Clipper syntax is generally used in "commands"
which are a kind of statement that is macro-evaluated; so, for example,
something like this:
USE clients
will be transformed into something like the following expression:
dbUseArea("clients")
during the preprocessor pass, care of a "builtin" definition of USE;
this function has the effect of opening a database file on disk for use.
Now, in order to use the command syntax with a database name stored in a
variable, one does something like the following:
USE (dbName)
which is transformed into something like:
dbUseArea(dbname)
with similar effects as before.)
--
mithrandi, i Ainil en-Balandor, a faer Ambar
signature.asc
Description: Digital signature
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New syntax for 'dynamic' attribute access
A few of you have expressed concern about how would that look to a
newbie. Being one, this is what I think:
- The idea sounds good. Setattr and getattr seems both unpythonic and illegible.
- please.(dont_torture) =
me(with_dots,that_look,like.(function),calls). Ok, so the dot _is_
needed in order to indicate that we are talking about objects. But if
I see something like spam.(eggs) , I would feel tempted to delete the
dot thinking it's a typo.
- Square brackets have enough overloading.
- Braces feel good. I think they are the best choice of the ones
proposed. Because spam{eggs} doesn't mean anything, then there would
be no confusion with a typo in spam.{eggs}
As someone said before, the problem is that it would limit braces for
serving further purposes in the future. If the chances of requiring
using braces in the next few versions seem low to you, I would say go
for it.
Sergio
On 2/12/07, M.-A. Lemburg <[EMAIL PROTECTED]> wrote:
> On 2007-02-12 16:19, Georg Brandl wrote:
> >> Tim Delaney asked in particular:
> >>> Have you checked if [the existing uses of getattr, where "getattr" in
> >>> that scope is a function argument with default value the built-in
> >>> "getattr"] are intended to bring the "getattr" name into local scope
> >>> for fast lookup, or to force a binding to the builtin "gettattr" at
> >>> compile time (two common (ab)uses of default arguments)? If they are,
> >>> they would be better served by the new syntax.
> >> They're all in Lib/codecs.py, and are of the form:
> >>
> >> class StreamRecoder:
> >> def __getattr__(self, name,
> >> getattr=getattr):
> >>
> >> """ Inherit all other methods from the underlying stream.
> >> """
> >> return getattr(self.stream, name)
> >>
> >> Without digging deeper into that code I'm afraid I can't say precisely
> >> what is going on.
> >
> > Since that is a special method and ought to have the signature
> > __getattr__(self, name), I think it's safe to assume that that's meant
> > as an optimization.
>
> I can confirm that: it's a case of fast-local-lookup optimization.
>
> You can add a -1 from me to the list as well: I don't think that
> dynamic lookups are common enough to warrant new syntax.
>
> Even if you do add a new syntax for this, using parenthesis is
> a poor choice IMHO as the resulting code looks too much like a
> function call (e.g. "callable.(variable)").
>
> Other choices would be square brackets [], but these have the
> same problem as they are in use for indexing.
>
> The only brackets that are not yet overloaded in the context
> of applying them to an object are curly brackets, so
> "callable.{variable}" would cause enough raising eyebrows
> to not think of a typo.
>
> --
> Marc-Andre Lemburg
> eGenix.com
>
> Professional Python Services directly from the Source (#1, Feb 12 2007)
> >>> Python/Zope Consulting and Support ...http://www.egenix.com/
> >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
> >>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/
>
>
> Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free !
> ___
> Python-Dev mailing list
> [email protected]
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/sergio.correia%2Bpydev%40gmail.com
>
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Porting Python 2.4.4. and 2.5 to Sparc Solaris 9 and 10
Dear Python Developers. I posted the following to the python-help yesterday. While I received some interesting responses, they indicated that I'd already gone beyond that group's knowledge, and suggested that I mail the developers directly on this list. By way of background, I'm a retired hardware-software engineer with substantial experience in kernel programming, device drivers, porting between hardware and software architectures, so am probably more prepared than the average systems administrator to get into program details. I have downloaded the tars for Python 2.4.4 and 2.5, and built both of them on two Solaris systems, using a variety of compilers. The original objective was to get a usable Python to support a Mailman 2.1.9 installation. While it was clear that a build of 2.5, using the Sun Studio 11 development system, was incomplete and did not pass all of the regression tests, it did prove to be adequate to support Mailman. I have now come back to audit building Python on Sparc Solaris and to get some idea of what needs to be done to get a clean and reasonably complete build. My attempts to find some support through the Python web site seemed quite inconclusive. I spent part of a day trying to weave my way through the Sourceforge bug tracker, but was unable to find anything that seemed meaningful vis-a-vis what my audits were showing. Accordingly, I chose to devote some time and effort to determining where the build problems lay, and what might be needed to fix some of them. Build Platform Data: System A: Sun Ultra 60, Solaris 10 11/06, with Sun Studio 11 development system (/opt/SUNWspro). A Solaris 10 11/06 full distribution install includes gcc-3.4.3, Python 2.3.3, along with libraries and header files for such things as Tcl/Tk and openssl. System B: Sun Ultra 10, Solaris 9 9/05. Development systems are Sun Studio 11 and a new local build of gcc-4.1.1 (c and c++ only). The O/S is patched with the mid-January Sun patch cluster. Solaris 9 does not include a Python or a gcc, but does include Tcl/Tk libraries and include files. In both cases, the Studio 11 programs have been patched to current. The cc patch level is 121015-03. Both O/S installs are recent "full system" installs on bare machines, plus the development systems noted. I consider these representative of what an average systems administrator would have for resources "out of the box" for building and installing Python from source. On both systems, the non-Solaris software resides in the /usr/sfw directory tree. After having tried several configure/make/make test runs, it appears to me that there are two problem areas. 1. There is no way in the distribution setup to get the build to find the /opt/sfw include libraries for building extension modules. It appears to me that this search is done by setup.py, and that configuring the appropriate paths in the main Makefile does not propogate out to the module build. If the configure script has a way to find these libraries and to set the build up for them, I can't see what that might be. This problem affects both 2.4.4 and 2.5. I have taken the time to add the search code for the Solaris /opt/sfw directories to setup.py. There were other changes needed to existing code to get it to work properly. I include a diff between the code I am using with success and the 2.5 distribution version at the end of this note. It may not be the most appropriate for distribution, but it works on Solaris 9 and 10. I had to revise the setup.py in the 2.4.4 distribution similarly. 2. On Python 2.5, the ctypes and libffi code is not only GNU-specific, but gcc-version-specific as well. The problem with the __attribute__ specification in ffi.h seems to be well-understood, and may be amenable to some sort of workaround. However, attempting to build with the gcc-3.4.3 included with Solaris 10 failed with a linker misalignment error. 3. Regression test failures: test test_ioctl failed -- errors occurred in test.test_ioctl.IoctlTests I get this output when running make test on any build (Sun cc or any gcc) on both 2.4.4 and 2.5. Running the ioctl test standalone does not produce a failure. At this point, I wonder if I am reinventing the wheel or finding things that need to be addressed for Solaris builds. I am new to Python, and while I'd like to use it to make some local changes to Mailman, I feel concerned over reliability and robustness. Just how are you getting full Solaris builds? Hank van Cleef [EMAIL PROTECTED] The test summary and setup.py diffs follow: 280 tests OK. 1 test failed: test_ioctl 38 tests skipped: test_aepack test_al test_applesingle test_bsddb test_bsddb185 test_bsddb3 test_cd test_cl test_codecmaps_cn test_codecmaps_hk test_codecmaps_jp test_codecmaps_kr test_codecmaps_tw test_ctypes test_curses test_gdbm test_gl test_imgfile test_linuxaudiod
Re: [Python-Dev] Alternative Python VM
On 2/14/07, Sokolov Yura <[EMAIL PROTECTED]> wrote: > It could be interesting. > > """ > - pyvm is * 2 times * faster than Python 2.4. In the source code there > is a collection of benchmarks which includes 65 python scripts collected > from the internet. At average if Python 2.4 needs 24 hours to do some > job, pyvm can do it in 12 hours. > - pyvm is a virtual machine that can run Python 2.3/2.4 bytecode. > There is a compiler written in python (the 'pyc' program) which is > executed by the vm to compile source code to bytecode. It is very easy > to do advanced optimizations in python and the compiler produces > bytecode of high quality (speed and size). > """ > > http://students.ceid.upatras.gr/~sxanth/pyvm/ > I love how the first line says this project "most of the times produce the expected output". =) Until this thing can run parrotbench and the entire regression test suite for Python source modules I will consider the speed claims suspect. -Brett ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Lack of sequential decompression in the zipfile module
Derek Shockey schrieb: > Since I was writing a script to work with potentially very large zipped > files, I took it upon myself to write an extract() method for zipfile, > which is essentially an adaption of the read() method modeled after > tarfile's extract(). I feel that this is something that should really be > provided in the zipfile module to make it more usable. I'm wondering if > this has been discussed before, or if anyone has ever viewed this as a > problem. I can post the code I wrote as a patch, though I'm not sure if > my file IO handling is as robust as it needs to be for the stdlib. I'd > appreciate any insight into the issue or direction on where I might > proceed from here so as to fix what I see as a significant problem. I think something like this is patch #1121142. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Py2.6 ideas
> Raymond Hettinger schrieb:
>> d, g, v, t, r = model(somecontract)
[MvL]
> I find that line quite unreadable
Of course, I can't give you the fully spelled-out line from proprietary code.
But at this point we're just talking about the use cases for tuples with or
without named attributes. Some functions return multiple values and
some calls to those functions do tuple unpacking. That is ubiquitous
throughout Python. If the tuple also happens to be a NamedTuple,
you get tooltips for it (reminding you which fields are which) and
any error messages will show the full repr with both the names
and values.
If not unpacked, then the attribute access is helpful. Something
like contract.vega or testresult.failures or somesuch.
Essentially, I'm proposing a variant of tuple that has self-documenting extra
features:
traditional positional arguments construction or option keyword argument
construction
annotated repr:
Contract(type='put', strike=45, security='IBM', expirymonth=4)
instead of:
('put', 45, 'IBM, 4)
optional attribute access:
contract.strike
nice docstring for tooltips:
'Contract(type, strike, security, expirymonth)'
The use cases are the same as the ones for tuples. The new type is
just more self-documenting. That's all there is to it.
FWIW, I've been using NamedTuples for at least six months and
have found them to be a nice improvement over straight-tuples
in situations where I can't easily remember what each tuple
position represents. If added to the collections module,
I think NamedTuples will become quite popular.
> If you absolutely want tuple unpacking on a record-like object,
> I'd suggest to support explicit tuple conversion, like
>
> d, g, v, t, r = model(somecontract).to_tuple()
Entirely unnecessary. The goal is to have better tuples
with low overhead and a near zero learning curve.
Raymond
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Py2.6 ideas
"Raymond Hettinger" <[EMAIL PROTECTED]> wrote: > > Raymond Hettinger schrieb: > >> d, g, v, t, r = model(somecontract) > > [MvL] > > I find that line quite unreadable > > Of course, I can't give you the fully spelled-out line from proprietary code. > But at this point we're just talking about the use cases for tuples with or > without named attributes. Some functions return multiple values and > some calls to those functions do tuple unpacking. That is ubiquitous > throughout Python. If the tuple also happens to be a NamedTuple, > you get tooltips for it (reminding you which fields are which) and > any error messages will show the full repr with both the names > and values. > > If not unpacked, then the attribute access is helpful. Something > like contract.vega or testresult.failures or somesuch. For what it's worth, I've actually been using a similar approach with lists and global names of list indices because I needed a mutable structure, the list instance was significantly smaller than an object with __slots__ (by a factor of 3), and because using global constants was actually competitive with a __slots__ name lookup. After having seen your tuple recipe, I've been planning on converting it to a list-based recipe for the same benefits (except for unpacking) in my own code. Then again, I'm also looking forward to adding the tuple-based recipe to my own library for all of the reasons you outlined. - Josiah ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Recent experience with the _ast module
On 2/14/07, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote:
> Collin Winter schrieb:
> > What's inconsistent about it? That classes are being used for the
> > _ast.{Add,Sub,Mult,etc} names?
>
> Exactly. These aren't names - they are nodes in the tree. All nodes
> are instances of _ast.AST.
>
> > I don't see the need for both _ast.Add and _ast.Add.singleton or
> > _ast.add or however else it might be spelled. I'd be perfectly happy
> > doing something like "_ast.Add = object()" (when initializing the _ast
> > module), so long as I can write "node.op is _ast.Add", "node.op ==
> > _ast.Add", or something equally brief and to-the-point.
>
> Would you like to do the same for Pass, Break, Continue, and Ellipsis?
>
> They are also "just names". If you make _ast.Add the entry in the
> tree itself, why not _ast.Break? Or, if you have a way to deal with
> _ast.Break, why can't the same way work for _ast.Add?
But Pass, Break, Continue and Ellipsis aren't in the same category as
Add, Mult, Div, etc.. The former stand alone, while the only time
you'll ever encounter Add, Mult, Mod and co. is in the context of a
BoolOp, BinOp or whatever. I could of course add separate visitor
methods for each op type to my code generator, but that's a lot of
boilerplate code for very little gain. I could use a dispatch table in
visitBoolOp, and dispatch like "dispatch[type(node.op)]", but why? In
this scenario, node.op doesn't convey any information other than its
type. Add, Mult, etc are just values in an enumeration, so why not
treat them that way and be able to write "dispatch[node.op]".
So: what if _ast.Add and co. were singleton instances of _ast.AST or
even of a subclass of AST, like _ast.Op? That way, we keep the
consistency that all AST nodes are instances of _ast.AST and I get the
nice property I'm looking for, ie, "node.op is _ast.Mult".
Collin Winter
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Recent experience with the _ast module
On 2/16/07, Collin Winter <[EMAIL PROTECTED]> wrote:
> On 2/14/07, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote:
> > Collin Winter schrieb:
> > > What's inconsistent about it? That classes are being used for the
> > > _ast.{Add,Sub,Mult,etc} names?
> >
> > Exactly. These aren't names - they are nodes in the tree. All nodes
> > are instances of _ast.AST.
> >
> > > I don't see the need for both _ast.Add and _ast.Add.singleton or
> > > _ast.add or however else it might be spelled. I'd be perfectly happy
> > > doing something like "_ast.Add = object()" (when initializing the _ast
> > > module), so long as I can write "node.op is _ast.Add", "node.op ==
> > > _ast.Add", or something equally brief and to-the-point.
> >
> > Would you like to do the same for Pass, Break, Continue, and Ellipsis?
> >
> > They are also "just names". If you make _ast.Add the entry in the
> > tree itself, why not _ast.Break? Or, if you have a way to deal with
> > _ast.Break, why can't the same way work for _ast.Add?
>
> But Pass, Break, Continue and Ellipsis aren't in the same category as
> Add, Mult, Div, etc.. The former stand alone, while the only time
> you'll ever encounter Add, Mult, Mod and co. is in the context of a
> BoolOp, BinOp or whatever. I could of course add separate visitor
> methods for each op type to my code generator, but that's a lot of
> boilerplate code for very little gain. I could use a dispatch table in
> visitBoolOp, and dispatch like "dispatch[type(node.op)]", but why? In
> this scenario, node.op doesn't convey any information other than its
> type. Add, Mult, etc are just values in an enumeration, so why not
> treat them that way and be able to write "dispatch[node.op]".
>
> So: what if _ast.Add and co. were singleton instances of _ast.AST or
> even of a subclass of AST, like _ast.Op? That way, we keep the
> consistency that all AST nodes are instances of _ast.AST and I get the
> nice property I'm looking for, ie, "node.op is _ast.Mult".
>
If you look back in the history of this thread I think Martin
suggested this: "I'd rather prefer
if the singletons where exposed under a name, e.g. _ast.Add.singleton,
or _ast.add (if that doesn't cause conflicts)."
Since the both of you have proposed this I think this is the best way
to move forward.
-Brett
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Recent experience with the _ast module
On 2/16/07, Brett Cannon <[EMAIL PROTECTED]> wrote:
> On 2/16/07, Collin Winter <[EMAIL PROTECTED]> wrote:
> > On 2/14/07, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote:
> > > Collin Winter schrieb:
> > > > What's inconsistent about it? That classes are being used for the
> > > > _ast.{Add,Sub,Mult,etc} names?
> > >
> > > Exactly. These aren't names - they are nodes in the tree. All nodes
> > > are instances of _ast.AST.
> > >
> > > > I don't see the need for both _ast.Add and _ast.Add.singleton or
> > > > _ast.add or however else it might be spelled. I'd be perfectly happy
> > > > doing something like "_ast.Add = object()" (when initializing the _ast
> > > > module), so long as I can write "node.op is _ast.Add", "node.op ==
> > > > _ast.Add", or something equally brief and to-the-point.
> > >
> > > Would you like to do the same for Pass, Break, Continue, and Ellipsis?
> > >
> > > They are also "just names". If you make _ast.Add the entry in the
> > > tree itself, why not _ast.Break? Or, if you have a way to deal with
> > > _ast.Break, why can't the same way work for _ast.Add?
> >
> > But Pass, Break, Continue and Ellipsis aren't in the same category as
> > Add, Mult, Div, etc.. The former stand alone, while the only time
> > you'll ever encounter Add, Mult, Mod and co. is in the context of a
> > BoolOp, BinOp or whatever. I could of course add separate visitor
> > methods for each op type to my code generator, but that's a lot of
> > boilerplate code for very little gain. I could use a dispatch table in
> > visitBoolOp, and dispatch like "dispatch[type(node.op)]", but why? In
> > this scenario, node.op doesn't convey any information other than its
> > type. Add, Mult, etc are just values in an enumeration, so why not
> > treat them that way and be able to write "dispatch[node.op]".
> >
> > So: what if _ast.Add and co. were singleton instances of _ast.AST or
> > even of a subclass of AST, like _ast.Op? That way, we keep the
> > consistency that all AST nodes are instances of _ast.AST and I get the
> > nice property I'm looking for, ie, "node.op is _ast.Mult".
> >
>
> If you look back in the history of this thread I think Martin
> suggested this: "I'd rather prefer
> if the singletons where exposed under a name, e.g. _ast.Add.singleton,
> or _ast.add (if that doesn't cause conflicts)."
>
> Since the both of you have proposed this I think this is the best way
> to move forward.
That's what I get for letting this thread drop for a few days.
I read Martin's original proposal as wanting to have _ast.add
(singleton instance) *in addition to* _ast.Add (presumably the class
of _ast.add). To be clear, I want to replace the current _ast.Add (a
class, subclassing AST) with an instance of _ast.Op or something
comparable; making _ast.{Load,Store,Del,AugLoad,AugStore,Param} into
instances of an _ast.Context class would be a part of this.
Collin Winter
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
On 16 Feb, 06:30 pm, [EMAIL PROTECTED] wrote: >I suggest it is possible to implement a PerfectReactor. I don't think this constitutes a sufficient existence proof. Perhaps you could write a prototype? There are a bunch of existing reactors you could either import or copy/paste from to bootstrap it, so you wouldn't necessarily need to do many of the platform-specific hacks it has already implemented for those loops. Personally I am highly skeptical that this is possible, and I didn't find any of your arguments convincing, but I would be ecstatic if it worked; it isn't *fun* maintaining 12 or so subtly incompatible implementations of the same interface :). I also think this discussion would be more appropriate to continue on the Twisted list, as we have veered pretty far afield of stdlib enhancements and are now talking about multiplexing method implementations, but I'll follow it wherever it continues. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Lack of sequential decompression in the zipfile module
Derek Shockey gmail.com> writes: > > Though I am an avid Python programmer, I've never forayed into the area of developing Python itself, so I'm not exactly sure how all this works.I was confused (and somewhat disturbed) to discover recently that the zipfile module offers only one-shot decompression of files, accessible only via the read() method. It is my understanding that the module will handle files of up to 4 GB in size, and the idea of decompressing 4 GB directly into memory makes me a little queasy. Other related modules (zlib, tarfile, gzip, bzip2) all offer sequential decompression, but this does not seem to be the case for zipfile (even though the underlying zlib makes it easy to do). > Since I was writing a script to work with potentially very large zipped files, I took it upon myself to write an extract() method for zipfile, which is essentially an adaption of the read() method modeled after tarfile's extract(). I feel that this is something that should really be provided in the zipfile module to make it more usable. I'm wondering if this has been discussed before, or if anyone has ever viewed this as a problem. I can post the code I wrote as a patch, though I'm not sure if my file IO handling is as robust as it needs to be for the stdlib. I'd appreciate any insight into the issue or direction on where I might proceed from here so as to fix what I see as a significant problem. This is definitely a significant problem. We had to face it at work, and at the end we decided to use zipstream (http://www.doxdesk.com/software/py/zipstream.html) instead of zipfile, but of course having the functionality in the standard library would be much better. Michele Simionato ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
