Re: [Python-Dev] Path object design
Martin:
It still should be possible to come up with examples for these as
well, no? For example, if you pass a relative URI as the base
URI, what would you like to see happen?
Until two days ago I didn't even realize that was an incorrect
use of urljoin. I can't be the only one. Hence, raise an
exception - just like 4Suite's Uri.py does.
That's true. Actually, it's probably not true; it will only get fixed
if some volunteer contributes a fix.
And it's not I. A true fix is a lot of work. I would rather use Uri.py,
now that I see it handles everything I care about, and then some.
Eg, file name - URI conversion.
So do you think this patch meets your requirements?
# new
uriparse.urljoin(http://spam/;, foo/bar)
'http://spam//foo/bar'
# existing
urlparse.urljoin(http://spam/;, foo/bar)
'http://spam/foo/bar'
No. That was the first thing I tried. Also found
urlparse.urljoin(http://blah;, /spam/)
'http://blah/spam/'
uriparse.urljoin(http://blah;, /spam/)
'http://blah/spam'
I reported these on the patch page. Nothing else strange
came up, but I did only try http urls and not the others.
My requirements, meaning my vague, spur-of-the-moment thoughts
without any research or experimentation to determing their validity,
are different than those for Python.
My real requirements are met by the existing code.
My imagined ones include support for edge cases, the idna
codec, unicode, and real-world use on a variety of OSes.
4Suite's Uri.py seems to have this. Eg, lots of edge-case
code like
# On Windows, ensure that '|', not ':', is used in a drivespec.
if os.name == 'nt' and scheme == 'file':
path = path.replace(':','|',1)
Hence the uriparse.py patch does not meet my hypothetical
requirements .
Python's requirements are probably to get closer to the spec.
In which case yes, it's at least as good as and likely generally
better than the existing module, modulo a few API naming debates
and perhaps some rough edges which will be found when put into use.
And perhaps various arguments about how bug compatible it should be
and if the old code should be available as well as the new one,
for those who depend on the existing 1808-allowed implementation
dependent behavior.
For those I have not the experience to guide me and no care to push
the debate. I've decided I'm going to experiment using 4Suite's Uri.py
for my code because it handles things I want which are outside of
the scope of uriparse.py
This topic (URL parsing) is not only inherently difficult to
implement, it is just as tedious to review. Without anybody
reviewing the contributed code, it's certain that it will never
be incorporated.
I have a different opinion.
Python's url manipulation code is a mess. urlparse, urllib, urllib2.
Why is urlencode part of urllib and not urllib2? For that matter,
urllib is labeled 'Open an arbitrary URL' and not 'and also do
manipulations on parts of URLs.
I don't want to start fixing code because doing it the way I want to
requires a new API and a much better understanding of the RFCs
than I care about, especially since 4Suite and others have already
done this.
Hence I would say to just grab their library. And perhaps update the
naming scheme.
Also, urlgrabber and pycURL are better for downloading arbitrary
URIs. For some definitions of better.
Andrew
[EMAIL PROTECTED]
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importing .pyc in -O mode and vice versa
Why not only import *.pyc files and no longer use *.pyo files. It is simpler to have one compiled python file extension. PYC files can contain optimized python byte code and normal byte code. -- bye by Wolfgang ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importing .pyc in -O mode and vice versa
Hi Martin, On Sat, Nov 04, 2006 at 04:47:37PM +0100, Martin v. L?wis wrote: Patch #1346572 proposes to also search for .pyc when OptimizeFlag is set, and for .pyo when it is not set. The author argues this is for consistency, as the zipimporter already does that. My strong opinion on the matter is that importing a .pyc file if the .py file is not present is wrong in the first place. It caused many headaches in several projects I worked on. Additionally trying to importing .pyo files looks like a complete semantic non-sense, but I can't really argue from experience, as I never run python -O. Typical example: someone in the project removes a .py file, and checks in this change; someone else does an 'svn up', which kills the .py in his working copy, but not the .pyc. These stale .pyc's cause pain, e.g. by shadowing the real module (further down sys.path), or simply by preventing the project's developers from realizing that they forgot to fix some imports. We regularly had obscure problems that went away as soon as we deleted all .pyc files around, but I cannot comment more on that because we never really investigated. I know it's a discussion that comes up and dies out regularly. My two cents is that it would be saner to have two separate concepts: cache files used internally by the interpreter for speed reasons only, and bytecode files that can be shipped and imported. This could e.g. be done with different file extensions (then you just rename the files if you want to ship them as bytecode without source), or with a temporary cache directory (from where you can fish bytecode files if you want to ship them). Experience suggests I should not be holding my breath until something is decided about this, though. If I were asked to come up with a patch I'd simply propose one that removes importing of stale .pyc files (I'm always running a version of Python with such a patch, to avoid the above-mentioned troubles). A bientot, Armin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Path object design
Martin v. Löwis wrote: Andrew Dalke schrieb: urlparse.urljoin(http://blah.com/;, ..) 'http://blah.com/' urlparse.urljoin(http://blah.com/;, ../) 'http://blah.com/../' urlparse.urljoin(http://blah.com/;, ../..) 'http://blah.com/' Does the result make sense to you? Does it make sense that the last of these is shorter than the middle one? It sure doesn't to me. I thought it was obvious that there was an error; That wasn't obvious at all to me. Now looking at the examples, I agree there is an error. The middle one is incorrect; urlparse.urljoin(http://blah.com/;, ../) should also give 'http://blah.com/'. make that: could also give 'http://blah.com/'. as I said, today's urljoin doesn't guarantee that the output is the *shortest* possible way to represent the resulting URI. /F ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Path object design
Andrew: urlparse.urljoin(http://blah.com/;, ..) 'http://blah.com/' urlparse.urljoin(http://blah.com/;, ../) 'http://blah.com/../' urlparse.urljoin(http://blah.com/;, ../..) 'http://blah.com/' /F: as I said, today's urljoin doesn't guarantee that the output is the *shortest* possible way to represent the resulting URI. I didn't think anyone was making that claim. The module claims RFC 1808 compliance. From the docstring: DESCRIPTION See RFC 1808: Relative Uniform Resource Locators, by R. Fielding, UC Irvine, June 1995. Now quoting from RFC 1808: 5.2. Abnormal Examples Although the following abnormal examples are unlikely to occur in normal practice, all URL parsers should be capable of resolving them consistently. Each example uses the same base as above. An empty reference resolves to the complete base URL: = URL:http://a/b/c/d;p?q#f Parsers must be careful in handling the case where there are more relative path .. segments than there are hierarchical levels in the base URL's path. My claim is that consistent implies in the spirit of the rest of the RFC and to a human trying to make sense of the results and not only mean does the same thing each time. Else urljoin(http://blah.com/;, ../../..) 'http://blah.com/there/were/too/many/dot-dot/path/elements/in/the/relative/url' would be equally consistent. for rel in .. ../ ../.. ../../ ../../.. ../../../ ../../../...split(): ... print repr(rel), repr(urlparse.urljoin(http://blah.com/;, rel)) ... '..' 'http://blah.com/' '../' 'http://blah.com/../' '../..' 'http://blah.com/' '../../' 'http://blah.com/../../' '../../..' 'http://blah.com/../' '../../../' 'http://blah.com/../../../' '../../../..' 'http://blah.com/../../' I grant there is a consistency there. It's not one most would have predicted beforehand. Then again, should is that wishy-washy unless you've got a good reason to do it a different way sort of constraint. Andrew [EMAIL PROTECTED] ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] __dir__, part 2
so, if you remember, i suggested adding __dir__ to objects, so as to make dir() customizable, remove the deprecated __methods__ and __members__, and make it symmetrical to other built-in functions. you can see the original post here: http://mail.python.org/pipermail/python-dev/2006-July/067095.html which was generally accepted by the forum: http://mail.python.org/pipermail/python-dev/2006-July/067139.html so i went on, now that i have some spare time, to research the issue. the current dir() works as follows: (*) builtin_dir calls PyObject_Dir to do the trick (*) if the object is NULL (dir with no argument), return the frame's locals (*) if the object is a *module*, we're just using it's __dict__ (*) if the object is a *type*, we're using it's __dict__ and __bases__, but not __class__ (so as not to show the metaclass) (*) otherwise, it's a normal object, so we take it's __dict__, along with __methods__, __members__, and dir(__class__) (*) create a list of keys from the dict, sort, return we'll have to change that if we were to introduce __dir__. my design is: (*) builtin_dir, if called without an argument, returns the frame's locals (*) otherwise, it calls PyObject_Dir(self), which would dispatch self.__dir__() (*) if `self` doesn't have __dir__, default to object.__dir__(self) (*) the default object.__dir__ implementation would do the same as today: collect __dict__, __members__, __methods__, and dir(__class__). by py3k, we'll remove looking into __methods__ and __members__. (*) type objects and module objects would implement __dir__ to their liking (as PyObject_Dir does today) (*) builtin_dir would take care of sorting the list returned by PyObject_Dir so first i'd want you people to react on my design, maybe you'd find flaws whatever. also, should this become a PEP? and last, how do i add a new method slot? does it mean i need to change all type-object definitions throughout the codebase? do i add it to some protocol? or directly to the object protocol? -tomer ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Feature Request: Py_NewInterpreter to create separate GIL (branch)
Talin [EMAIL PROTECTED] wrote: Guido van Rossum wrote: I don't know how you define simple. In order to be able to have separate GILs you have to remove *all* sharing of objects between interpreters. And all other data structures, too. It would probably kill performance too, because currently obmalloc relies on the GIL. Nitpick: You have to remove all sharing of *mutable* objects. One day, when we get pure GC with no refcounting, that will be a meaningful distinction. :) Python already grew that feature a couple years back, but it never became mainline. Search google (I don't know the magic incantation off the top of my head), buf if I remember correctly, it wasn't a significant win if any at all. - Josiah ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importing .pyc in -O mode and vice versa
Armin Rigo [EMAIL PROTECTED] wrote: Hi Martin, On Sat, Nov 04, 2006 at 04:47:37PM +0100, Martin v. L?wis wrote: Patch #1346572 proposes to also search for .pyc when OptimizeFlag is set, and for .pyo when it is not set. The author argues this is for consistency, as the zipimporter already does that. My strong opinion on the matter is that importing a .pyc file if the .py file is not present is wrong in the first place. It caused many headaches in several projects I worked on. Typical example: someone in the project removes a .py file, and checks in this change; someone else does an 'svn up', which kills the .py in his working copy, but not the .pyc. These stale .pyc's cause pain, e.g. by shadowing the real module (further down sys.path), or simply by preventing the project's developers from realizing that they forgot to fix some imports. We regularly had obscure problems that went away as soon as we deleted all .pyc files around, but I cannot comment more on that because we never really investigated. I had a very similar problem the other week when mucking about with a patch to ntpath . I had it in a somewhat small temporary projects folder and needed to run another project. It picked up the local ntpath.py when importing path.py, but then failed because I was working on a 2.5 derived ntpath, but I was using 2.3 to run the other project. After renaming the local ntpath, I continued to get the error until I realized damn pyc and was halfway through a filsystem wide search for the problem code (10 minutes elapsed). About the only place where I have found the need for pyc-without-py importing is for zipimports, specifically as used by py2exe and other freezing applications. I don't know if we want to add a new command line option, or a __future__ import, or something, but I think there should be some method of warning people that an import was performed without source code. - Josiah ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Path object design
Andrew Dalke schrieb: Hence I would say to just grab their library. And perhaps update the naming scheme. Unfortunately, this is not an option. *You* can just grab their library; the Python distribution can't. Doing so would mean to fork, and history tells that forks cause problems in the long run. OTOH, if the 4Suite people would contribute the library, integrating it would be an option. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importing .pyc in -O mode and vice versa
Armin Rigo schrieb: My strong opinion on the matter is that importing a .pyc file if the .py file is not present is wrong in the first place. There is, of course, an important use case (which you are addressing with a different approach): people want to ship only byte code, not source code, because they feel it protects their IP better, and also for space reasons. So outright ignoring pyc files is not really an option. I know it's a discussion that comes up and dies out regularly. My two cents is that it would be saner to have two separate concepts: cache files used internally by the interpreter for speed reasons only, and bytecode files that can be shipped and imported. There once was a PEP to better control byte code file generation; it died because it wasn't implemented. I don't think there is a strong opposition to changing the status quo - it's just that you need a well-designed specification before you start, a serious, all-singing-all-dancing implementation, and a lot of test cases. I believe it is these constraints which have prevented any progress here. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Path object design
Fredrik Lundh schrieb: urlparse.urljoin(http://blah.com/;, ../) should also give 'http://blah.com/'. make that: could also give 'http://blah.com/'. How so? If that would implement RFC 3986, you can get only a single outcome, if urljoin is meant to implement section 5 of that RFC. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importing .pyc in -O mode and vice versa
Wolfgang Langner schrieb: Why not only import *.pyc files and no longer use *.pyo files. It is simpler to have one compiled python file extension. PYC files can contain optimized python byte code and normal byte code. So what would you do with the -O option of the interpreter? Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importing .pyc in -O mode and vice versa
Armin Rigo wrote:
Typical example: someone in the project removes a .py file, and checks
in this change; someone else does an 'svn up', which kills the .py in
his working copy, but not the .pyc. These stale .pyc's cause pain,
e.g.
by shadowing the real module (further down sys.path), or simply by
preventing the project's developers from realizing that they forgot to
fix some imports. We regularly had obscure problems that went away as
soon as we deleted all .pyc files around, but I cannot comment more on
that because we never really investigated.
This is exactly why I always use this module:
== nobarepyc.py
#!/usr/bin/env python
#-*- coding: utf-8 -*-
import ihooks
import os
class _NoBarePycHooks(ihooks.Hooks):
def load_compiled(self, name, filename, *args, **kwargs):
sourcefn = os.path.splitext(filename)[0] + .py
if not os.path.isfile(sourcefn):
raise ImportError('forbidden import of bare .pyc file: %r' %
filename)
return ihooks.Hooks.load_compiled(name, filename, *args, **kwargs)
ihooks.ModuleImporter(ihooks.ModuleLoader(_NoBarePycHooks())).install()
== /nobarepyc.py
Just import it before importing anything else (or in site.py if you prefer)
and you'll be done.
Ah, it doesn't work with zipimports...
--
Giovanni Bajo
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Path object design
Fredrik Lundh wrote: Andrew Dalke wrote: as I said, today's urljoin doesn't guarantee that the output is the *shortest* possible way to represent the resulting URI. I didn't think anyone was making that claim. The module claims RFC 1808 compliance. From the docstring: DESCRIPTION See RFC 1808: Relative Uniform Resource Locators, by R. Fielding, UC Irvine, June 1995. Now quoting from RFC 1808: 5.2. Abnormal Examples Although the following abnormal examples are unlikely to occur in normal practice, all URL parsers should be capable of resolving them consistently. My claim is that consistent implies in the spirit of the rest of the RFC and to a human trying to make sense of the results and not only mean does the same thing each time. Else urljoin(http://blah.com/;, ../../..) 'http://blah.com/there/were/too/many/dot-dot/path/elements/in/the/relative/url' would be equally consistent. perhaps, but such an urljoin wouldn't pass the minimize(base + relative) == minimize(urljoin(base, relative)) test that today's urljoin passes (where minimize is defined as create the shortest possible URI that identifies the same target, according to the relevant RFC). isn't the real issue in this subthread whether urljoin should be expected to pass the minimize(base + relative) == urljoin(base, relative) test? I should hope that *is* the issue, and I should further hope that the general wish would be for it to pass that test. Of course web systems have been riddled with canonicalization errors in the past, so it'd be best if you and/or Andrew could provide a minimize() implementation :-) regards Steve -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Skype: holdenweb http://holdenweb.blogspot.com Recent Ramblings http://del.icio.us/steve.holden ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importing .pyc in -O mode and vice versa
Martin v. Löwis wrote: Why not only import *.pyc files and no longer use *.pyo files. It is simpler to have one compiled python file extension. PYC files can contain optimized python byte code and normal byte code. So what would you do with the -O option of the interpreter? I just had an idea: we could have only pyc files, and *no* way to identify whether specific optimizations (-O, -OO --only-strip-docstrings, whatever) were performed on them or not. So, if you regularly run different python applications with different optimization settings, you'll end up with .pyc files containing bytecode that was generated with mixed optimization settings. It doesn't really matter in most cases, after all. Then, we add a single command line option (eg: -I) which is: ignore *every* .pyc file out there, and regenerate them as needed. So, the few times that you really care that a certain application is run with a specific setting, you can use python -I -OO app.py. And that's all. -- Giovanni Bajo ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importing .pyc in -O mode and vice versa
On 11/6/06, Giovanni Bajo [EMAIL PROTECTED] wrote: Martin v. Löwis wrote: Why not only import *.pyc files and no longer use *.pyo files. It is simpler to have one compiled python file extension. PYC files can contain optimized python byte code and normal byte code. So what would you do with the -O option of the interpreter?I just had an idea: we could have only pyc files, and *no* way to identifywhether specific optimizations (-O, -OO --only-strip-docstrings, whatever) were performed on them or not. So, if you regularly run different pythonapplications with different optimization settings, you'll end up with .pycfiles containing bytecode that was generated with mixed optimization settings. It doesn't really matter in most cases, after all.I don't know about that. If you suspected that a failure could be because of some bytecode optimization you were trying wouldn't you like to be able to tell easily that fact? Granted our situation is not as bad as gcc in terms the impact of having to regenerate a compiled version, but it still would be nice to be able to make sure that every .pyc file is the same. We would need to make it easy to blast out every .pyc file found if we did allow mixing of optimizations (as you suggest below). Then, we add a single command line option (eg: -I) which is: ignore *every* .pyc file out there, and regenerate them as needed. So, the fewtimes that you really care that a certain application is run with a specificsetting, you can use python -I -OO app.py. That might work.-Brett ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __dir__, part 2
cool. first time i build the entire interpreter, 'twas fun :) currently i retained support for __members__ and __methods__, so it doesn't break anything and is compatible with 2.6. i really hope it will be included in 2.6 as today i'm using ugly hacks in RPyC to make remote objects appear like local ones. having __dir__ solves all of my problems. besides, it makes a lot of sense of define __dir__ for classes that define __getattr__. i don't think it should be pushed back to py3k. here's the patch: http://sourceforge.net/tracker/index.php?func=detailaid=1591665group_id=5470atid=305470 here's a demo: class foo(object): ... def __dir__(self): ... return [kan, ga, roo] ... f = foo() f __main__.foo object at 0x00A90C78 dir() ['__builtins__', '__doc__', '__name__', 'f', 'foo'] dir(f) ['ga', 'kan', 'roo'] dir(foo) ['__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__getattribute__ ', '__hash__', '__init__', '__module__', '__new__', '__reduce__', '__reduce_ex__ ', '__repr__', '__setattr__', '__str__', '__weakref__'] class bar(object): ... __members__ = [bow, wow] ... b=bar() dir(b) ['__class__', '__delattr__', '__dict__', '__doc__', '__getattribute__', '__hash_ _', '__init__', '__members__', '__module__', '__new__', '__reduce__', '__reduce_ ex__', '__repr__', '__setattr__', '__str__', '__weakref__', 'bow', 'wow'] -tomer On 11/6/06, Guido van Rossum [EMAIL PROTECTED] wrote: Sounds like a good plan, though I'm not sure if it's worth doing in 2.6 -- I'd be happy with doing this just in 3k. I'm not sure what you mean by adding a method slot -- certainly it's possible to define a method __foo__ and call it directly without having a special tp_foo in the type object, and I recommend doing it that way since the tp_foo slots are just there to make things fast; in this case I don't see a need for dir() to be fast. --Guido On 11/6/06, tomer filiba [EMAIL PROTECTED] wrote: so, if you remember, i suggested adding __dir__ to objects, so as to make dir() customizable, remove the deprecated __methods__ and __members__, and make it symmetrical to other built-in functions. you can see the original post here: http://mail.python.org/pipermail/python-dev/2006-July/067095.html which was generally accepted by the forum: http://mail.python.org/pipermail/python-dev/2006-July/067139.html so i went on, now that i have some spare time, to research the issue. the current dir() works as follows: (*) builtin_dir calls PyObject_Dir to do the trick (*) if the object is NULL (dir with no argument), return the frame's locals (*) if the object is a *module*, we're just using it's __dict__ (*) if the object is a *type*, we're using it's __dict__ and __bases__, but not __class__ (so as not to show the metaclass) (*) otherwise, it's a normal object, so we take it's __dict__, along with __methods__, __members__, and dir(__class__) (*) create a list of keys from the dict, sort, return we'll have to change that if we were to introduce __dir__. my design is: (*) builtin_dir, if called without an argument, returns the frame's locals (*) otherwise, it calls PyObject_Dir(self), which would dispatch self.__dir__() (*) if `self` doesn't have __dir__, default to object.__dir__(self) (*) the default object.__dir__ implementation would do the same as today: collect __dict__, __members__, __methods__, and dir(__class__). by py3k, we'll remove looking into __methods__ and __members__. (*) type objects and module objects would implement __dir__ to their liking (as PyObject_Dir does today) (*) builtin_dir would take care of sorting the list returned by PyObject_Dir so first i'd want you people to react on my design, maybe you'd find flaws whatever. also, should this become a PEP? and last, how do i add a new method slot? does it mean i need to change all type-object definitions throughout the codebase? do i add it to some protocol? or directly to the object protocol? -tomer ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __dir__, part 2
tomer filiba wrote: cool. first time i build the entire interpreter, 'twas fun :) currently i retained support for __members__ and __methods__, so it doesn't break anything and is compatible with 2.6. i really hope it will be included in 2.6 as today i'm using ugly hacks in RPyC to make remote objects appear like local ones. having __dir__ solves all of my problems. besides, it makes a lot of sense of define __dir__ for classes that define __getattr__. i don't think it should be pushed back to py3k. here's the patch: http://sourceforge.net/tracker/index.php?func=detailaid=1591665group_id=5470atid=305470 As I noted on the tracker, PyObject_Dir is a public C API function, so it's behaviour needs to be preserved as well as the behaviour of calling dir() from Python code. So the final form of the patch will likely need to include stronger tests for that section of the API, as well as updating the documentation in various places (the dir and PyObject_Dir documentation, obviously, but also the list of magic methods in the language reference). +1 on targeting 2.6, too. Cheers, Nick. -- Nick Coghlan | [EMAIL PROTECTED] | Brisbane, Australia --- http://www.boredomandlaziness.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importing .pyc in -O mode and vice versa
Martin v. Löwis wrote: A stat call will not only look at the directory entry, but also look at the inode. This will require another disk access, as the inode is at a different location of the disk. That should be in favour of the directory-reading approach, since e.g. to find out which if any of x.py/x.pyc/x.pyo exists, you only need to look for the names. It depends on the file system you are using. An NTFS directory lookup is a B-Tree search; ... Yes, I know that some file systems are smarter; MacOS HFS is another one that uses b-trees. However it still seems to me that looking up a path in a file system is a much heavier operation than looking up a Python dict, even if everything is in memory. You have to parse the path, and look up each component separately in a different directory tree or whatever. The way I envisage it, you would read all the directories and build a single dictionary mapping fully-qualified module names to pathnames. Any given import then takes at most one dict lookup and one access of a known-to-exist file. For a large directory, the cost of reading in the entire directory might be higher than the savings gained from not having to search it. Possibly. I guess we'd need some timings to assess the meaning of large. Also, if we do our own directory caching, the question is when to invalidate the cache. I think I'd be happy with having to do that explicitly. I expect the vast majority of Python programs don't need to track changes to the set of importable modules during execution. The exceptions would be things like IDEs, and they could do a cache flush before reloading a module, etc. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importing .pyc in -O mode and vice versa
On Tue, 07 Nov 2006 12:20:00 +1300, Greg Ewing [EMAIL PROTECTED] wrote: I think I'd be happy with having to do that explicitly. I expect the vast majority of Python programs don't need to track changes to the set of importable modules during execution. The exceptions would be things like IDEs, and they could do a cache flush before reloading a module, etc. Another questionable optimization which changes application- level semantics. No, please? Jean-Paul ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importing .pyc in -O mode and vice versa
Greg Ewing schrieb: I think I'd be happy with having to do that explicitly. I expect the vast majority of Python programs don't need to track changes to the set of importable modules during execution. The exceptions would be things like IDEs, and they could do a cache flush before reloading a module, etc. That would be a change in behavior, of course. Currently, you can put a file on disk and import it immediately; that will stop working. I'm pretty sure that there are a number of applications that rely on this specific detail of the current implementation (and not only IDEs). It still might be worthwhile to make such a change, but I'd like to see practical advantages demonstrated first. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importing .pyc in -O mode and vice versa
Martin v. Löwis wrote: Greg Ewing schrieb: I think I'd be happy with having to do that explicitly. I expect the vast majority of Python programs don't need to track changes to the set of importable modules during execution. The exceptions would be things like IDEs, and they could do a cache flush before reloading a module, etc. That would be a change in behavior, of course. Currently, you can put a file on disk and import it immediately; that will stop working. I'm pretty sure that there are a number of applications that rely on this specific detail of the current implementation (and not only IDEs). Would it be reasonable to always do a stat() on the directory, reloading if there's been a change? Would this be reliable across platforms? Tim Delaney ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] valgrind
Hi! I've embedded python into my application. Using valgrind I got a lot of errors. I understand that Conditional jump or move depends on uninitialised value(s) errors are completely ok (from Misc/README.valgrind). However, I don't understand why Invalid read's are legal, like this: ==21737== Invalid read of size 4 ==21737==at 0x408DDDF: PyObject_Free (in /usr/lib/libpython2.4.so.1.0) ==21737==by 0x4096F67: (within /usr/lib/libpython2.4.so.1.0) ==21737==by 0x408A5AC: PyCFunction_Call (in /usr/lib/libpython2.4.so.1.0) ==21737==by 0x40C65F8: PyEval_EvalFrame (in /usr/lib/libpython2.4.so.1.0) ==21737== Address 0xC02E010 is 32 bytes inside a block of size 40 free'd ==21737==at 0x401D139: free (vg_replace_malloc.c:233) ==21737==by 0x408DE00: PyObject_Free (in /usr/lib/libpython2.4.so.1.0) ==21737==by 0x407BB4D: (within /usr/lib/libpython2.4.so.1.0) ==21737==by 0x407A3D6: (within /usr/lib/libpython2.4.so.1.0) Here python reads from an already-freed memory area, right? (I don't think that Misc/README.valgrind answers this question). Or is it a false alarm? Thanks, Geza Herman ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] valgrind
Herman Geza schrieb: Here python reads from an already-freed memory area, right? It looks like it, yes. Of course, it could be a flaw in valgrind, too. To find out, one would have to understand what the memory block is, and what part of PyObject_Free accesses it. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] valgrind
On 11/6/06, Martin v. Löwis [EMAIL PROTECTED] wrote: Herman Geza schrieb: Here python reads from an already-freed memory area, right? It looks like it, yes. Of course, it could be a flaw in valgrind, too. To find out, one would have to understand what the memory block is, and what part of PyObject_Free accesses it. I'm a bit confused. I ran with valgrind ./python -c pass which returns 23 invalid read problems (some are the same chunk of memory). This is with 2.5 (more or less). Valgrind 3.2.1 on amd64. Every address ended with 0x5...020. That seems odd. I looked through the valgrind bug reports and didn't see anything. The first problem reported was: Invalid read of size 4 at 0x44FA06: Py_ADDRESS_IN_RANGE (obmalloc.c:1741) by 0x44E225: PyObject_Free (obmalloc.c:920) by 0x44EB90: _PyObject_DebugFree (obmalloc.c:1361) by 0x444A28: dictresize (dictobject.c:546) by 0x444D5B: PyDict_SetItem (dictobject.c:655) by 0x462533: PyString_InternInPlace (stringobject.c:4920) by 0x448450: PyDict_SetItemString (dictobject.c:2120) by 0x4C240A: PyModule_AddObject (modsupport.c:615) by 0x428B00: _PyExc_Init (exceptions.c:2117) by 0x4C449A: Py_InitializeEx (pythonrun.c:225) by 0x4C4827: Py_Initialize (pythonrun.c:315) by 0x41270A: Py_Main (main.c:449) Address 0x52AE020 is 4,392 bytes inside a block of size 5,544 free'd at 0x4A1A828: free (vg_replace_malloc.c:233) by 0x5071635: qsort (in /lib/libc-2.3.5.so) by 0x474E4B: init_slotdefs (typeobject.c:5368) by 0x47522E: add_operators (typeobject.c:5511) by 0x46E3A1: PyType_Ready (typeobject.c:3209) by 0x46E2D4: PyType_Ready (typeobject.c:3173) by 0x44D13E: _Py_ReadyTypes (object.c:1864) by 0x4C4362: Py_InitializeEx (pythonrun.c:183) by 0x4C4827: Py_Initialize (pythonrun.c:315) by 0x41270A: Py_Main (main.c:449) by 0x411CD2: main (python.c:23) Note that the free is inside qsort. The memory freed under qsort should definitely not be the bases which we allocated under PyType_Ready. I'll file a bug report with valgrind to help determine if this is a problem in Python or valgrind. http://bugs.kde.org/show_bug.cgi?id=136989 One other thing that is weird is that the complaint is about 4 bytes which should not be possible. All pointers should be 8 bytes AFAIK since this is amd64. I also ran this on x86. There were 32 errors and all of their addresses were 0x4...010. n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] valgrind
[Herman Geza] Here python reads from an already-freed memory area, right? [Martin v. Löwis] It looks like it, yes. Of course, it could be a flaw in valgrind, too. To find out, one would have to understand what the memory block is, and what part of PyObject_Free accesses it. When PyObject_Free is handed an address it doesn't control, the arena base address it derives from that address may point at anything the system malloc controls, including uninitialized memory, memory the system malloc has allocated to something, memory the system malloc has freed, or internal system malloc bookkeeping bytes. The Py_ADDRESS_IN_RANGE macro has no way to know before reading it up. So figure out which line of code valgrind is complaining about (doesn't valgrind usually produce that?). If it's coming from the expansion of Py_ADDRESS_IN_RANGE, it's not worth more thought. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
