Re: [Python-Dev] Implementing PEP 382, Namespace Packages
Looking at that proposal, I don't follow how changing *loaders* (vs. importers) would help. If an importer's find_module doesn't natively support PEP 382, then there's no way to get a loader for the package in the first place. True. However, this requires no changes to the API, AFAICT. The *finder* (there are no importers, right?) will need to accept a folder with just a pth file as a package, but then still return a loader. The issue is then how to make the loader scan the folder for .pth files. One option would be to ask it "give me the contents of all the pth files", the other would be to have a method "load all the pth files". In the PEP 302 scheme, then, it's either importers that have to change, or the process that invokes them. Being able to ask an importer the equivalents of os.path.join, listdir, and get_data would suffice to make an import process that could do the trick. That would also work, but it would make a fairly wide interface. IIRC, MAL complained that doing so would break importers which can't do listdir. Essentially, you'd ask each importer to first attempt to find the module, and then asking it (or the loader, if the find worked) whether packagename/*.pth exists, and then processing their contents. The latter is my proposal, yes: ask the loader to process all pth files. If at any point the find_module() call succeeds, then subsequent importers will just be asked for .pth files, which can then be processed into the __path__ of the now-loaded module. IOW, something like this (very rough draft): >pth_contents = [] >module = None > >for pathitem in syspath_or_parent__path__: > >importer = pkgutil.get_importer(pathitem) >if importer is None: >continue > >if module is None: >try: >loader = importer.find_module(fullname) >except ImportError: >pass Is this really how it works today? Shouldn't it abort here if there is an ImportError? >else: ># errors here should propagate >module = loader.load_module(fullname) >if not hasattr(module, '__path__'): ># found, but not a package >return module > >pc = get_pth_contents(importer) Assuming we always get here with a loader, I'd rather call this on the loader. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Implementing PEP 382, Namespace Packages
For finders, their search algorithm is changed in a couple of ways.
One is that modules are given priority over packages (is that
intentional, Martin, or just an oversight?).
That's an oversight. Notice, however, that it's really not the case that
currently directories have precedence over modules, either: if a
directory is later on the __path__ than a module, it's still the module
that gets imported. So the precedence takes place only when a module and
a directory exist in the same directory.
In any case, I have now fixed it.
Two, the package search
requires checking for a .pth file on top of an __init__.py. This will
change finders that could before simply do an existence check on an
__init__ "file"
You are reading something into the PEP that isn't there yet. PEP 302
currently isn't considered, and the question of this discussion is
precisely how the loaders API should be changed.
(or whatever the storage back-end happened to be) and
make it into a list-and-search which one would hope wasn't costly, but
in same cases might be if the paths to files is not stored in a
hierarchical fashion (e.g. zip files list entire files paths in their
TOC or a sqlite3 DB which uses a path for keys will have to list
**all** keys, sort them to just the relevant directory, and then look
for .pth or some such approach).
First, I think it's up to the specific loader mechanism whether
PEP 382 should be supported at all. It should be possible to implement
it if desired, but if it's not feasible (e.g. for URL loaders), pth
files just may not get considered. The loader may well provide a
different mechanism to support namespace packages.
> Are we worried about possible
performance implications of this search?
For the specific case of zip files, I'm not. I don't think performance
will suffer at all.
And then the search for the __init__.py begins on the newly modified
__path__, which I assume ends with the first __init__ found on
__path__, but if no file is found it's okay and essentially an empty
module with just module-specific attributes is used?
Correct.
In other words,
can a .pth file replace an __init__ file in delineating a package?
That's what it means by '''a directory is considered a package if it
either contains a file named __init__.py, or a file whose name ends with
".pth".'''
Or
is it purely additive? I assume the latter for compatibility reasons,
but the PEP says "a directory is considered a package if it **either**
contains a file named __init__.py, **or** a file whose name ends with
".pth"" (emphasis mine).
Why do you think this causes an incompatibility?
Otherwise I assume that the search will be
done simply with ``os.path.isdir(os.path.join(sys_path_entry,
top_level_package_name)`` and all existing paths will be added to
__path__. Will they come before or after the directory where the *.pth
was found? And will any subsequent *.pth files found in other
directories also be executed?
I may misremember, but from reading the text, it seems to say "no".
It should work like the current pth mechanism (plus *, minus import).
As for how "*" works, is this limited to top-level packages, or will
sub-packages participate as well? I assume the former, but it is not
directly stated in the PEP.
And indeed, the latter is intended. You should be able to create
namespace packages on all levels.
If the latter, is a dotted package name
changed to ``os.sep.join(sy_path_entry, package_name.replace('".",
os.sep)``?
No. Instead, the parent package's __path__ is being searched for
directories; sys.path is not considered anymore. I have fixed the text.
For sys.path_hooks, I am assuming import will simply skip over passing
that as it is a marker that __path__ represents a namsepace package
and not in any way functional. Although with sys.namespace_packages,
is leaving the "*" in __path__ truly necessary?
It would help with efficiency, no?
For the search of paths to use to extend, are we limiting ourselves to
actual file system entries on sys.path (as pkgutil does), or do we
want to support other storage back-ends? To do the latter I would
suggest having a successful path discovery be when a finder can be
created for the hypothetical directory from sys.path_hooks.
Again: PEP 302 isn't really considered yet. Proposals are welcome.
The PEP (seems to) ask finders to look for a .pth file(s), calculate
__path__, and then get a loader for the __init__. You could have
finders grow a find_namespace method which returns the contents of the
requisite .pth file(s).
I must be misunderstanding the concept of finders. Why is it that it
would be their function to process the pth files, and not the function
of the loader?
Regards,
Martin
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Implementing PEP 382, Namespace Packages
The PEP says the goal is to span packages across directories. If you split something like zope into multiple directories, does having a separate zope.pth file in each of those directories really cause a problem here? I think pje already answered this: yes, you do split zope into multiple packages. But then, you may install all of them into the same folder. This has caused pain for Linux package management tools in particular, because they dislike if to packages install the same file. So if you can arrange the pth files to have non-overlapping names, you can still install them into the same directory, and dpkg is happy. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bugfix releases should not change APIs
Is it possible with svn or hg to get a list of the commits that changed version x to version y? A regular "svn log" on the maintenance branch will give you all the changes. You'll recognize from the checkin messages when the previous release was. Would is not be possible to get a diff between at least the .rst versions of the docs for version x and version y? That's most certainly possible, using "svn diff -rX:Y". Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Implementing PEP 382, Namespace Packages
At 09:24 AM 5/31/2010 +0200, Martin v. Löwis wrote: Is this really how it works today? Shouldn't it abort here if there is an ImportError? Oops. You're right - I was confusing find_module with the path_hooks protoocol. >else: ># errors here should propagate >module = loader.load_module(fullname) >if not hasattr(module, '__path__'): ># found, but not a package >return module > >pc = get_pth_contents(importer) Assuming we always get here with a loader, I'd rather call this on the loader. We don't, unless the finder natively supports PEP 382. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] tp_dealloc
Hi, when embedding python 3.1, I have my own free-method in tp_dealloc. The allocated memory is in host-memory, not in python (dll). Now, the problem is, Python appears to read-access the deallocated memory still after tp_dealloc. After tp_dealloc, I get an access violation if the pyobject-header fields have been modified inside tp_dealloc. If I leave the header unmodified, then no access violation occurs. Accessing deallocated memory sounds like a bug, or is this intended design? Thank you Marvin -- GRATIS für alle GMX-Mitglieder: Die maxdome Movie-FLAT! Jetzt freischalten unter http://portal.gmx.net/de/go/maxdome01 ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Implementing PEP 382, Namespace Packages
On Mon, May 31, 2010 at 00:53, "Martin v. Löwis" wrote:
>> For finders, their search algorithm is changed in a couple of ways.
>> One is that modules are given priority over packages (is that
>> intentional, Martin, or just an oversight?).
>
> That's an oversight. Notice, however, that it's really not the case that
> currently directories have precedence over modules, either: if a directory
> is later on the __path__ than a module, it's still the module that gets
> imported. So the precedence takes place only when a module and a directory
> exist in the same directory.
>
> In any case, I have now fixed it.
>
>> Two, the package search
>> requires checking for a .pth file on top of an __init__.py. This will
>> change finders that could before simply do an existence check on an
>> __init__ "file"
>
> You are reading something into the PEP that isn't there yet. PEP 302
> currently isn't considered, and the question of this discussion is precisely
> how the loaders API should be changed.
>
>> (or whatever the storage back-end happened to be) and
>> make it into a list-and-search which one would hope wasn't costly, but
>> in same cases might be if the paths to files is not stored in a
>> hierarchical fashion (e.g. zip files list entire files paths in their
>> TOC or a sqlite3 DB which uses a path for keys will have to list
>> **all** keys, sort them to just the relevant directory, and then look
>> for .pth or some such approach).
>
> First, I think it's up to the specific loader mechanism whether
> PEP 382 should be supported at all. It should be possible to implement
> it if desired, but if it's not feasible (e.g. for URL loaders), pth
> files just may not get considered. The loader may well provide a
> different mechanism to support namespace packages.
>
>> Are we worried about possible
>>
>> performance implications of this search?
>
> For the specific case of zip files, I'm not. I don't think performance will
> suffer at all.
>
>> And then the search for the __init__.py begins on the newly modified
>> __path__, which I assume ends with the first __init__ found on
>> __path__, but if no file is found it's okay and essentially an empty
>> module with just module-specific attributes is used?
>
> Correct.
>
>> In other words,
>> can a .pth file replace an __init__ file in delineating a package?
>
> That's what it means by '''a directory is considered a package if it either
> contains a file named __init__.py, or a file whose name ends with ".pth".'''
>
>> Or
>> is it purely additive? I assume the latter for compatibility reasons,
>> but the PEP says "a directory is considered a package if it **either**
>> contains a file named __init__.py, **or** a file whose name ends with
>> ".pth"" (emphasis mine).
>
> Why do you think this causes an incompatibility?
It's just if a project has no __init__ older finders won't process it,
that's all. But it looks like they are going to have to change
somewhat anyway so that's not an issue.
>
>> Otherwise I assume that the search will be
>> done simply with ``os.path.isdir(os.path.join(sys_path_entry,
>> top_level_package_name)`` and all existing paths will be added to
>> __path__. Will they come before or after the directory where the *.pth
>> was found? And will any subsequent *.pth files found in other
>> directories also be executed?
>
> I may misremember, but from reading the text, it seems to say "no".
> It should work like the current pth mechanism (plus *, minus import).
>
>> As for how "*" works, is this limited to top-level packages, or will
>> sub-packages participate as well? I assume the former, but it is not
>> directly stated in the PEP.
>
> And indeed, the latter is intended. You should be able to create namespace
> packages on all levels.
>
>> If the latter, is a dotted package name
>> changed to ``os.sep.join(sy_path_entry, package_name.replace('".",
>> os.sep)``?
>
> No. Instead, the parent package's __path__ is being searched for
> directories; sys.path is not considered anymore. I have fixed the text.
>
>> For sys.path_hooks, I am assuming import will simply skip over passing
>> that as it is a marker that __path__ represents a namsepace package
>> and not in any way functional. Although with sys.namespace_packages,
>> is leaving the "*" in __path__ truly necessary?
>
> It would help with efficiency, no?
Not sure how having "*" in __path__ helps with efficiency. What are
you thinking it will help with specifically?
>
>> For the search of paths to use to extend, are we limiting ourselves to
>> actual file system entries on sys.path (as pkgutil does), or do we
>> want to support other storage back-ends? To do the latter I would
>> suggest having a successful path discovery be when a finder can be
>> created for the hypothetical directory from sys.path_hooks.
>
> Again: PEP 302 isn't really considered yet. Proposals are welcome.
>
>> The PEP (seems to) ask finders to look for a .pth file(s), calculate
>> __path__, and then get a loader for the __init__. Y
Re: [Python-Dev] Implementing PEP 382, Namespace Packages
On Sun, May 30, 2010 at 22:03, P.J. Eby wrote: > At 06:18 PM 5/30/2010 -0700, Brett Cannon wrote: >> >> On Sun, May 30, 2010 at 00:40, P.J. Eby wrote: >> > >> > Which would completely break one of the major use cases of the PEP, >> > which is >> > precisely to ensure that you can install two pieces of code to the same >> > namespace without either one overwriting the other's files. >> >> The PEP says the goal is to span packages across directories. > > The goal of namespace packages is to allow separately-distributed pieces of > code to live in the same package namespace. That this is sometimes achieved > by installing them to different paths is an implementation detail. > > In the case of e.g. Linux distributions and other system packaging > scenarios, the code will all be installed to the *same* directory -- so > there cannot be any filename collisions among the separately-distributed > modules. That's why we want to get rid of the need for an __init__.py to > mark the directory as a package: it's a collision point for system package > management tools. > > >> > pkgutil doesn't have such a limitation, except in the case extend_path, >> > and >> > that limitation is one that PEP 382 intends to remove. >> >> It's because pkgutil.extend_path has that limitation I am asking as >> that's what the PEP refers to. If the PEP wants to remove the >> limitation it should clearly state how it is going to do that. > > I'm flexible on it either way. The only other importer I know of that does > anything else is one that actually allows (unsafely) importing from URLs. > > If we allow for other things, then we need to extend the PEP 302 protocol to > have a way to ask an importer for a subpath string. > > > >> As for adding to the PEP 302 protocols, it's a question of how much we >> want importer implementors to have control over this versus us. I >> personally would rather keep any protocol extensions simple and have >> import handle as many of the details as possible. > > I lean the other way a bit, in that the more of the importer internals you > expose, the harder you make it for an importer to be anything other than a > mere virtual file system. (As it is, I think there is too much "file-ness" > coupling in the protocol already, what with file extensions and the like.) > Yes, there is a VERY strong file path connection in loaders and they have essentially become simple virtual file systems. But that does make implementation of alternative back-ends easier which seems to be the common case. Plus pre-existing code already mutates __path__, __file__, etc. as if they are file paths using os.path.join. And imp's new source_from_cache and cache_from_source are not weakening the tie to file paths either. But as long as whatever mechanism gets exposed allows people to work from a module name that will be enough. The path connection is not required as load_module is the end-all-be-all method. If we have a similar API added for .pth files that works off of module names then those loaders that don't want to work from file paths don't have to. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] _XOPEN_SOURCE on Solaris
In issue 1759169 people have been demanding for quite some time that the definition of _XOPEN_SOURCE on Solaris should be dropped, as it was unneeded and caused problems for other software. Now, issue 8864 reports that the multiprocessing module fails to compile, and indeed, if _XOPEN_SOURCE is not defined, control messages stop working. Several of the CMSG interfaces are only available if _XPG4_2 is defined (and, AFAICT, under no other condition); this, in turn, apparently is only defined if _XOPEN_SOURCE is 500, 600, or (has an arbitrary value and _XOPEN_SOURCE_EXTENDED is 1). So how should I go about fixing that? a) revert the patch for #1759169, documentating that Python compilation actually requires _XOPEN_SOURCE to be defined, or b) define _XOPEN_SOURCE only for the multiprocessing module. Any input appreciated. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] tp_dealloc
[email protected] wrote: Now, the problem is, Python appears to read-access the deallocated memory still after tp_dealloc. It's not clear exactly what you mean by "after tp_dealloc". The usual pattern is for a type's tp_dealloc method to call the base type's tp_dealloc, which can make further references to the object's memory. At the end of the tp_dealloc chain, tp_free gets called, which is what actually deallocates the memory. I would say your tp_dealloc shouldn't be modifying anything in the object struct that your corresponding tp_alloc method didn't set up, because code further along the tp_dealloc chain may rely on it. That includes fields in the object header. -- Greg ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Implementing PEP 382, Namespace Packages
At 01:19 PM 5/31/2010 -0700, Brett Cannon wrote: But as long as whatever mechanism gets exposed allows people to work from a module name that will be enough. The path connection is not required as load_module is the end-all-be-all method. If we have a similar API added for .pth files that works off of module names then those loaders that don't want to work from file paths don't have to. Right - that's why I suggested that a high-level request like get_pth_contents() would give the implementer the most flexibility. Then they don't have to fake a filesystem if they don't actually work that way. For example, a database that maps module names to code objects has no need for paths at all, and could just return either ['*'] or None depending on whether the package was marked as a namespace package in the database... without needing to fake up the existence of a .pth file in a virtual file system. (Of course, since lots of implementations *do* use filesystem-like backends, giving them some utility functions they can use to implement the API on top of filesystem operations gives us the best of both worlds.) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
