On Thu, Mar 28, 2013 at 11:38 AM, Paul Moore <p.f.mo...@gmail.com> wrote:
> On 28 March 2013 13:42, Brett Cannon <br...@python.org> wrote: > >> importer, as I wanted to try a proof of concept importer based on the > >> new importlib stuff (which is intended to make writing custom > >> importers easier), and I really struggled to get something working. > > > > Struggling how? With the finder? The loader? What exactly were you > trying to > > accomplish and how were you deviating from the standard import system? > > What I was trying to do was to write a path hook that would allow me > to add a string to sys.path which contained a base-64 encoded zipfile > (plus some sort of marker so it could be distinguished from a normal > path entry) and as a result the contents of that embedded zip file > would be available as if I'd added an actual zip file with that > content to sys.path. > > I got in a complete mess as I tried to strip out the (essentially > non-interesting) zipfile handling to get to a dummy "do nothing, > everything is valid" type of example. But I don't think I would have > fared much better if I'd stuck to the original full requirement. > > >> It seems to me that the importlib documentation doesn't help much for > >> people trying to import path hooks. > > > > There is a bug to clarify the docs to have more geared towards writing > new > > importers instead of just documenting what's available: > > http://bugs.python.org/issue15867 > > Thanks. I'll keep an eye on that. > > >> But it might be just me. Does > >> anyone have an example of a simple importlib-based finder/loader? > > > > Define simple. =) I would argue importlib itself is easy enough to read. > > :-) Fair point. I guess importlib is not *that* hard to read, but the > only case that implements packages in the filesystem one, and that > also deals with C extensions and other complexities that I don't have > a need for. I'll try again to have a deeper look at it, but I didn't > find it easy to extract the essentials when I looked before. > > >> That > >> would be a huge help for me. In return for any pointers, I'll look at > >> putting together a doc patch to clarify how to use importlib to build > >> your own path hooks :-) > > > > Do you specifically mean the path hook aspect or the whole package of > hook, > > finder, and loader? > > OK, after some more digging, it looks like I misunderstood the process > somewhat. Writing a loader that inherits from *both* FileLoader and > SourceLoader, You only need SourceLoader since you are dealing with Python source. You don't need FileLoader since you are not reading from disk but an in-memory zipfile. > and then implementing get_data (and module_repr - why do > I need that, couldn't the ABC offer a default implementation?) http://bugs.python.org/issue17093 and http://bugs.python.org/issue17566 > does > the job for that. > You should be implementing get_data, get_filename, and path_stats for SourceLoader. > > But the finder confuses me. I assume I want a PathEntryFinder and > hence I should implement find_loader(). Yes since you are pulling from sys.path. > The documentation on what I > need to return from there is very sparse... In the end I worked out > that for a package, I need to return (MyLoader(modulename, > 'foo/__init__.py'), ['foo']) (here, "foo" is my dummy marker for my > example). The second argument should just be None: "An empty list can be used for portion to signify the loader is not part of a [namespace] package". Unfortunately a key word is missing in that sentence. http://bugs.python.org/issue17567 > In essence, PathEntryFinder really has to implement some > form of virtual filesystem mount point, and preserve the standard > filesystem semantics of modules having a filename of .../__init__.py. > Well, if your zip file decided to create itself with a different file extension then it wouldn't be required, but then other people's code might break if they don't respect module abstractions (i.e. looking at __package__/__name__ or __path__ to see if something is a package). > > So I managed to work out what was needed in the end, but it was a lot > harder than I'd expected. On reflection, getting the finder semantics > right (and in particular the path entry finder semantics) was the hard > bit. > Yep, that bit has had the least API tweaks as most people don't muck with finders but with loaders. > > I'm now 100% sure that some cookbook examples would help a lot. I'll > see what I can do. > I plan on writing a pure Python zip importer for Python 3.4 which should be fairly minimal and work out as a good example chunk of code. And no one need bother writing it as I'm going to do it myself regardless to make sure I plug any missing holes in the API. If you really want something to try for fun go for a sqlite3-backed setup (don't see it going in the stdlib but it would be a project to have).
_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
