Re: [Python-Dev] Can I introspect/reflect to get arguments exec()?

[Rocky Bernstein]

Thank you for your very thoughtful and detailed explanation of what is
going on and for your considerations as to how to get this done (as opposed
to why it *can't *be done).

It looks like DecoratorTools or more likely a customized version of it is
the way to go!

(The important info is above. The rest are just some geeky details)

You see, there are some places where additional care may be needed in my
setting.

The debuggers I write use sometimes don't use just the getline module but
also use my own pyficache module. I want to also cache things like file
stat() info, provide a SHA1 for the text of the file, and provide colorized
syntax-highlighted versions of the text when desired. But since I control
pyficache, I can mirror the changes made to getline.

Of course the debugger uses sys.settrace too, so the evil-ness of that is
definitely not a concern. But possibly I need to make sure that since the
DecoratorTools and the debugger both hook into trace hooks they play nice
together and fire in the right order. And for that I created another module
called tracer() which takes into account that one might want to specify
priorities in the chain hook order, and that one might want a to filter out
(i.e. ignore) certain calls to the hook function for specific hooks.

It may be a while before I seriously get to this, but again, it is good to
have in mind an approach to take. So thanks again.

On Thu, Mar 28, 2013 at 1:45 AM, PJ Eby  wrote:

> On Tue, Mar 26, 2013 at 11:00 PM, Rocky Bernstein  wrote:
> > Okay. But is the string is still somewhere in the CPython VM stack? (The
> > result of LOAD_CONST 4 above). Is there a way to pick it up from there?
>
> Maybe using C you could peek into the frame's value stack, but that's
> not exposed to any Python API I know of.  But that still doesn't help
> you, because the value will be removed from the stack before exec() is
> actually called, which means if you go looking for it in code called
> from the exec (e.g. the call event itself), you aren't going to see
> the data.
>
> > At the point that we are stopped the exec action hasn't taken place yet.
>
> That doesn't help if you're using line-level tracing events.  At the
> beginning of the line, the data's not on the call stack yet, and by
> the time you enter the frame of the code being exec'd, it'll be off
> the stack again.
>
> Basically, there is no way to do what you're asking for, short of
> replacing the built-in exec function with your own version.  And it
> still won't help you with stepping through the source of functions
> that are *compiled* using an exec() or compile(), or any other way of
> ending up with dynamically-generated code you want to debug.
>
> (Unless you use something like DecoratorTools to generate it, that is
> -- DecoratorTools has some facilities for caching
> dynamically-generated code so that it works properly with debuggers.
> But that has to be done by the code doing the generation, not the
> debugger.  If the code generator uses DecoratorTools' caching support,
> then any debugger that uses the linecache module will Just Work.  It
> might be nice for  the stdlib should have something like this, but you
> could also potentially fake it by replacing the builtin eval, exec,
> compile, etc. functions w/versions that cache the source.)
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Can we triple quoted string as a comment?

[Xavier de Gaye]

On Tue, Mar 26, 2013 at 2:28 PM, R. David Murray wrote:
> On Mon, 25 Mar 2013 18:16:47 -0700, Raymond Hettinger wrote:
>> If you're editing with Emacs, it is really easy to reflow paragraphs
>> and to insert or remove multiline comments each prefixed with #.
>> But with other editors, it can be a PITA and a multiline string is
>> the easiest to maintain and works well when cutting-and-pasting
>> the comments from somewhere else.
>
> Just FYI it is also very easy in vim: gq plus whatever movement prefix
> suits the situation.

And to comment out multiple lines in vim, each prefixed with #, see
":help v_b_I" and ":help v_b_I_example".

Xavier
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Writing importers and path hooks

[Brett Cannon]

On Wed, Mar 27, 2013 at 6:59 PM, Paul Moore  wrote:

> On 27 March 2013 21:19, Bradley M. Froehle  wrote:
> > I implemented just such a path hook  zipimporter plus the magic
> required
> > for C extensions --- as a challenge to myself to learn more about the
> Python
> > import mechanisms.
> >
> > See https://github.com/bfroehle/pydzipimport.
>
> Apologies for hijacking the thread, but it's interesting that you
> implemented your hook like this. I notice that you didn't use any of
> the importlib functionality in doing so. Was there a particular
> reason? I ask because a few days ago, I was writing a very similar
> importer, as I wanted to try a proof of concept importer based on the
> new importlib stuff (which is intended to make writing custom
> importers easier), and I really struggled to get something working.
>

Struggling how? With the finder? The loader? What exactly were you trying
to accomplish and how were you deviating from the standard import system?


>
> It seems to me that the importlib documentation doesn't help much for
> people trying to import path hooks.


There is a bug to clarify the docs to have more geared towards writing new
importers instead of just documenting what's available:
http://bugs.python.org/issue15867


> But it might be just me. Does
> anyone have an example of a simple importlib-based finder/loader?


Define simple. =) I would argue importlib itself is easy enough to read.


> That
> would be a huge help for me. In return for any pointers, I'll look at
> putting together a doc patch to clarify how to use importlib to build
> your own path hooks :-)
>

Do you specifically mean the path hook aspect or the whole package of hook,
finder, and loader?
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Safely importing zip files with C extensions

[Thomas Heller]

Am 27.03.2013 20:38, schrieb Vinay Sajip:

This quote is here to stop GMane complaining that I'm top-posting. Ignore.


I've already posted this to distutils-sig, but thought that it might be of
interest to readers here as it relates to importing C extensions ...

zipimport is great, but there can be issues importing software that contains
C extensions. But the new wheel format (PEP 427) may give us a better way of
importing zip files containing C extensions. Since wheels are .zip files, they
can sometimes be used to provide functionality without needing to be installed.
But whereas .zip files contain no convention for indicating compatibility with
a particular Python, wheels do contain this compatibility information. Thus, it
is possible to check if a wheel can be directly imported from, and the wheel
support in distlib allows you to take advantage of this using the mount() and
unmount() methods. When you mount a wheel, its absolute path name is added to
sys.path, allowing the Python code in it to be imported. (A DistlibException is
raised if the wheel isn't compatible with the Python which calls the mount()
method.)


The zip-file itself could support importing compiled extensions when it 
contains a python-wrapper module that unpacks the .so/.dll file 
somewhere, and finally calls imp.load_dynamic() to import it and replace 
itself.


Thomas


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Writing importers and path hooks

[Paul Moore]

On 28 March 2013 13:42, Brett Cannon  wrote:
>> importer, as I wanted to try a proof of concept importer based on the
>> new importlib stuff (which is intended to make writing custom
>> importers easier), and I really struggled to get something working.
>
> Struggling how? With the finder? The loader? What exactly were you trying to
> accomplish and how were you deviating from the standard import system?

What I was trying to do was to write a path hook that would allow me
to add a string to sys.path which contained a base-64 encoded zipfile
(plus some sort of marker so it could be distinguished from a normal
path entry) and as a result the contents of that embedded zip file
would be available as if I'd added an actual zip file with that
content to sys.path.

I got in a complete mess as I tried to strip out the (essentially
non-interesting) zipfile handling to get to a dummy "do nothing,
everything is valid" type of example. But I don't think I would have
fared much better if I'd stuck to the original full requirement.

>> It seems to me that the importlib documentation doesn't help much for
>> people trying to import path hooks.
>
> There is a bug to clarify the docs to have more geared towards writing new
> importers instead of just documenting what's available:
> http://bugs.python.org/issue15867

Thanks. I'll keep an eye on that.

>> But it might be just me. Does
>> anyone have an example of a simple importlib-based finder/loader?
>
> Define simple. =) I would argue importlib itself is easy enough to read.

:-) Fair point. I guess importlib is not *that* hard to read, but the
only case that implements packages in the filesystem one, and that
also deals with C extensions and other complexities that I don't have
a need for. I'll try again to have a deeper look at it, but I didn't
find it easy to extract the essentials when I looked before.

>> That
>> would be a huge help for me. In return for any pointers, I'll look at
>> putting together a doc patch to clarify how to use importlib to build
>> your own path hooks :-)
>
> Do you specifically mean the path hook aspect or the whole package of hook,
> finder, and loader?

OK, after some more digging, it looks like I misunderstood the process
somewhat. Writing a loader that inherits from *both* FileLoader and
SourceLoader, and then implementing get_data (and module_repr - why do
I need that, couldn't the ABC offer a default implementation?) does
the job for that.

But the finder confuses me. I assume I want a PathEntryFinder and
hence I should implement find_loader(). The documentation on what I
need to return from there is very sparse... In the end I worked out
that for a package, I need to return (MyLoader(modulename,
'foo/__init__.py'), ['foo']) (here, "foo" is my dummy marker for my
example). In essence, PathEntryFinder really has to implement some
form of virtual filesystem mount point, and preserve the standard
filesystem semantics of modules having a filename of .../__init__.py.

So I managed to work out what was needed in the end, but it was a lot
harder than I'd expected. On reflection, getting the finder semantics
right (and in particular the path entry finder semantics) was the hard
bit.

I'm now 100% sure that some cookbook examples would help a lot. I'll
see what I can do.

Thanks,
Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Writing importers and path hooks

[Brett Cannon]

On Thu, Mar 28, 2013 at 11:38 AM, Paul Moore  wrote:

> On 28 March 2013 13:42, Brett Cannon  wrote:
> >> importer, as I wanted to try a proof of concept importer based on the
> >> new importlib stuff (which is intended to make writing custom
> >> importers easier), and I really struggled to get something working.
> >
> > Struggling how? With the finder? The loader? What exactly were you
> trying to
> > accomplish and how were you deviating from the standard import system?
>
> What I was trying to do was to write a path hook that would allow me
> to add a string to sys.path which contained a base-64 encoded zipfile
> (plus some sort of marker so it could be distinguished from a normal
> path entry) and as a result the contents of that embedded zip file
> would be available as if I'd added an actual zip file with that
> content to sys.path.
>
> I got in a complete mess as I tried to strip out the (essentially
> non-interesting) zipfile handling to get to a dummy "do nothing,
> everything is valid" type of example. But I don't think I would have
> fared much better if I'd stuck to the original full requirement.
>
> >> It seems to me that the importlib documentation doesn't help much for
> >> people trying to import path hooks.
> >
> > There is a bug to clarify the docs to have more geared towards writing
> new
> > importers instead of just documenting what's available:
> > http://bugs.python.org/issue15867
>
> Thanks. I'll keep an eye on that.
>
> >> But it might be just me. Does
> >> anyone have an example of a simple importlib-based finder/loader?
> >
> > Define simple. =) I would argue importlib itself is easy enough to read.
>
> :-) Fair point. I guess importlib is not *that* hard to read, but the
> only case that implements packages in the filesystem one, and that
> also deals with C extensions and other complexities that I don't have
> a need for. I'll try again to have a deeper look at it, but I didn't
> find it easy to extract the essentials when I looked before.
>
> >> That
> >> would be a huge help for me. In return for any pointers, I'll look at
> >> putting together a doc patch to clarify how to use importlib to build
> >> your own path hooks :-)
> >
> > Do you specifically mean the path hook aspect or the whole package of
> hook,
> > finder, and loader?
>
> OK, after some more digging, it looks like I misunderstood the process
> somewhat. Writing a loader that inherits from *both* FileLoader and
> SourceLoader,


You only need SourceLoader since you are dealing with Python source. You
don't need FileLoader since you are not reading from disk but an in-memory
zipfile.


> and then implementing get_data (and module_repr - why do
> I need that, couldn't the ABC offer a default implementation?)


http://bugs.python.org/issue17093 and http://bugs.python.org/issue17566


> does
> the job for that.
>

You should be implementing get_data, get_filename, and path_stats for
SourceLoader.


>
> But the finder confuses me. I assume I want a PathEntryFinder and
> hence I should implement find_loader().


Yes since you are pulling from sys.path.


> The documentation on what I
> need to return from there is very sparse... In the end I worked out
> that for a package, I need to return (MyLoader(modulename,
> 'foo/__init__.py'), ['foo']) (here, "foo" is my dummy marker for my
> example).


The second argument should just be None: "An empty list can be used for
portion to signify the loader is not part of a [namespace] package".
Unfortunately a key word is missing in that sentence.
http://bugs.python.org/issue17567


> In essence, PathEntryFinder really has to implement some
> form of virtual filesystem mount point, and preserve the standard
> filesystem semantics of modules having a filename of .../__init__.py.
>

Well, if your zip file decided to create itself with a different file
extension then it wouldn't be required, but then other people's code might
break if they don't respect module abstractions (i.e. looking at
__package__/__name__ or __path__ to see if something is a package).


>
> So I managed to work out what was needed in the end, but it was a lot
> harder than I'd expected. On reflection, getting the finder semantics
> right (and in particular the path entry finder semantics) was the hard
> bit.
>

Yep, that bit has had the least API tweaks as most people don't muck with
finders but with loaders.


>
> I'm now 100% sure that some cookbook examples would help a lot. I'll
> see what I can do.
>

I plan on writing a pure Python zip importer for Python 3.4 which should be
fairly minimal and work out as a good example chunk of code.  And no one
need bother writing it as I'm going to do it myself regardless to make sure
I plug any missing holes in the API. If you really want something to try
for fun go for a sqlite3-backed setup (don't see it going in the stdlib but
it would be a project to have).
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.p

Re: [Python-Dev] Safely importing zip files with C extensions

[Brett Cannon]

On Thu, Mar 28, 2013 at 10:44 AM, Thomas Heller  wrote:

> Am 27.03.2013 20:38, schrieb Vinay Sajip:
>
>  This quote is here to stop GMane complaining that I'm top-posting. Ignore.
>>>
>>
>> I've already posted this to distutils-sig, but thought that it might be of
>> interest to readers here as it relates to importing C extensions ...
>>
>> zipimport is great, but there can be issues importing software that
>> contains
>> C extensions. But the new wheel format (PEP 427) may give us a better way
>> of
>> importing zip files containing C extensions. Since wheels are .zip files,
>> they
>> can sometimes be used to provide functionality without needing to be
>> installed.
>> But whereas .zip files contain no convention for indicating compatibility
>> with
>> a particular Python, wheels do contain this compatibility information.
>> Thus, it
>> is possible to check if a wheel can be directly imported from, and the
>> wheel
>> support in distlib allows you to take advantage of this using the mount()
>> and
>> unmount() methods. When you mount a wheel, its absolute path name is
>> added to
>> sys.path, allowing the Python code in it to be imported. (A
>> DistlibException is
>> raised if the wheel isn't compatible with the Python which calls the
>> mount()
>> method.)
>>
>
> The zip-file itself could support importing compiled extensions when it
> contains a python-wrapper module that unpacks the .so/.dll file somewhere,
> and finally calls imp.load_dynamic() to import it and replace itself.


Which must be done carefully to prevent a security issue. It shouldn't be
unzipped anywhere but into a directory only writable by the process.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Writing importers and path hooks

[Paul Moore]

On 28 March 2013 16:08, Brett Cannon  wrote:
> You only need SourceLoader since you are dealing with Python source. You
> don't need FileLoader since you are not reading from disk but an in-memory
> zipfile.
>
> You should be implementing get_data, get_filename, and path_stats for
> SourceLoader.

OK, cool. That helps a lot.

The biggest gap here is that I don't think that anywhere has a good
explanation of the required semantics of get_filename - particularly
where we're not actually dealing with real filenames. My initial stab
at this would be:

A module name is a dot-separated list of parts.
A filename is an arbitrary token that can be used with get_data to get
the module content. However, the following rules should be followed:
- Filenames should be made up of parts separated by the OS path separator.
- For packages, the final section of the filename *must* be
__init__.py if the standard package detection is being used.
- The initial part of the filename needs to match your path entry if
submodule lookups are going to work sanely

In practice, you need to implement filenames as if your finder is
managing a virtual filesystem mounted at your sys.path entry, with
module->filename semantics being the usual subdirectory layout. And
packages have a basename of __init__.py.

I'd like to know how to implement packages without the artificial
__init__.py (something like a sqlite database can attach content and
an "is_package" flag to the same entry). But that's advanced usage,
and I can probably hack around until I work out how to do that now.

>> The documentation on what I
>> need to return from there is very sparse... In the end I worked out
>> that for a package, I need to return (MyLoader(modulename,
>> 'foo/__init__.py'), ['foo']) (here, "foo" is my dummy marker for my
>> example).
>
> The second argument should just be None: "An empty list can be used for
> portion to signify the loader is not part of a [namespace] package".
> Unfortunately a key word is missing in that sentence.
> http://bugs.python.org/issue17567

Ha. Yes, that makes a lot of difference :-) Did you mean None or [], by the way?

>> In essence, PathEntryFinder really has to implement some
>> form of virtual filesystem mount point, and preserve the standard
>> filesystem semantics of modules having a filename of .../__init__.py.
>
> Well, if your zip file decided to create itself with a different file
> extension then it wouldn't be required, but then other people's code might
> break if they don't respect module abstractions (i.e. looking at
> __package__/__name__ or __path__ to see if something is a package).

I'm not quite sure what you mean by this, but I take your point about
making sure to break people's expectations as little as possible...

>> So I managed to work out what was needed in the end, but it was a lot
>> harder than I'd expected. On reflection, getting the finder semantics
>> right (and in particular the path entry finder semantics) was the hard
>> bit.
>
> Yep, that bit has had the least API tweaks as most people don't muck with
> finders but with loaders.

Hmm. I'm not sure how you can ever write a loader without needing to
write an associated finder. The existing finders wouldn't return your
loader, surely?

>> I'm now 100% sure that some cookbook examples would help a lot. I'll
>> see what I can do.
>
> I plan on writing a pure Python zip importer for Python 3.4 which should be
> fairly minimal and work out as a good example chunk of code.  And no one
> need bother writing it as I'm going to do it myself regardless to make sure
> I plug any missing holes in the API. If you really want something to try for
> fun go for a sqlite3-backed setup (don't see it going in the stdlib but it
> would be a project to have).

I'm pretty sure I'll write a zip importer first - it feels like one of
those essential but largely useless exercises that people have to
start with - a bit like scales on the piano :-) But I'd be interested
in trying a sqlite importer as well. I might well see how I go with
that.

Thanks for the help with this.
Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Safely importing zip files with C extensions

[Christian Heimes]

Am 28.03.2013 17:09, schrieb Brett Cannon:
> Which must be done carefully to prevent a security issue. It shouldn't
> be unzipped anywhere but into a directory only writable by the process. 

Cleanup is going to be tricky or even impossible. Windows locks loaded
DLLs and therefore prevents their removal. It's possible to unload DLLs
but I don't know the implications.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] noob contributions to unit tests

[Daniel Wozniak]

Sean,

During the PyCon sprints I was helping work on unittests in urllib. I 
think as it stands right now urllib/error.py and urllib/parse.py are at 
100% line coverage. I have some additions to urllib/request.py which I 
have yet to submit a patch for (anything above line 700 is covered thus 
far) and I noticed there is a large chunk of code there which has line 
coverage when running tests on OSX. There is a note to refactor that to 
run on all platforms since there is nothing OSX specific in those tests. 
I believe if those tests were running on all platforms it would 
drastically increase line coverage for request.py, assuming your not on 
OSX of coarse. I have not looked at response.py or robotparser.py yet.


Just wanted to give you a little brain dump in case it can save you some 
time.


~Daniel

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Writing importers and path hooks

[Brett Cannon]

On Thu, Mar 28, 2013 at 12:33 PM, Paul Moore  wrote:

> On 28 March 2013 16:08, Brett Cannon  wrote:
> > You only need SourceLoader since you are dealing with Python source. You
> > don't need FileLoader since you are not reading from disk but an
> in-memory
> > zipfile.
> >
> > You should be implementing get_data, get_filename, and path_stats for
> > SourceLoader.
>
> OK, cool. That helps a lot.
>
> The biggest gap here is that I don't think that anywhere has a good
> explanation of the required semantics of get_filename - particularly
> where we're not actually dealing with real filenames.


It's because there aren't any. =) This is the first time alternative
storage mechanisms are really easily viable without massive amounts of
work, so no one has figured this out. The real question is how code out in
the wild would react if you did something like /path/to/sqlite3:pkg.mod
which is very much not a file path.


> My initial stab
> at this would be:
>
> A module name is a dot-separated list of parts.
> A filename is an arbitrary token that can be used with get_data to get
> the module content. However, the following rules should be followed:
> - Filenames should be made up of parts separated by the OS path separator.
>

And why is that? A database doesn't need those separators as the module
name would just be the primary key.


> - For packages, the final section of the filename *must* be
> __init__.py if the standard package detection is being used.
>

Once again, why? A column in a database that is nothing more than a package
flag would solve this as well, negating the need for this. The whole point
of is_package() on loaders is to get away from this reliance on __file__
having any meaning beyond "this is the string that represents where this
module's code was loaded from".


> - The initial part of the filename needs to match your path entry if
> submodule lookups are going to work sanely
>

When applicable that's fine.


>
> In practice, you need to implement filenames as if your finder is
> managing a virtual filesystem mounted at your sys.path entry, with
> module->filename semantics being the usual subdirectory layout. And
> packages have a basename of __init__.py.
>

That's one way of doing it, but it does very much tie imports to files and
it doesn't generalize the concept to places where file paths simply do not
need to apply.


>
> I'd like to know how to implement packages without the artificial
> __init__.py (something like a sqlite database can attach content and
> an "is_package" flag to the same entry). But that's advanced usage,
> and I can probably hack around until I work out how to do that now.
>

Define is_package(). I personally want to change the API somehow so you ask
for what __path__ should be set to. Unfortunately without going down the
"False means not a package, everything else means it is and what is
returned should be set on __path__" is a bit hairy and not
backwards-compatible unless you require a list that always evaluates to
True for packages.


>
> >> The documentation on what I
> >> need to return from there is very sparse... In the end I worked out
> >> that for a package, I need to return (MyLoader(modulename,
> >> 'foo/__init__.py'), ['foo']) (here, "foo" is my dummy marker for my
> >> example).
> >
> > The second argument should just be None: "An empty list can be used for
> > portion to signify the loader is not part of a [namespace] package".
> > Unfortunately a key word is missing in that sentence.
> > http://bugs.python.org/issue17567
>
> Ha. Yes, that makes a lot of difference :-) Did you mean None or [], by
> the way?
>

Empty list. You can check the code to see if it would work with None, but a
list is expected to be used so an empty list is more consistent and still
false.


>
> >> In essence, PathEntryFinder really has to implement some
> >> form of virtual filesystem mount point, and preserve the standard
> >> filesystem semantics of modules having a filename of .../__init__.py.
> >
> > Well, if your zip file decided to create itself with a different file
> > extension then it wouldn't be required, but then other people's code
> might
> > break if they don't respect module abstractions (i.e. looking at
> > __package__/__name__ or __path__ to see if something is a package).
>
> I'm not quite sure what you mean by this, but I take your point about
> making sure to break people's expectations as little as possible...
>

To tell if a module is a package, you should do either ``if mod.__name__ ==
mod.__package__`` or ``if hasattr(mod, '__path__')``.


>
> >> So I managed to work out what was needed in the end, but it was a lot
> >> harder than I'd expected. On reflection, getting the finder semantics
> >> right (and in particular the path entry finder semantics) was the hard
> >> bit.
> >
> > Yep, that bit has had the least API tweaks as most people don't muck with
> > finders but with loaders.
>
> Hmm. I'm not sure how you can ever write a loader without needin

Re: [Python-Dev] noob contributions to unit tests

[Walter Dörwald]

Am 27.03.2013 um 03:24 schrieb R. David Murray :

> On Tue, 26 Mar 2013 16:59:06 -0700, Maciej Fijalkowski  
> wrote:
>> On Tue, Mar 26, 2013 at 4:49 PM, Sean Felipe Wolfe  
>> wrote:
>>> Hey everybody how are you all :)
>>> 
>>> I am an intermediate-level python coder looking to get help out. I've
>>> been reading over the dev guide about helping increase test coverage
>>> -->
>>> http://docs.python.org/devguide/coverage.html
>>> 
>>> And also the third-party code coverage referenced in the devguide page:
>>> http://coverage.livinglogic.de/
>>> 
>>> I'm seeing that according to the coverage tool, two of my favorite
>>> libraries, urllib/urllib2, have no unit tests? Is that correct or am I
>>> reading it wrong?
>>> 
>>> If that's correct it seems like a great place perhaps for me to cut my
>>> teeth and I would be excited to learn and help out here.
>>> 
>>> And of course any thoughts or advice for an aspiring Python
>>> contributor would be appreciated. Of course the dev guide gives me
>>> plenty of good info.
>>> 
>>> Thanks!
>> 
>> That looks like an error in the coverage report, there are certainly
>> urllib and urllib2 tests in test/test_urllib*
> 
> The devguide contains instructions for running coverage yourself,
> and if I recall correctly the 'fullcoverage' recipe does a better
> job than what runs at coverage.livinglogic.de.

The job that produces that output has been broken for some time now, and I 
haven't found the time to look into it. If someone wants to try, here's the 
code:

   https://pypi.python.org/pypi/pycoco/0.7.2

> […]

Servus,
   Walter

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Can I introspect/reflect to get arguments exec()?

[PJ Eby]

On Thu, Mar 28, 2013 at 6:43 AM, Rocky Bernstein  wrote:
> Of course the debugger uses sys.settrace too, so the evil-ness of that is
> definitely not a concern. But possibly I need to make sure that since the
> DecoratorTools and the debugger both hook into trace hooks they play nice
> together and fire in the right order.

DecoratorTools' trace hooking is unrelated to its linecache
functionality.  All you need from it is the cache_source() function;
you can pretty much ignore everything else for your purposes.  You'll
just need to give it a phony filename to work with, and the associated
string.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] cpython (2.7): Issue 17538: Document XML vulnerabilties

[Ezio Melotti]

Hi,

On Tue, Mar 26, 2013 at 6:53 PM, christian.heimes
 wrote:
> http://hg.python.org/cpython/rev/e87364449954
> changeset:   82973:e87364449954
> branch:  2.7
> parent:  82963:d321885ff8f3
> user:Christian Heimes 
> date:Tue Mar 26 17:53:05 2013 +0100
> summary:
>   Issue 17538: Document XML vulnerabilties
>
> [...]
>
> diff --git a/Doc/library/xml.rst b/Doc/library/xml.rst
> new file mode 100644
> --- /dev/null
> +++ b/Doc/library/xml.rst
> @@ -0,0 +1,131 @@
> +.. _xml:
> +
> +XML Processing Modules
> +==
> +
> +.. module:: xml
> +   :synopsis: Package containing XML processing modules
> +.. sectionauthor:: Christian Heimes 
> +.. sectionauthor:: Georg Brandl 
> +
> +
> +Python's interfaces for processing XML are grouped in the ``xml`` package.
> +
> +.. warning::
> +
> +   The XML modules are not secure against erroneous or maliciously
> +   constructed data.  If you need to parse untrusted or unauthenticated data 
> see
> +   :ref:`xml-vulnerabilities`.
> +
> +It is important to note that modules in the :mod:`xml` package require that
> +there be at least one SAX-compliant XML parser available. The Expat parser is
> +included with Python, so the :mod:`xml.parsers.expat` module will always be
> +available.
> +
> +The documentation for the :mod:`xml.dom` and :mod:`xml.sax` packages are the
> +definition of the Python bindings for the DOM and SAX interfaces.
> +
> +The XML handling submodules are:
> +
> +* :mod:`xml.etree.ElementTree`: the ElementTree API, a simple and lightweight

Something is missing here ^

> +
> +..
> +
> +* :mod:`xml.dom`: the DOM API definition
> +* :mod:`xml.dom.minidom`: a lightweight DOM implementation
> +* :mod:`xml.dom.pulldom`: support for building partial DOM trees
> +
> +..
> +
> +* :mod:`xml.sax`: SAX2 base classes and convenience functions
> +* :mod:`xml.parsers.expat`: the Expat parser binding
> +
> +
> +.. _xml-vulnerabilities:
> +
> [...]
> +
> +defused packages
> +
> +
> +`defusedxml`_ is a pure Python package with modified subclasses of all stdlib
> +XML parsers that prevent any potentially malicious operation. The courses of
> +action are recommended for any server code that parses untrusted XML data.

This last sentence doesn't make much sense to me.  Is it even correct?

> The
> +package also ships with example exploits and an extended documentation on 
> more
> +XML exploits like xpath injection.
> +
> +`defusedexpat`_ provides a modified libexpat and patched replacment

s/replacment/replacement/

> +:mod:`pyexpat` extension module with countermeasures against entity expansion
> +DoS attacks. Defusedexpat still allows a sane and configurable amount of 
> entity
> +expansions. The modifications will be merged into future releases of Python.
> +
> +The workarounds and modifications are not included in patch releases as they
> +break backward compatibility. After all inline DTD and entity expansion are
> +well-definied XML features.

s/definied/defined/

> +
> +
> +.. _defusedxml: 
> +.. _defusedexpat: 
> +.. _Billion Laughs: http://en.wikipedia.org/wiki/Billion_laughs
> +.. _ZIP bomb: http://en.wikipedia.org/wiki/Zip_bomb
> +.. _DTD: http://en.wikipedia.org/wiki/Document_Type_Definition
>  [...]

Best Regards,
Ezio Melotti
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Safely importing zip files with C extensions

[Gregory P. Smith]

On Thu, Mar 28, 2013 at 9:09 AM, Brett Cannon  wrote:

>
>
>
> On Thu, Mar 28, 2013 at 10:44 AM, Thomas Heller wrote:
>
>> Am 27.03.2013 20:38, schrieb Vinay Sajip:
>>
>>  This quote is here to stop GMane complaining that I'm top-posting.
 Ignore.

>>>
>>> I've already posted this to distutils-sig, but thought that it might be
>>> of
>>> interest to readers here as it relates to importing C extensions ...
>>>
>>> zipimport is great, but there can be issues importing software that
>>> contains
>>> C extensions. But the new wheel format (PEP 427) may give us a better
>>> way of
>>> importing zip files containing C extensions. Since wheels are .zip
>>> files, they
>>> can sometimes be used to provide functionality without needing to be
>>> installed.
>>> But whereas .zip files contain no convention for indicating
>>> compatibility with
>>> a particular Python, wheels do contain this compatibility information.
>>> Thus, it
>>> is possible to check if a wheel can be directly imported from, and the
>>> wheel
>>> support in distlib allows you to take advantage of this using the
>>> mount() and
>>> unmount() methods. When you mount a wheel, its absolute path name is
>>> added to
>>> sys.path, allowing the Python code in it to be imported. (A
>>> DistlibException is
>>> raised if the wheel isn't compatible with the Python which calls the
>>> mount()
>>> method.)
>>>
>>
>> The zip-file itself could support importing compiled extensions when it
>> contains a python-wrapper module that unpacks the .so/.dll file somewhere,
>> and finally calls imp.load_dynamic() to import it and replace itself.
>
>
> Which must be done carefully to prevent a security issue. It shouldn't be
> unzipped anywhere but into a directory only writable by the process.
>
>
Once http://sourceware.org/bugzilla/show_bug.cgi?id=11767 is implemented
and available in libc, no extraction of .so's should be needed (they will
likely need to be stored uncompressed in the .zip file for that though).
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com