Re: [Python-Dev] PEP: Collecting information about git

[PJ Eby]

On Sat, Sep 12, 2015 at 9:54 AM, Oleg Broytman  wrote:
> The plan is to extend the PEP in the future collecting information
> about equivalence of Mercurial and git scenarios to help migrating
> Python development from Mercurial to git.

I couldn't find any previous discussion about this, but I figure I
should mention:

If the motivation here is to get away from the often-awful bitbucket
to the friendlier and more-popular Github, then it might be useful to
know that hg-git works beautifully with Github.  I have over a dozen
open source projects on Github that I manage entirely using hg command
lines without having yet touched git at all.  Even the forks and pull
requests I've done of others' projects on Github worked just fine, so
long as I remember to use hg bookmarks instead of hg branches.

It's possible there are things you can't do with Mercurial on Github,
but I haven't encountered one thus far.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Minimal async event loop and async utilities (Was: PEP 492: async/await in Python; version 4)

[PJ Eby]

On Mon, May 11, 2015 at 6:05 PM, Guido van Rossum gu...@python.org wrote:
 OTOH you may look at micropython's uasyncio -- IIRC it doesn't have Futures
 and it definitely has I/O waiting.

Here's a sketch of an *extremely* minimal main loop that can do I/O
without Futures, and might be suitable as a PEP example.  (Certainly,
it would be hard to write a *simpler* example than this, since it
doesn't even use any *classes* or require any specially named methods,
works with present-day generators, and is (I think) both 2.x/3.x
compatible.)

coroutines = [] # round-robin of currently running coroutines

def schedule(coroutine, val=None, err=None):
coroutines.insert(0, (coroutine, val, err))

def runLoop():
while coroutines:
(coroutine, val, err) = coroutines.pop()
try:
if err is not None:
suspend = coroutine.throw(err)
else
suspend = coroutine.send(val)
except StopIteration:
# coroutine is finished, so don't reschedule it
continue

except Exception:
# framework-specific detail  (i.e., log it, send
# to an error handling coroutine, or just stop the program
# Here, we just ignore it and stop the coroutine
continue

else:
if hasattr(suspend, '__call__') and suspend(coroutine):
continue
else:
# put it back on the round-robin list
schedule(coroutine)

To use it, `schedule()` one or more coroutines, then call `runLoop()`,
which will run as long as there are things to do.  Each coroutine
scheduled must yield *thunks*: callable objects that take a coroutine
as a parameter, and return True if the coroutine should be suspended,
or False if it should continue to run.  If the thunk returns true,
that means the thunk has taken responsibility for arranging to
`schedule()` the coroutine with a value or error when it's time to
send it the result of the suspension.

You might be asking, wait, but where's the I/O?  Why, in a
coroutine, of course...

readers = {}
writers = {}
timers = []

def readable(fileno):
yield readable(fileno) resumes when fileno is readable
def suspend(coroutine):
readers[fileno] = coroutine
return True
return suspend

def writable(fileno):
yield writable(fileno) resumes when fileno is writable
def suspend(coroutine):
writers[fileno] = coroutine
return True
return suspend

def sleepFor(seconds):
yield sleepFor(seconds) resumes after that much time
return suspendUntil(time.time() + seconds)

def suspendUntil(timestamp):
yield suspendUntil(timestamp) resumes when that time is reached
def suspend(coroutine)
heappush(timers, (timestamp, coroutine)
return suspend

   def doIO():
while coroutines or readers or writers or timers:

# Resume scheduled tasks
while timers and timers[0][0] = time.time():
ts, coroutine = heappop(timers)
schedule(coroutine)

if readers or writers:
if coroutines:
# Other tasks are running; use minimal timeout
timeout = 0.001
else if timers:
timeout = max(timers[0][0] - time.time(), 0.001)
else:
timeout = 0 # take as long as necessary
r, w, e = select(readers, writers, [], timeout)
for rr in r: schedule(readers.pop(rr))
for ww in w: schedule(writers.pop(ww))

yield   # allow other coroutines to run

schedule(doIO())  # run the I/O loop as a coroutine

(This is painfully incomplete for a real framework, but it's a rough
sketch of how one of peak.events' first drafts worked, circa early
2004.)

Basically, you just need a coroutine whose job is to resume coroutines
whose scheduled time has arrived, or whose I/O is ready.  And of
course, some data structures to keep track of such things, and an API
to update the data structures and suspend the coroutines.  The I/O
loop exits once there are no more running tasks and nothing waiting on
I/O...  which will also exit the runLoop.  (A bit like a miniature
version of NodeJS for Python.)

And, while you need to preferably have only *one* such I/O coroutine
(to prevent busy-waiting), the I/O coroutine is completely
replaceable.  All that's required to implement one is that the core
runloop expose the count of active coroutines.  (Notice that, apart
from checking the length of `coroutines`, the I/O loop shown above
uses only the public `schedule()` API and the exposed thunk-suspension
protocol to do its thing.)

Also, note that you *can* indeed have multiple I/O 

Re: [Python-Dev] async/await in Python; v2

[PJ Eby]

On Tue, Apr 21, 2015 at 1:26 PM, Yury Selivanov yselivanov...@gmail.com wrote:
 It is an error to pass a regular context manager without ``__aenter__``
 and ``__aexit__`` methods to ``async with``.  It is a ``SyntaxError``
 to use ``async with`` outside of a coroutine.

I find this a little weird.  Why not just have `with` and `for` inside
a coroutine dynamically check the iterator or context manager, and
either behave sync or async accordingly?  Why must there be a
*syntactic* difference?

Not only would this simplify the syntax, it would also allow dropping
the need for `async` to be a true keyword, since functions could be
defined via def async foo(): rather than async def foo():

...which, incidentally, highlights one of the things that's been
bothering me about all this async foo stuff: async def looks like
it *defines the function* asynchronously (as with async with and
async for), rather than defining an asynchronous function.  ISTM it
should be def async bar(): or even def bar() async:.

Also, even that seems suspect to me: if `await` looks for an __await__
method and simply returns the same object (synchronously) if the
object doesn't have an await method, then your code sample that
supposedly will fail if a function ceases to be a coroutine *will not
actually fail*.

In my experience working with coroutine systems, making a system
polymorphic (do something appropriate with what's given) and
idempotent (don't do anything if what's wanted is already done) makes
it more robust.  In particular, it eliminates the issue of mixing
coroutines and non-coroutines.

To sum up: I can see the use case for a new `await` distinguished from
`yield`, but I don't see the need to create new syntax for everything;
ISTM that adding the new asynchronous protocols and using them on
demand is sufficient.  Marking a function asynchronous so it can use
asynchronous iteration and context management seems reasonably useful,
but I don't think it's terribly important for the type of function
result.  Indeed, ISTM that the built-in `object` class could just
implement `__await__` as a no-op returning self, and then *all*
results are trivially asynchronous results and can be awaited
idempotently, so that awaiting something that has already been waited
for is a no-op.  (Prior art: the Javascript Promise.resolve() method,
which takes either a promise or a plain value and returns a promise,
so that you can write code which is always-async in the presence of
values that may already be known.)

Finally, if the async for and with operations have to be distinguished
by syntax at the point of use (vs. just always being used in
coroutines), then ISTM that they should be `with async foo:` and `for
async x in bar:`, since the asynchronousness is just an aspect of how
the main keyword is executed.

tl;dr: I like the overall ideas but hate the syntax and type
segregation involved: declaring a function async at the top is OK to
enable async with/for semantics and await expressions, but the rest
seems unnecessary and bad for writing robust code.  (e.g. note that
requiring different syntax means a function must either duplicate code
or restrict its input types more, and type changes in remote parts of
the program will propagate syntax changes throughout.)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 487 vs 422 (dynamic class decoration)

[PJ Eby]

On Sat, Apr 4, 2015 at 9:33 PM, Nick Coghlan ncogh...@gmail.com wrote:
 So actually reading https://gist.github.com/pjeby/75ca26f8d2a7a0c68e30
 properly, you're starting to convince me that a noconflict metaclass
 resolver would be a valuable and viable addition to the Python 3 type
 system machinery.

 The future possible language level enhancement would then be to make
 that automatic resolution of metaclass conflicts part of the *default*
 metaclass determination process. I realise you've been trying to
 explain that to me for a few days now, I'm just writing it out
 explicitly to make it clear I finally get it :)

I'm glad you got around to reading it.  Sometimes it's really
frustrating trying to get things like that across.

What's funny is that once I actually 1) wrote that version, and 2)
ended up doing a version of six's with_metaclass() function so I could
write 2/3 mixed code in DecoratorTools, I realized that there isn't
actually any reason why I can't write a Python 2 version of
noconflict.  Indeed, with a slight change to eliminate ClassType from
the metaclass candidate list, the Python 3 version would also work as
the Python 2 version: just use it as the explicit __metaclass__, or
use with_metaclass, i.e.:

class something(base1, base2, ...):
__metaclass__ = noconflict

# ...

or:

class something(with_metaclass(noconflict, base1, base2, ...)):
# ...

And the latter works syntactically from Python 2.3 on up.


 My apologies for that - while I don't actually recall what I was
 thinking when I said it, I suspect I was all fired up that PEP 422 was
 definitely the right answer, and hence thought I'd have an official
 solution in place for you in fairly short order. I should have let you
 know explicitly when I started having doubts about it, so you could
 reassess your porting options.

Well, at least it's done now.  Clearing up the issue allowed me to
spend some time on porting some of the relevant libraries this
weekend, where I promptly ran into challenges with several of the
*other* features removed from Python 3 (like tuple arguments), but
fortunately those are issues more of syntactic convenience than
irreplaceable functionality.  ;-)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 487 vs 422 (dynamic class decoration)

[PJ Eby]

On Fri, Apr 3, 2015 at 8:44 AM, Martin Teichmann
lkb.teichm...@gmail.com wrote:
 This proposal can actually be seen as an extension to the __class__
 and super() mechanism of normal methods: methods currently have the
 priviledge to know which classes they are defined in, while descriptors
 don't. So we could unify all this by giving functions a __post_process__
 method which sets the __class__ in the function body. This is about the
 same as what happened when functions got a __get__ method to turn
 them into object methods.

 While this all is in the making, PJ could monkey-patch __build_class__
 to do the steps described above, until it gets accepted into cpython.
 So I pose the question to PJ: would such an approach solve the
 problems you have?

Universal member post-processing actually works *better* for the
motivating use case than the metaclass or class level hooks, so yes.

In practice, there is one potential hiccup, and that's that decorators
which aren't aware of __post_process__ will end up masking it.  But
that's not an insurmountable obstacle.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 487 vs 422 (dynamic class decoration)

[PJ Eby]

On Fri, Apr 3, 2015 at 11:04 AM, Nick Coghlan ncogh...@gmail.com wrote:
 Extending the descriptor protocol to include a per-descriptor hook that's
 called at class definition time sounds like a potentially nice way to go to
 me. While you *could* still use it to arbitrarily mutate the class object,
 it's much clearer that's not the intended purpose, so I don't see it as a
 major problem.

Just to be clear, mutating the class object was never the point for my
main use case that needs the PEP 422 feature; it was for method
overloads that are called remotely and need to be registered
elsewhere.  For some of my other use cases, adding metadata to the
class is a convenient way to do things, but classes are generally
weak-referenceable so the add-on data can be (and often is) stored in
a weak-key dictionary rather than placed directly on the class.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 487 vs 422 (dynamic class decoration)

[PJ Eby]

On Fri, Apr 3, 2015 at 4:21 AM, Nick Coghlan ncogh...@gmail.com wrote:
 That means I'm now OK with monkeypatching __build_class__ being the
 only way to get dynamic hooking of the class currently being defined
 from the class body - folks that really want that behaviour can
 monkeypatch it in, while folks that think it's a bad idea don't need
 to worry about.

I'd still prefer to only do that as an emulation of an agreed-upon
descriptor notification protocol, such that it's a backport of an
approved PEP, so I hope we can work that out.  But I guess if not,
then whatever works.  I just wish you'd been okay with it in 2012, as
there was more than once in the last few years where I had some
downtime and thought about trying to do some porting work.  :-(

And in the meantime, the only alternative Python implementation I know
of that's made *any* headway on Python 3 in the last few years (i.e.,
PyPy 3) *includes* a compatibly monkeypatchable __build_class__.  It
appears that the *other* obstacles to making a compatible Python 3
implementation are a lot tougher for implementers to get over than
compatibility with __build_class__.  ;-)


 Neither PEP 422 nor 487 are designed to eliminate metaclass conflicts
 in general, they're primarily designed to let base classes run
 arbitrary code after the namespace has been executed in a subclass
 definition *without* needing a custom metaclass.

And yet the argument was being made that the lack of custom metaclass
was a feature because it avoided conflict.  I'm just trying to point
out that if avoiding conflict is desirable, building *every possible
metaclass feature* into the Python core isn't a scalable solution.  At
this point, co-operative inheritance is a well-understood model in
Python, so providing an API to automatically mix metaclasses
(explicitly, at first) seems like a good step towards solving the
metaclass conflict problem in general.

When Guido introduced the new MRO scheme in Python 2.2, he noted that
the source he'd gotten that scheme from had explained that it could be
extended to automatically mixing metaclasses, but he (Guido) didn't
want to do that in Python until more experience was had with the new
MRO scheme in general.  And I think we have enough experience with
that *now*, to be able to take a step forward, by providing a
stdlib-blessed metaclass mixer.  It not only makes the prototype,
PyPI-based version of PEP 487 more usable immediately, it will also
encourage people to develop metaclasses as *mixins* rather than
one-size-fits-all monoliths.

For example, there's no reason that both of PEP 487''s features need
to live in the *same* metaclass, if you could trivially mix
metaclasses at the point where you inherit from bases with different
metaclasses.  (And eventually, a future version of Python could do the
mixing automatically, without the `noconflict` function.  The theory
was well-understood for other languages, after all, before Python 2.2
even came out.)


 No, you can't do it currently without risking a backwards
 incompatibility through the introduction of a custom metaclass.

Right...  which is precisely why I'm suggesting  the `noconflict()`
metaclass factory function as a *general* solution for providing
useful metaclasses, and why I think that PEP 487 should break the
namespacing and subclass init features into separate metaclasses, and
add that noconflict feature.  It will then become a good example for
people moving forward writing metaclasses.

Basically, as long as you don't have the pointless conflict errors,
you can write co-operative metaclass mixins as easily as you can write
regular co-operative mixins.  I was missing this point myself because
I've been too steeped in Python 2's complexities: writing a usable
version of `noconflict()` is a lot more complex and its invocation far
more obscure.  In Python 2, there's classic classes, class- and
module-level __metaclass__, ExtensionClass, and all sorts of other
headaches for automatic mixing.  In Python 3, though, all that stuff
goes out the window, and even my 90-line version that's almost half
comments is probably still overengineered compared to what's actually
needed to do the mixing.


 Further, if the claim is that metaclass conflict potential makes PEP
 487 worthy of a language change, then by the same logic method
 decorators are just as worthy of a language change, since any mixin
 required to use a method decorator would be *just as susceptible* to
 metaclass conflicts as SubclassInit.

 There wouldn't be a custom metaclass involved in the native
 implementation of PEP 487, only in the backport.

Right...  and if there were a native implementation of PEP 422, that
would also be the case for PEP 422.  The point is that if the PEP 487
can justify a *language* change to avoid needing a metaclass, then
arguably PEP 422 has an even *better* justification, because its need
to avoid needing a metaclass is at least as strong.  Indeed, you said
the same yourself as recently as 

Re: [Python-Dev] PEP 487 vs 422 (dynamic class decoration)

[PJ Eby]

On Wed, Apr 1, 2015 at 10:39 PM, Nick Coghlan ncogh...@gmail.com wrote:
 On 2 April 2015 at 07:35, PJ Eby p...@telecommunity.com wrote:
 I recently got an inquiry from some of my users about porting some of
 my libraries to Python 3 that make use of the Python 2 __metaclass__
 facility.  While checking up on the status of PEP 422 today, I found
 out about its recently proposed replacement, PEP 487.

 While PEP 487 is a generally fine PEP, it actually *rules out* the
 specific use case that I wanted PEP 422 for in the first place:
 dynamic addition of callbacks or decorators for use at class creation
 time without requiring explicit inheritance or metaclass
 participation.  (So that e.g. method decorators can access the
 enclosing class at class definition time.)

 How hard is the requirement against relying on a mixin class or class
 decorator to request the defining class aware method decorator
 support? Is the main concern with the fact that failing to apply the
 right decorator/mixin at the class level becomes a potentially silent
 failure where the class aware method decorators aren't invoked
 properly?

The concern is twofold: it breaks proper information hiding/DRY, *and*
it fails silently.  It should not be necessary for clients of package
A1 (that uses a decorator built using package B2) to mixin a metaclass
or decorator from package C3 (because B2 implemented its decorators
using C3), just for package A1's decorator to work properly in the
*client package's class*.  (And then, of course, this all silently
breaks if you forget, and the breakage might happen at the A1, B2, or
C3 level.)

Without a way to hook into the class creation process, there is no way
to verify correctness and prevent the error from passing silently.
(OTOH, if there *is* a way to hook into the creation process, the
problem is solved: there's no need to mix anything in anyway, because
the hook can do whatever the mixin was supposed to do.)

The only way PEP 487 could be a solution is if the default
`object.__init_subclass__` supported one of the earlier __decorators__
or __autodecorate__ proposals, or if the PEP were for an
`__init_class__` that operated on the defining class, instead of
operating only on subclasses.  (I need to hook the creation of a class
that's *being defined*, not the definition of its future subclasses.)


 My preference at this point would definitely be to introduce a mixin
 class into the affected libraries and frameworks with an appropriate
 PEP 487 style __init_subclass__ that was a noop in Python 2 (which
 would rely on metaclass injection instead), but implemented the
 necessary defining class aware method decorator support in Python 3.

If this were suitable for the use case, I'd have done it already.
DecoratorTools has had a mixin that provides a __class_init__ feature
since 2007, which could be ported to Python 3 in a straighforward
manner as a third-party module.  (It's just a mixin that provides a
metaclass; under 3.x it could probably just be a plain metaclass with
no mixin.)


 The question of dynamically injecting additional base classes from the
 class body to allow the use of certain method decorators to imply
 specific class level behaviour could then be addressed as a separate
 proposal (e.g. making the case for an __append_mixins__ attribute),
 rather than being linked directly to the question of how we going
 about defining inherited creation time behaviour without needing a
 custom metaclass.

Then maybe we should do that first, since PEP 487 doesn't do anything
you can't *already* do with a mixin, all the way back to Python 2.2.

IOW, there's no need to modify the core just to have *that* feature,
since if you control the base class you can already do what PEP 487
does in essentially every version of Python, ever.  If that's all PEP
487 is going to do, it should just be a PyPI package on a
stdlib-inclusion track, not a change to core Python.  It's not
actually adding back any of the dynamicness (dynamicity?
hookability?) that PEP 3115 took away.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 487 vs 422 (dynamic class decoration)

[PJ Eby]

On Thu, Apr 2, 2015 at 1:42 PM, PJ Eby p...@telecommunity.com wrote:
 If the PEP 487 metaclass library,
 however, were to just port some bits of my code to Python 3 this could
 be a done deal already and available in *all* versions of Python 3,
 not just the next one.

Just for the heck of it, here's an actual implementation and demo of
PEP 487, that I've tested with 3.1, 3.2, and 3.4 (I didn't have a copy
of 3.3 handy):

https://gist.github.com/pjeby/75ca26f8d2a7a0c68e30

The first module is just a demo that shows the features in use.  The
second module is the implementation.

Notice that the actual *functionality* of PEP 487 is just *16 lines*
in Python 3...  including docstrings and an `__all__` definition.  ;-)

The other 90 lines of code are only there to implement the
`noconflict` feature for fixing metaclass conflicts... and quite a lot
of *those* lines are comments and docstrings.  ;-)

Anyway, I think this demo is a knockout argument for why PEP 487
doesn't need a language change: if you're writing an __init_subclass__
method you just include the `pep487.init_subclasses` base in your base
classes, and you're done.  It'll silently fail if you leave it out
(but you'll notice that right away), and it *won't* fail in
third-party subclasses because the *third party* didn't include it.

In contrast, PEP 422 provided a way to have both the features
contemplated by 487, *and* a way to allow method-level decorators to
discover the class at class creation time.  If there's going to be a
language change, it should include that latter feature from the
outset.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 487 vs 422 (dynamic class decoration)

[PJ Eby]

On Thu, Apr 2, 2015 at 9:31 PM, Nick Coghlan ncogh...@gmail.com wrote:
 On 3 April 2015 at 08:24, Martin Teichmann lkb.teichm...@gmail.com wrote:
 However, I'm also now wondering if it may be possible to reach out to
 the pylint authors (similar to what Brett did for the pylint --py3k
 flag) and ask for a way to make it easy to register base class,
 decorator pairs where pylint will complain if it sees a particular
 method decorator but can't determine at analysis time if the named
 base class is in the MRO for the class defining the method.

Will it *also* check the calling chain of the decorator, or any other
thing that's called or invoked in the class body,to find out if
somewhere, somehow, it asks for a class decoration?  If not, it's not
going to help with this use case.

There are many ways to solve this problem by re-adding a hook -- you
and I have proposed several, in 2012 and now.

There are none, however, which do not involve putting back the
hookability that Python 3 took out, except by using hacks like
sys.set_trace() or monkeypatching __build_class__.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 487 vs 422 (dynamic class decoration)

[PJ Eby]

On Thu, Apr 2, 2015 at 6:24 PM, Martin Teichmann
lkb.teichm...@gmail.com wrote:
 The whole point of PEP 487 was to reduce PEP 422 so much that
 it can be written in python and back-ported.

As I said earlier, it's a fine feature and should be in the stdlib for
Python 3.  (But it should have a `noconflict` feature added, and it
doesn't need a language change.)

However, since my specific use case was the one PEP 422 was originally
written to solve, and PEP 487 does not address that use case, it is
not a suitable substitute *for PEP 422*.

This is also not your fault; you didn't force Nick to withdraw it,
after all.  ;-)

My main concern in this thread, however, is ensuring that either the
use case behind PEP 422 doesn't get dropped, or that Nick is now okay
with me implementing that feature by monkeypatching __build_class__.
Since he practically begged me not to do that in 2012, and IIRC
*specifically created* PEP 422 to provide an alternative way for me to
accomplish this *specific* use case, I wanted to see what his current
take was.  (That is, did he forget the history of the PEP, or does he
no longer care about userspace code hooking __build_class__?  Is there
some other proposal that would be a viable alternative? etc.)


 Now you want to be able to write decorators whose details
 are filled in at class creation time.

Not now; it's been possible to do this in Python 2 for over a
decade, and code that does so is in current use by other packages.
The package providing this feature (DecoratorTools) was downloaded 145
times today, and 3274 times in the past month, so there is active,
current use of it by other Python 2 packages.  (Though I don't know
how many of them depend directly or indirectly upon this particular
feature.)

Currently, however, it is not possible to port this feature of
DecoratorTools (or any other package that uses that feature,
recursively) to Python 3, due to the removal of __metaclass__ and the
lack of any suitable substitute hook.


 Your point is that you want to be able to use your decorators
 without having to ask users to also inherit a specific class.
 I personally don't think that's desirable.  Many frameworks out
 there have such kind of decorators and mandatory base classes
 and that works fine.

The intended use case is for generic method decorators that have
nothing to do with the base class per se, so inheriting from a
specific base-class is an anti-feature in this case.


 The only problem remains once you need to
 inherit more than one of those classes, as their metaclasses
 most likely clash. This is what PEP 487 fixes.

No, it addresses the issue for certain *specific* metaclass use cases.
It does not solve the problem of metaclass conflict in general; for
that you need something like the sample `noconflict` code I posted,
which works for Python 3.1+ and doesn't require a language change.


 So my opinion is that it is not too hard a requirement to ask
 a user to inherit a specific mixin class for the sake of using
 a decorator.

If this logic were applied to PEP 487 as it currently stands, the PEP
should be rejected, since its use case is even *more* easily
accomplished by inheriting from a specific mixin class.  (Since the
feature only works on subclasses anyway!)

Further, if the claim is that metaclass conflict potential makes PEP
487 worthy of a language change, then by the same logic method
decorators are just as worthy of a language change, since any mixin
required to use a method decorator would be *just as susceptible* to
metaclass conflicts as SubclassInit.  (Notably, the stdlib's ABCMeta
is a common cause of metaclass conflicts in Python 2.6+ -- if you mix
in anything that implements an ABC by subclassing it, you will get a
metaclass conflict.)

Finally, I of course disagree with the conclusion that it's okay to
require mixins in order for method decorators to access the containing
class, since it is not a requirement in Python 2, due to the
availability of the __metaclass__ hook.  Further, PEP 422 was
previously approved to fix this problem, and has a patch in progress,
so I'm understandably upset by its sudden withdrawal and lack of
suitable replacement.

So personally, I think that PEP 422 should be un-withdrawn (or
replaced with something else), and PEP 487 should be retargeted
towards defining a `metaclass` module for the stdlib, including a
`noconflict` implementation to address metaclass conflict issues.
(Mine or someone else's, as long as it works.)  PEP 487 should not be
a proposal to change the language, as the provided features don't
require it.  (And it definitely shouldn't pre-empt a separately useful
feature that *does* require a language change.)

At this point, though, I mostly just want to get some kind of closure.
After three years, I'd like to know if this is a yea or nay, so I can
port the thing and move on, whether it's through a standardized
mechanism or ugly monkeypatching.  Honestly, the only reason I'm even
discussing this in the 

Re: [Python-Dev] PEP 487 vs 422 (dynamic class decoration)

[PJ Eby]

On Thu, Apr 2, 2015 at 10:29 PM, Greg Ewing greg.ew...@canterbury.ac.nz wrote:
 On 04/03/2015 02:31 PM, Nick Coghlan wrote:

 If I'm understanding PJE's main concern correctly it's that this
 approach requires explicitly testing that the decorator has been
 applied correctly in your automated tests every time you use it, as
 otherwise there's a risk of a silent failure when you use the
 decorator but omit the mandatory base class that makes the decorator
 work correctly.


 Could the decorator be designed to detect that situation
 somehow? E.g. the first time the decorated method is called,
 check that the required base class is present.

No, because in the most relevant use case, the method will never be
called if the base class isn't present.  For more details, see also
the previous discussion at
https://mail.python.org/pipermail/python-dev/2012-June/119883.html
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 487 vs 422 (dynamic class decoration)

[PJ Eby]

On Thu, Apr 2, 2015 at 4:46 AM, Nick Coghlan ncogh...@gmail.com wrote:
 On 2 April 2015 at 16:38, PJ Eby p...@telecommunity.com wrote:

 IOW, there's no need to modify the core just to have *that* feature,
 since if you control the base class you can already do what PEP 487
 does in essentially every version of Python, ever.  If that's all PEP
 487 is going to do, it should just be a PyPI package on a
 stdlib-inclusion track, not a change to core Python.  It's not
 actually adding back any of the dynamicness (dynamicity?
 hookability?) that PEP 3115 took away.

 The specific feature that PEP 487 is adding is the ability to
 customise creation of subclasses without risking the introduction of a
 metaclass conflict.  That allows it to be used in situations where
 adopting any of the existing metaclass based mechanisms would require
 a potential compatibility break

But metaclass conflicts are *also* fixable in end-user code, and have
been since 2.2.  All you need to do is use a metaclass *function* that
automatically merges the metaclasses involved, which essentially
amounts to doing `class MergedMeta(base1.__class__,
base2.__class__,...)`.  (Indeed, I've had a library for doing just
that since 2002, that originally ran on Python 2.2,.)

On Python 3, it's even easier to use that approach, because you can
just use something like `class whatever(base1, base2,
metaclass=noconflict)` whenever a conflict comes up.  (And because the
implementation wouldn't have to deal with classic classes or
__metaclass__, as my Python 2 implementation has to.)

IOW, *all* of PEP 487 is straightforward to implement in userspace as
a metaclass and a function that already exist off-the-shelf in Python
2...  and whose implementations would be simplified by porting them to
Python 3, and dropping any extraneous features:

* http://svn.eby-sarna.com/PEAK/src/peak/util/Meta.py?view=markup
(the `makeClass` function does what my hypothetical `noconflict` above
does, with a slightly different API, and support for classic classes,
__metaclass__, etc., that could all be stripped out)

* http://svn.eby-sarna.com/DecoratorTools/peak/util/decorators.py?view=markup
 (see the `classy_class` metaclass and `classy` mixin base that
implement features similar to `__init_subclass__`, plus others that
could be stripped out)

Basically, you can pull out those functions/classes (and whatever else
they use in those modules), port 'em to Python 3, make any API changes
deemed suitable, and call it a day.  And the resulting code could go
to a stdlib metaclass utility module after a reasonable break-in
period.


 (as well as being far more
 approachable as a mechanism than the use of custom metaclasses).

Sure, nobody's arguing that it's not a desirable feature.  I
*implemented* that mechanism for Python 2 (eight years ago) because
it's easier to use even for those of us who are fully versed in the
dark metaclass arts. ;-)   Here's the documentation:

http://peak.telecommunity.com/DevCenter/DecoratorTools#meta-less-classes

So the feature doesn't even require *stdlib* adoption, let alone
changes to Python core.  (Heck, I wasn't even the first to implement
this feature: Zope had it for Python *1.5.2*, in their
ExtensionClass.)

It's a totally solved problem in Python 2, although the solution is
admittedly not widely known.  If the PEP 487 metaclass library,
however, were to just port some bits of my code to Python 3 this could
be a done deal already and available in *all* versions of Python 3,
not just the next one.


 The gap I agree this approach leaves is a final
 post-namespace-execution step that supports establishing any class
 level invariants implied by decorators and other functions used in the
 class body. Python 2 allowed that to be handled with a dynamically
 generated __metaclass__ and PEP 422 through __autodecorate__, while
 PEP 487 currently has no equivalent mechanism.

Right.  And it's *only* having such a mechanism available by *default*
that requires a language change.  Conversely, if we *are* making a
language change, then adding a hook that allows method decorators to
access the just-defined class provides roughly the same generality
that Python 2 had in this respect.

All I want is the ability for method decorators to find out what class
they were added to, at the time the class is built, rather than having
to wait for an access or invocation that may never come.  This could
be as simple as __build_class__ or type.__call__ looking through the
new class's dictionary for objects with a `__used_in_class__(cls,
name)` method, e.g.:

for k, v in dict.items():
if hasattr(v, '__used_in_class__'):
v.__used_in_class__(cls, k)

This doesn't do what PEP 487 or 422 do, but it's the bare minimum for
what I need, and it actually allows this type of decorator to avoid
any frame inspection because they can just add a __used_in_class__
attribute to the functions being decorated.  (It actually also
eliminates another common use

[Python-Dev] PEP 487 vs 422 (dynamic class decoration)

[PJ Eby]

I recently got an inquiry from some of my users about porting some of
my libraries to Python 3 that make use of the Python 2 __metaclass__
facility.  While checking up on the status of PEP 422 today, I found
out about its recently proposed replacement, PEP 487.

While PEP 487 is a generally fine PEP, it actually *rules out* the
specific use case that I wanted PEP 422 for in the first place:
dynamic addition of callbacks or decorators for use at class creation
time without requiring explicit inheritance or metaclass
participation.  (So that e.g. method decorators can access the
enclosing class at class definition time.)

As discussed previously prior to the creation of PEP 422, it is not
possible to port certain features of my libraries to work on Python 3
without some form of that ability, and the only thing that I know of
that could even *potentially* provide that ability outside of PEP 422
is monkeypatching __build_class__ (which might not even work).  That
is, the very thing that PEP 422 was created to avoid the need for.
;-)

One possible alteration would be to replace __init_subclass__ with
some sort of __init_class__ invoked on the class that provides it, not
just subclasses.  That would allow the kind of dynamic decoration that
PEP 422 allows.  However, this approach was rather specifically ruled
out in earlier consideration of PEP 422, so

Another alternative would be to have the default __init_subclass__
look at a class-level __decorators__ attribute, as originally
discussed for PEP 422.  That would solve *my* problem, but feels too
much like adding more than One Way To Do It.

So...  honestly, I'm not sure where to go from here.  Is there any
chance that this is going to be changed, or revert to the PEP 422
approach, or...  something?  If so, what Python version will the
something be in?  Or is this use case just going to be a dead parrot
in Python 3, period?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Multiple inheritance from builtin (C) types [still] supported in Python3?

[PJ Eby]

On Mon, Apr 28, 2014 at 7:26 PM, Paul Sokolovsky pmis...@gmail.com wrote:

 Well, sure I did, as I mentioned, but as that's first time I see that
 code (that specific piece is in typeobject.c:extra_ivars()), it would
 take quite some time be sure I understand all aspects of it. Thanks for
 confirming that it's governed essentially by CPython implementation
 details and not some language-level semantics like metaclasses (I
 mentioned them because error message in Python2 did so, though Python3
 doesn't refer to metaclasses).

 An example would really help me to get a feel of the issue, but I
 assume lack of them means that there's no well-known idiom where such
 inheritance is used, and that's good enough on its own. I also tried to
 figure how it's important to support such multi-base cases, so the code
 I write didn't require complete rewrite if it hits one day, but
 everything seems to turn out to be pretty extensible.


From memory of the last time I dealt with this, the rules were that you
could mix two classes only if their __slots__ differed from their common
__base__ by *at most* __dict__ and/or __weakref__.  The dict and weakref
slots are special, in that the type structure contains their offsets, which
makes them relocatable in subclasses.  But any other __slots__ aren't
relocatable in subclasses, because the type structure doesn't directly keep
track of the offsets.  (The slot descriptors do.)

But I don't think there's anything in principle that requires this, it's
just the implementation.  You could in theory relocate __slots__ defined
from Python code in order to make a merged subclass.  It's just that the
effective __slots__ of C code can't be moved, because C code is expecting
to find them at specific offsets.  Therefore, if two types define their own
struct fields, they can't be inherited from unless one is a subtype of the
other.

In the C code (again if I recall correctly), this is done using the
__base__ attribute of the type, which indicates what struct layout the
object will use.  A type can have a larger struct than its base type,
adding its own fields after the base type's struct fields.  (The dict and
weakref fields are added -- if they are added -- *after* the base struct
fields.  If your __base__ already has them, those offsets within the
existing layout are used, which is why them being in another base class's
__slots__ isn't a problem.)

When you create a new type, CPython looks at your bases to find a suitable
__base__.  If two of your bases inherit from each other, the ancestor can
be ignored, keeping the more-derived one as a candidate __base__.  If a
base adds only __dict__ and/or __weakref__ (or neither) to its __base__,
then its __base__ is a candidate (not recursively, though).  If at the end
there is more than one base left standing, then it's an error, since you
have bases with incompatible layouts.

That is not a precise description of the algorithm, but that's the gist of
how it works.  __base__ is a slot on the type object and is tracked at the
C level in order to sort out layouts like this.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 451 update

[PJ Eby]

On Thu, Oct 31, 2013 at 5:52 AM, Nick Coghlan ncogh...@gmail.com wrote:

 On 31 Oct 2013 18:52, Eric Snow ericsnowcurren...@gmail.com wrote:

 On Wed, Oct 30, 2013 at 10:24 PM, Nick Coghlan ncogh...@gmail.com wrote:
  There's also the option of implementing the constraint directly in the
  finder, which *does* have the necessary info (with the change to pass
  the
  previous spec to find_spec).

 Yeah, I thought of that.  I just prefer the more explicit
 supports_reload().  That said...

 
  I still think it makes more sense to leave this out for the moment -
  it's
  not at all clear we need the extra method, and adding it later would be
  a
  straightforward protocol update.

 ...I agree that makes the most sense for now. :)

 BTW, thanks for pushing these issues.  I think the API has gotten
 pretty solid.  I just need to make sure the PEP covers the cases and
 conclusions we're discussing.

 Thanks are also due to PJE for making me realise we were handwaving too much
 when it came to the expected reload semantics :)

You're welcome.  ;-)  But speaking of handwaving, I also want to be
sure that loader developers know that reloading is only really
reloading if there's a previous existing spec, or the module type
is...

Hm.   Actually, I think I now know how to state what's bugging me
every time I see this supports_reload() or reload=True or other
reloading flags in this process.

I think that references to reloading should be replaced with
references to what's *actually* at issue, because reloading itself
is vague and carries too many assumptions for a loader author to
understand or get right.  (Look how hard it is for *us*!)

That is, I think we should clarify what use cases there are for
knowing whether a reload is happening, and address those use cases
explicitly rather than lumping them under a general heading.

For example, if the reason a loader cares about reloading is because
it's a C extension using a custom module type, and the existing module
isn't of the right type, then we should just spell out how to handle
it.  (e.g. raise an exception)

If the reason a loader cares about reloading is because of some sort
of caching or reuse, then we should just spell out how to handle that,
too.

Lumping these cases together under a reloading flag or a check for
reloading support is a nasty code smell, because it requires a
loader developer to have the *same* vaguely-defined idea of
reloading as the PEP authors.  ;-)

I also suspect, that if properly spelled out, those use cases are
going to boil down to:

1. Throwing errors if you have an existing module object you can't
load into, and
2. Passing in a previous spec object, if available

In other words, loaders should not really have any responsibility for
or concept of reloading -- they always load into a module object
(that they may or may not have created), and they may get given a spec
from a previous load.  They should deal only in module reuse and
spec reuse.  While a typical reload() might involve both reuses,
there are cases where one sort of reuse could occur independently, and
not all loaders care about both (or even either) condition.

At any rate, it means a loader author doesn't have to figure out how
to handle reloading, all they have to figure out is whether they can
load into a particular module object, and whether they can do
something useful with a spec that was previously used to load a module
with the same name -- a spec that may or may not refer to a similar
previous loader.  These are rather more well-defined endeavors than
trying to determine in the abstract whether one supports reload.
;-)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 451 update

[PJ Eby]

On Sun, Oct 27, 2013 at 1:03 AM, Nick Coghlan ncogh...@gmail.com wrote:
 Now, regarding the signature of exec_module(): I'm back to believing
 that loaders should receive a clear indication that a reload is taking
 place. Legacy loaders have to figure that out for themselves (by
 seeing that the module already exists in sys.modules), but we can do
 better for the new API by making the exec_module signature look like:

 def exec_module(self, module, previous_spec=None):
 # module is as per the current PEP 451 text
 # previous_spec would be set *only* in the reload() case
 # loaders that don't care still need to accept it, but can
 just ignore it

Just to be clear, this means that a lazy import implementation that
creates a module object without a __spec__ in the first place will
look like an initial import?  Or will that crash importlib because of
a missing __spec__ attribute?

That is, is reload()'s contract adding a new prerequisite for the
object passed to it?

(The specific use case is creating a ModuleType subclass instance for
lazy importing upon attribute access.  Pre-importlib, all that was
needed was a working __name__ attribute on the module.)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 451 update

[PJ Eby]

On Sun, Oct 27, 2013 at 4:59 PM, Nick Coghlan ncogh...@gmail.com wrote:

 On 28 Oct 2013 02:37, PJ Eby p...@telecommunity.com wrote:

 On Sun, Oct 27, 2013 at 1:03 AM, Nick Coghlan ncogh...@gmail.com wrote:
  Now, regarding the signature of exec_module(): I'm back to believing
  that loaders should receive a clear indication that a reload is taking
  place. Legacy loaders have to figure that out for themselves (by
  seeing that the module already exists in sys.modules), but we can do
  better for the new API by making the exec_module signature look like:
 
  def exec_module(self, module, previous_spec=None):
  # module is as per the current PEP 451 text
  # previous_spec would be set *only* in the reload() case
  # loaders that don't care still need to accept it, but can
  just ignore it

 Just to be clear, this means that a lazy import implementation that
 creates a module object without a __spec__ in the first place will
 look like an initial import?  Or will that crash importlib because of
 a missing __spec__ attribute?

 That is, is reload()'s contract adding a new prerequisite for the
 object passed to it?

 (The specific use case is creating a ModuleType subclass instance for
 lazy importing upon attribute access.  Pre-importlib, all that was
 needed was a working __name__ attribute on the module.)

 For custom loaders, that's part of the contract for create_module() (since
 you'll get an ordinary module otherwise),

Huh?  I don't understand where custom loaders come into it.  For that
matter, I don't understand what get an ordinary module object means
here, either.

I'm talking about userspace code that implements lazy importing
features, like the lazyModule() function in this module:

   http://svn.eby-sarna.com/Importing/peak/util/imports.py?view=markup

Specifically, I'm trying to get an idea of how much that code will
need to change under the PEP (and apparently under importlib in
general).

 and so long as *setting* the
 special module attributes doesn't cause the module to be imported during the
 initial load operation, attribute access based lazy loading will work fine
 (and you don't even have to set __name__, since the import machinery will
 take care of that).

There's no initial load operation, just creation of a dummy module
and stuffing it into sys.modules.  The way it works is that in, say,
foo/__init__.py, one uses:

 bar = lazyModule('foo.bar')
 baz = lazyModule('foo.baz')

Then anybody importing 'foo.bar' or 'foo.baz'  (or using from foo
import bar, etc.) ends up with the lazy module.  That is, it's for
lazily exposing APIs, not something used as an import hook.


 For module level lazy loading that injects a partially initialised module
 object into sys.modules rather than using a custom loader or setting a
 __spec__ attribute, yes, the exec_module invocation on reloading would
 always look like a fresh load operation (aside from the fact that the custom
 instance would already be in sys.modules from the first load operation).

Right.


 It *will* still work, though (at least, it won't break any worse than such 
 code
 does today, since injecting a replacement into sys.modules really isn't
 reload friendly in the first place).

Wait, what?  Who's injecting a replacement into sys.modules?  A
replacement of what?  Or do you mean that loaders aren't supposed to
create new modules, but use the one in sys.modules?

Honestly, I'm finding all this stuff *really* confusing, which is kind
of worrying.  I mean, I gather I'm one of the handful of people who
really understood how importing *used to work*, and I'm having a lot
of trouble wrapping my brain around the new world.

(Granted, I think that may be because I understand how a lot of old
corner cases work, but what's bugging me is that I no longer
understand how those old corners work under the new regime, nor do I
feel I understand what the *new* corners will be.  This may also just
be communication problems, and the fact that it's been months since I
really walked through importlib line by line, and have never really
walked through it (or PEP 451) quite as thoroughly as I have import.c.
 I also seem to be having trouble grokking why the motivating use
cases for PEP 451 can't be solved by just providing people with good
base classes to use for writing loaders -- i.e., I don't get why the
core protocol has to change to address the use case of writing loaders
more easily.  The new protocol seems way more complex than PEP 302,
and ISTM the complexity could just be pushed off to the loader side of
the protocol without creating more interdependency between importlib
and the loaders.)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 451 update

[PJ Eby]

I've not really had time to review this PEP yet, but from skimming
discussion to date, the only thing I'm still worried about is whether
this will break lazy import schemes that use a module subclass that
hooks __getattribute__ and calls reload() in order to perform what's
actually an *initial* load.

IOW, does anything in this proposal rely on a module object having
*any* attributes besides __name__ set at reload() time?  That is, is
there an assumption that a module being reloaded has

1. Been loaded, and
2. Is being reloaded via the same location, __loader__, etc. as before?

At least through all 2.x, reload() just uses module.__name__ to
restart the module find-and-load process, and does not assume that
__loader__ is valid in advance.

(Also, if this has changed in recent Python versions independent of
this PEP, it's a backwards-compatibility break that should be
documented somewhere.)


On Thu, Oct 24, 2013 at 2:05 AM, Eric Snow ericsnowcurren...@gmail.com wrote:
 I've had some offline discussion with Brett and Nick about PEP 451
 which has led to some meaningful clarifications in the PEP.  In the
 interest of pulling further discussions back onto this
 (archived/public) list, here's an update of what we'd discussed and
 where things are at. :)

 * path entry finders indicate that they found part of a possible
 namespace package by returning a spec with no loader set (but with
 submodule_search_locations set).  Brett wanted some clarification on
 this.
 * The name/path signature and attributes of file-based finders in
 importlib will no longer be changing.  Brett had some suggestions on
 the proposed change and it became clear that the the change was
 actually pointless.
 * I've asserted that there shouldn't be much difficulty in adjusting
 pkgutil and other modules to work with ModuleSpec.
 * Brett asked for clarification on whether the load() example from
 the PEP would be realized implicitly by the import machinery or
 explicitly as a method on ModuleSpec.  This has bearing on the ability
 of finders to return instances of ModuleSpec subclasses or even
 ModuleSpec-like objects (a la duck typing).  The answer is the it will
 not be a method on ModuleSpec, so it is effectively just part of the
 general import system implementation.  Finders may return any object
 that provides the attributes of ModuleSpec.  I will be updating the
 PEP to make these points clear.

 * Nick suggested writing a draft patch for the language reference
 changes (the import page).  Such a patch will be a pretty good
 indicator of the impact of PEP 451 on the import system and should
 highlight any design flaws in the API.  This is on my to-do list
 (hopefully by tomorrow).
 * Nick also suggested moving all ModuleSpec methods to a separate
 class that will simply make use of a separate, existing ModuleSpec
 instance.  This will help address several issues, particularly by
 relaxing the constraints on what finders can return, but also by
 avoiding the unnecessary exposure of the methods via every
 module.__spec__.  I plan on going with this, but currently am trying
 out the change to see if there are any problems I've missed.  Once I
 feel good about it I'll update the PEP.

 That about sums up our discussions.  I have a couple of outstanding
 updates to the PEP to make when I get a chance, as well as putting up
 a language reference patch for review.

 -eric
 ___
 Python-Dev mailing list
 Python-Dev@python.org
 https://mail.python.org/mailman/listinfo/python-dev
 Unsubscribe: 
 https://mail.python.org/mailman/options/python-dev/pje%40telecommunity.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 451 update

[PJ Eby]

On Fri, Oct 25, 2013 at 1:15 PM, Brett Cannon br...@python.org wrote:

 On Fri, Oct 25, 2013 at 12:24 PM, PJ Eby p...@telecommunity.com wrote:
 At least through all 2.x, reload() just uses module.__name__ to
 restart the module find-and-load process, and does not assume that
 __loader__ is valid in advance.


 That doesn't make much sense in a post-importlib world where import makes
 sure that __loader__ is set (which it has since Python 3.3). Otherwise you
 are asking for not just a reload but a re-find as well.

That's a feature, not a bug.  A reload() after changing sys.path
*should* take into account the change, not to mention any changes to
meta_path, path hooks, etc.  (And it's how reload() worked before
importlib.)

I suppose it's not really documented all that well, but way way back
in the 2.3 timeframe I asked for a tweak to PEP 302 to make sure that
reload() (in the re-find sense) would work properly with PEP 302
loaders like zipimport -- some of the language still in the PEP is
there specifically to support this use case.  (Specifically, the bit
that says loaders *must* use the existing module object in sys.modules
if there's one there already, so that reload() will work.  It was
actually in part to ensure that reload() would work in the case of a
re-find.)

It appears that since then, the PEP has been changed in a way that
invalidates part of the purpose of the prior change; I guess I missed
the discussion of that change last year.  :-(

ISTM there should've been such a discussion, since IIRC importlib
wasn't supposed to change any Python semantics, and this is a
non-trivial change to the semantics of reload() even in cases that
aren't doing lazy imports or other such munging.  reload() used to
take sys.* and __path__ changes into account, and IMO should continue
to do so.  If this is an intentional change in reload() semantics,
other Python implementations need to know about this too!

(That being said, I'm not saying I shouldn't or couldn't have tested
this in 3.3 and found out about it that way.  And the existence of
issue18698 suggests that nobody's relying yet on even the
*fundamental* semantics of PEP 302 reload() working properly in 3.3,
since that was an even bigger change that nobody spotted till a couple
of months ago.)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 451 update

[PJ Eby]

On Fri, Oct 25, 2013 at 3:15 PM, Brett Cannon br...@python.org wrote:
 On Fri, Oct 25, 2013 at 2:10 PM, PJ Eby p...@telecommunity.com wrote:
 On Fri, Oct 25, 2013 at 1:15 PM, Brett Cannon br...@python.org wrote:
  On Fri, Oct 25, 2013 at 12:24 PM, PJ Eby p...@telecommunity.com wrote:
  At least through all 2.x, reload() just uses module.__name__ to
  restart the module find-and-load process, and does not assume that
  __loader__ is valid in advance.
 
 
  That doesn't make much sense in a post-importlib world where import
  makes
  sure that __loader__ is set (which it has since Python 3.3). Otherwise
  you
  are asking for not just a reload but a re-find as well.

 That's a feature, not a bug.  A reload() after changing sys.path
 *should* take into account the change, not to mention any changes to
 meta_path, path hooks, etc.  (And it's how reload() worked before
 importlib.)


 Fair enough, but in my mind that obviously doesn't really click for what I
 view as a reload in an importlib world where secret import code no longer
 exists. When I think re-load I think load again, not find the module
 again and execute a load with a possibly new loader.

Sure, and the reference manual is rather vague on this point.
However, I would guess that at least some web frameworks with
automatic reload support are going to barf on this change in at least
some edge cases.

(OTOH, it's unlikely the bugs will ever be reported, because the
problem will mysteriously go away once the process is restarted,
probably never to occur again.)

Mostly, this just seems like an ugly wart -- Python should be dynamic
by default, and that includes reloading.  While the import machinery
has lots of ugly caching under the hood, a user-level function like
reload() should not require you to do the equivalent of saying, no,
really...  I want you to *really* reload, not just pull in whatever
exists where you found it last time, while ignoring whether I switched
from module to package or vice versa, or just fixed my sys.path so I
can load the right version of the module.

It is a really tiny thing in the overall scheme of things, because
reload() is not used all that often, but it's still a thing.  If this
isn't treated as a bug, then the docs for reload() at least need to
include a forward-supported workaround so you can say no, really...
*really* reload in an approved fashion.

(ISTM that any production code out there that currently uses reload()
would want to perform the really reload incantation in order to
avoid the edge cases, even if they haven't actually run into any of
them yet.)


 And in a PEP 451 world it should be dead-simple to make this work the way
 you want in your own code even if this doesn't go the way you want::

   spec = importlib.find_spec(name)
   module.__spec__ = spec
   importlib.reload(module)  # Which in itself is essentially
 init_module_attrs(spec, module); spec.loader.exec_module(module)

 Heck, you can do this in Python 3.3 right now::

   loader = importlib.find_loader(name)
   module = sys.modules[name]
   module.__loader__ = loader
   importlib.reload(module)

And will that later version still work correctly in a PEP 451 world,
or will you have to detect which world you live in before waving this
particular dead chicken?  ;-)


 Ah, okay. That is not explicit in the PEP beyond coming off a total nuisance
 in order to support reloading by the loader, not an explicit finder + loader
 use-case.

Yeah, it actually was to ensure that you could reload a module using a
different loader than the one that originally loaded it, e.g. due to a
change in path hooks, etc.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore().

[PJ Eby]

On Tue, Oct 15, 2013 at 8:57 AM, Nick Coghlan ncogh...@gmail.com wrote:
 So, having been convinced that ignore was the wrong choice of name,
 reviewing the docs made it clear to me what the name *should* be.

From the point of view of code *outside* a block, the error is indeed
suppressed.

But, as one of those examples actually points out, what's happening
from the POV *inside* the block is that the exception is trapped.

So using suppress creates an ambiguity: are we suppressing these
errors *inside* the block, or *outside* the block?  The way it
actually works is errors are suppressed from the code *surrounding*
the block, but the word can equally be interpreted as suppressing
errors *inside* the block, in exactly the same way that ignore can
be misread.

So, if we're going with words that have precedent in the doc, the term
trap, as used here:

 If an exception is trapped merely in order to log it or to perform
 some action (rather than to suppress it entirely), the generator must
 reraise that exception.

is the only one used to describe the POV from inside the block, where
the error is...  well, being trapped.  ;-)

It is a more apt description of what actually happens, even if it's
only usable for the specific use case where an exception is trapped in
order to suppress it.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore().

[PJ Eby]

On Sun, Oct 13, 2013 at 10:05 AM, Antoine Pitrou solip...@pitrou.net wrote:
 And for the record, it's not *my* objection; several other core
 developers have said -1 too: Ezio, Serhiy, Giampaolo, etc.

FWIW, I'm -1 also; the thread quickly convinced me that this is a
horrible idea, at least with the current name.

The feature itself I consider +0, maybe +0.5 if a good but short name
can be found.  I kind of like abort_on() as an accurate description
of what it actually does, but it most certainly does not *ignore*
exceptions, and it's going to create problems as soon as anybody adds
more than one statement to the block, and then reads their code back
without *really* thinking about it.  Not to mention how it's going to
bite people who copy and modify code snippets containing it.


On Sun, Oct 13, 2013 at 11:11 AM, Nick Coghlan ncogh...@gmail.com wrote:
 It's just as broken as the try/except equivalent. I consider that a
 feature, not a bug.

(Note: the following rant is about the *name*, not the context manager itself.)

Misleadingness and lack of readability is not a feature, it's a bug.

For example, even though I've been coding in Python since 1997, and
even participated in the original design process for with, I *still*
misread the with ignore: block as ignoring the exceptions until I
*really* thought about it.

Wait, no, I misread it *entirely*, until somebody *else* pointed it out.  ;-)

And this is *despite* knowing on a gut level that *actually* ignoring
all the errors in a block *isn't possible in Python*.

I would not give most people much chance of noticing they made this
mistake, and even less chance of finding the problem afterwards.

This is like the infamous Stroop test, where you have a word like
brown only it's printed in blue ink and you have to say blue to
get the answer right.

If you've never taken a Stroop test, by the way, it's *really* hard.
It almost literally makes your brain *hurt* to disregard the text and
say the ink color instead, because your brain automatically reads the
word before you can stop it, so you are straining to stop yourself
from saying it so you can then try to *think* what color you're
supposed to say, and then your brain reads the word *again*, and...
well, it's really quite unpleasant is what it is.

Anyway, this feature, with its current name, is just the same: you
have to override your instinctive response to understand what it
*really* does, in any but the one-liner case.

And you have to do it *every time you read it in code*.

Only, because it'll mostly be used in the one-line case, you'll get
used to it being correct, until one day you make a change without
thinking, and create a bug that lies dormant for an extended period.

Plus, as soon as people see it being used, they'll think, oh cool,
and use it in their code, not even knowing or thinking that it does
something they don't want, because they will never read the docs in
the first place.  (As Guido says, people learn languages by example.)

So call it catching.  Call it catch_and_exit_on.  Even passing
or skipping would be better.  And maybe abort_on or
abort_without_raising would be better still, as they describe what
will *really* happen.

But calling it ignore isn't fits your brain, it's abuses your
brain in a cruelly misleading manner.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore().

[PJ Eby]

On Sun, Oct 13, 2013 at 1:58 PM, Alexander Belopolsky
alexander.belopol...@gmail.com wrote:
 People who write code using contextlib
 are expected to know

People who *read* that code while learning Python cannot be expected
to know that it is not really possible to ignore errors in Python.

If this feature is used under any name that implies such, it will
within a few years become a FAQ and well-known wart, not to mention a
meme that contextlib.ignore() is buggy, it only works if the error is
thrown from a single operation performed in C.

I say this latter phrasing because now that I've had time to think
about it, it is not at all merely a question of whether you wrap a
single line or single operation.

Quick, is this a safe use, or not:

with ignore(OSError):
 delete_widget_files(spam)

It sort of depends on the implementation of delete_widget_files, doesn't it?

In contrast:

   with abort_on(OSError):
 delete_widget_files(spam)

it's immediately clear that the error isn't going to be ignored; the
operation will be aborted.  Very different concept.


 that it is not a good idea to keep resources
 multiple unrelated statements within the
 with block will raise a mental red flag.

How will someone know this when they are reading code they found on
the internet?

It's one thing to have an operation whose name implies, you need to
do more research to understand this.  But it's an entirely different
(and dangerous) thing to have an operation whose name implies you
already know everything you need to know, no need to think or study
further...  especially if what you know is actually wrong!


 It is also easy for
 lint-like tools to warn about abuse of ignore().

Since it's not sufficient to require a single operation, how will a
lint-like tool check this?

For example:

with ignore(AnError, OtherError):
 ping.pongy(foo, bar.spam(), fizzy())

Is this valid code, or not?  If you can't tell, how will a non-human
lint tool tell?


 Let's not try to improve readability of bad code

Actually, I want the good code to be readable in the sense of
understanding what the operation does, so that people copying it don't
end up with a serious misunderstanding of how the context manager
actually works.

There is no way that naive users aren't going to read it as ignoring
errors, and use it with something like:

with ignore(OSError):
for f in myfiles: os.unlink(f)

But this version is obviously *wrong*:

with abort_on(OSError):
for f in myfiles: os.unlink(f)

Upon looking at this code, you will quickly realize that you don't
intend to abort the loop, only the unlink, and will therefore rewrite
it to put the loop on the outside.

So, I am only trying to improve readability of bad code in the sense
of making it *obvious* that the code is in fact bad.  ;-)

(To put it another way, ignore() improves the readability of bad
code in the above example, because it makes the bad code look like
it's good.)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] sys.intern should work on bytes

[PJ Eby]

On Fri, Sep 20, 2013 at 9:54 AM, Jesus Cea j...@jcea.es wrote:
 Why str/bytes doesn't support weakrefs, beside memory use?

The typical use case for weakrefs is to break reference cycles, but
str and bytes can't *be* part of a reference cycle, so outside of
interning-like use cases, there's no need for weakref support there.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

[PJ Eby]

On Fri, Aug 23, 2013 at 4:50 AM, Stefan Behnel stefan...@behnel.de wrote:
 Reloading and Sub-Interpreters
 ==

 To reload an extension module, the module create function is executed
 again and returns a new module type. This type is then instantiated as by
 the original module loader and replaces the previous entry in sys.modules.
 Once the last references to the previous module and its type are gone, both
 will be subject to normal garbage collection.

I haven't had a chance to address this on the import-sig discussion
yet about ModuleSpec, but I would like to just mention that one
property of the existing module system that I'm not sure either this
proposal or the ModuleSpec proposal preserves is that it's possible to
implement lazy importing of modules using standard reload() semantics.

My Importing package offers lazy imports by creating module objects
in sys.modules that are a subtype of ModuleType, and use a
__getattribute__ hook so that trying to use them fires off a reload()
of the module.  Because the dummy module doesn't have __file__ or
anything else initialized, the import system searches for the module
and then loads it, reusing the existing module object, even though
it's actually only executing the module code for the first time.

That the existing object be reused is important, because once the
dummy is in sys.modules, it can also be imported by other modules, so
references to it can abound everywhere, and we wish only for it to be
loaded lazily, without needing to trace down and replace all instances
of it.  This also preserves other invariants of the module system.

Anyway, the reason I was asking why reloading is being handled as a
special case in the ModuleSpec proposal -- and the reason I'm curious
about certain provisions of this proposal -- is that making the
assumption you can only reload something with the same
spec/location/etc. it was originally loaded with, and/or that if you
are reloading a module then you previously had a chance to do things
to it, doesn't jibe with the way things work currently.

That is to say, in the pure PEP 302 world, there is no special status
for reload that is different from load -- the *only* thing that's
different is that there is already a module object to use, and there
is *no guarantee that it's a module object that was initialized by the
loader now being invoked*.

AFAICT both this proposal and the ModuleSpec one are making an invalid
assumption per PEP 302, and aren't explicitly proposing to change the
status quo: they just assume things that aren't actually assured by
the prior specs or implementations.

So, for example, this extension module proposal needs to cover what
happens if an extension module is reloaded and the module object is
not of the type or instance it's expecting.  Must it do its own
checking?  Error handling?  Will some other portion of the import
system be expected to handle it?

For that matter, what happens (in either proposal) if you reload() a
module which only has a __name__, and no other attributes?  I haven't
tested with importlib, but with earlier Pythons this results in a
standard module search being done by reload().  But the ModuleSpec
proposal and this one seem to assume that a reload()-ed module must
already be associated with a loader, location, and/or spec.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Classes with ordered namespaces

[PJ Eby]

On Thu, Jun 27, 2013 at 4:48 AM, Nick Coghlan ncogh...@gmail.com wrote:
 I'd be tempted to kill PEP 422 as not worth the hassle if we did this.

I assume you mean the namespace keyword part of PEP 422, since PEP
422's class initialization feature is entirely orthogonal to
definition order or namespace customization.  (Indeed, I cannot recall
a single instance of class initialization in my code that actually
*cared* about definition order.  Certainly I haven't had any
situations where a pre-existing definition order would've eliminated
the need for a class-level initialization.)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] eval and triple quoted strings

[PJ Eby]

On Fri, Jun 14, 2013 at 2:11 PM, Ron Adam ron3...@gmail.com wrote:


 On 06/14/2013 10:36 AM, Guido van Rossum wrote:

 Not a bug. The same is done for file input -- CRLF is changed to LF before
 tokenizing.



 Should this be the same?


 python3 -c 'print(bytes(\r\n, utf8))'
 b'\r\n'


 eval('print(bytes(\r\n, utf8))')
 b'\n'

No, but:

eval(r'print(bytes(\r\n, utf8))')

should be.  (And is.)

What I believe you and Walter are missing is that the \r\n in the eval
strings are converted early if you don't make the enclosing string
raw.  So what you're eval-ing is not what you think you are eval-ing,
hence the confusion.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] doctest and pickle

[PJ Eby]

On Fri, Jun 7, 2013 at 1:54 PM, Mark Janssen dreamingforw...@gmail.com wrote:
 On Fri, Jun 7, 2013 at 10:50 AM, Mark Janssen dreamingforw...@gmail.com 
 wrote:
  from pickle import dumps, loads
  Fruit.tomato is loads(dumps(Fruit.tomato))
 True

 Why are you using is here instead of ==?  You're making a circular
 loop using is

 I should add that when you're serializing with pickle and then
 reloading, the objects should be seen as essentially equivalent.
 This means that they are either byte-by-byte equivalent (not sure
 actually if Python actually guarantees this), or every element would
 still compare equal and that is what matters.

For global objects such as functions and classes -- and singletons
such as None, Ellipsis, True, and False -- pickling and unpickling is
actually supposed to retain the is relationship as well.

I don't know if enums *actually* preserve this invariant, but my
default expectation of the One Obvious Way would be that enums, being
uniquely-named objects that know their name and container, should be
considered global objects in the same fashion as classes and
functions, *and* that as singletons, they'd also be treated in the
same way as None, Ellipsis, etc.  That is, there are two independent
precedents for objects like that preserving is upon pickling and
unpickling.

(As another precedent, my own SymbolType library (available on PyPI)
preserves the is-ness of its named symbol objects upon pickling and
unpickling.)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] cpython: Add reference implementation for PEP 443

[PJ Eby]

On Fri, Jun 7, 2013 at 10:27 AM, Thomas Wouters tho...@python.org wrote:
 This isn't a new bug, but it's exposed by always importing weakref and
 atexit during interpreter startup. I'm wondering if that's really necessary
 :)

Importing it during startup isn't necessary per se; imports needed
only by the generic function implementation can and should be imported
late, rather than at the time functools is imported.

However, if pkgutil was/is migrated to using this implementation of
generics, then it's likely going to end up imported during startup
anyway, because at least the -m startup path involves pkgutil.

In short, the overall answer right now is, maybe, and the answer
later is rather likely.  ;-)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] cpython: Add reference implementation for PEP 443

[PJ Eby]

On Fri, Jun 7, 2013 at 5:16 PM, Łukasz Langa luk...@langa.pl wrote:
 On 7 cze 2013, at 22:50, PJ Eby p...@telecommunity.com wrote:

 On Fri, Jun 7, 2013 at 10:27 AM, Thomas Wouters tho...@python.org wrote:
 This isn't a new bug, but it's exposed by always importing weakref and
 atexit during interpreter startup. I'm wondering if that's really necessary
 :)

 In short, the overall answer right now is, maybe, and the answer
 later is rather likely.  ;-)

 I would rather say that it's rather certain.

 functools is necessary for setup.py to work while bootstrapping, whereas
 pkgutil is used in runpy.py which is always imported in Modules/main.c.

 So we're left with having to fix atexit to support subinterpreters. I wonder
 how difficult that will be.

If the problem really has to do with interpreter startup, then there
actually is a workaround possible, at the cost of slightly hairier
code.

If dispatch() looked in the registry *first* and avoided the cache in
that case, and lazily created the cache (including the weakref
import), then interpreter startup would not trigger an import of
weakref in the default case.

(Of course, depending on whether site/sitecustomize results in the use
of importer subclasses and such, this might not help.  It might be
necessary to take the even more complex tack of avoiding the use of a
cache entirely until an ABC is registered, and walking mro's.)

Anyway, it remains to be seen whether these workarounds are easier or
more difficult than fixing the atexit problem.  ;-)

Hmm...  actually, there are a couple other ways around this.
singledispatch doesn't use finalize(), so it doesn't really need
atexit.  It doesn't even do much with WeakKeyDictionary, so it could
actually just from _weakref import ref, and inline the relevant
operations.

Or, WeakKeyDictionary could be pulled out into a separate module,
where singledispatch could pull it from without importing finalize.

Or, weakref.finalize could be fixed so that the atexit import and
register() are deferred until actual use.

(Of all of these, that last one actually sounds like the least
invasive workaround, with fewest lines of code likely to be changed.)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 443 - Single-dispatch generic functions (including ABC support)

[PJ Eby]

On Tue, May 28, 2013 at 3:41 PM, Russell E. Owen ro...@uw.edu wrote:
 Is it true that this cannot be used for instance and class methods? It
 dispatches based on the first argument, which is self for instance
 methods, whereas the second argument would almost certainly be the
 argument one would want to use for conditional dispatch.

You can use a staticmethod and then delegate to it, of course.  But it
probably wouldn't be too difficult to allow specifying which argument
to dispatch on, e.g.:

@singledispatch.on('someArg')
def my_method(self, someArg, ...):
  ...

The code would look something like this:

def singledispatch(func, argPosn=0):
...
# existing code here...
...
def wrapper(*args, **kw):
return dispatch(args[argPosn].__class__)(*args, **kw)  #
instead of args[0]

def _dispatch_on(argname):
def decorate(func):
argPosn = # code to find argument position of argname for func
   return dispatch(func, argPosn)
return decorate

singledispatch.on = _dispatch_on

So, it's just a few lines added, but of course additional doc, tests,
etc. would have to be added as well.  (It also might be a good idea
for there to be some error checking in wrapper() to raise an
approriate TypeError if len(args)=arg.)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 443 - Single-dispatch generic functions (including ABC support)

[PJ Eby]

On Sat, May 25, 2013 at 8:08 AM, Łukasz Langa luk...@langa.pl wrote:
 The most important
 change in this version is that I introduced ABC support and completed
 a reference implementation.

Excellent!  A couple of thoughts on the implementation...

While the dispatch() method allows you to look up what implementation
would be *selected* for a target type, it does not let you figure out
whether a particular method has been *registered* for a type.

That is, if I have a class MyInt that subclasses int, I can't use
dispatch() to check whether a MyInt implementation has been
registered, because I might get back an implementation registered for
int or object.  ISTM there should be some way to get at the raw
registration info, perhaps by exposing a dictproxy for the registry.

Second, it should be possible to memoize dispatch() using a weak key
dictionary that is cleared if new ABC implementations have been
registered or when a call to register() is made.  The way to detect
ABC registrations is via the ABCMeta._abc_invalidation_counter
attribute: if its value is different than the previous value saved
with the cache, the cache must be cleared, and the new value stored.

(Unfortunately, this is a private attribute at the moment; it might be
a good idea to make it public, however, because it's needed for any
sort of type dispatching mechanism, not just this one particular
generic function implementation.)

Anyway, doing the memoizing in the wrapper function should bring the
overall performance very close to a hand-written type dispatch.  Code
might look something like:

# imported inside closure so that functools module
# doesn't force import of these other modules:
#
from weakref import ref, WeakKeyDictionary
from abc import ABCMeta

cache = WeakKeyDictionary()
valid_as_of = ABCMeta._abc_invalidation_counter

def wrapper(*args, **kw):
nonlocal valid_as_of
if valid_as_of != ABCMeta._abc_invalidation_counter:
cache.clear()
valid_as_of = ABCMeta._abc_invalidation_counter
cls = args[0].__class__
try:
impl = cache.data[ref(cls)]
except KeyError:
impl = cache[cls] = dispatch(cls)
return impl(*args, **kw)

def register(typ, func=None):
...
cache.clear()
...

This would basically eliminate doing any extra (Python) function calls
in the common case, and might actually be faster than my current
simplegeneric implementation on PyPI (which doesn't even do ABCs at
the moment).
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 443 - Single-dispatch generic functions (including ABC support)

[PJ Eby]

On Sat, May 25, 2013 at 10:59 AM, Nick Coghlan ncogh...@gmail.com wrote:
 Given the global nature of the cache invalidation, it may be better as
 a module level abc.get_cache_token() function.

Well, since the only reason to ever use it is to improve performance,
it'd be better to expose it as an attribute than as a function.  ;-)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] __subclasses__() return order

[PJ Eby]

On Sat, May 25, 2013 at 9:18 AM, Antoine Pitrou solip...@pitrou.net wrote:
 In http://bugs.python.org/issue17936, I proposed making tp_subclasses
 (the internal container implementing object.__subclasses__) a dict.
 This would make the return order of __subclasses__ completely
 undefined, while it is right now slightly predictable. I have never seen
 __subclasses__ actually used in production code, so I'm wondering
 whether someone might be affected by such a change.

FWIW, when I've used __subclasses__, I've never depended on it having
a stable or predictable order.  (I find it somewhat difficult to
imagine *why* one would do that, but of course that doesn't mean
nobody has done it.)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 443 - Single-dispatch generic functions (including ABC support)

[PJ Eby]

On Sat, May 25, 2013 at 4:16 PM, Łukasz Langa luk...@langa.pl wrote:
 So, the latest document is live:
 http://www.python.org/dev/peps/pep-0443/

 The code is here:
 http://hg.python.org/features/pep-443/file/tip/Lib/functools.py#l363

 The documentation here:
 http://hg.python.org/features/pep-443/file/tip/Doc/library/functools.rst#l189

Code and tests look great!

Nitpick on the docs and PEP, though: generic functions are not
composed of functions sharing the same name; it would probably be more
correct to say they're composed of functions that perform the same
operations on different types.  (I think the names language might be
left over from discussion of *overloaded* functions in PEP 3124 et al;
in any case we're actually recommending people *not* use the same
names now, so it's confusing.)

We should probably also standardize on the term used for the
registered functions.  The standard terminology is method, but that
would be confusing in Python, where methods usually have a self
argument.  The PEP uses the term implementation, and I think that
actually makes a lot of sense: a generic function is composed of
functions that implement the same operation for different types.

So I suggest changing this:


Transforms a function into a single-dispatch generic function. A **generic
function** is composed of multiple functions sharing the same name. Which
form should be used during a call is determined by the dispatch algorithm.
When the implementation is chosen based on the type of a single argument,
this is known as **single dispatch**.

Adding an overload to a generic function is achieved by using the
:func:`register` attribute of the generic function. The
:func:`register` attribute is a decorator, taking a type paramater
and decorating a function implementing the overload for that type.

to:


Transforms a function into a single-dispatch generic function. A **generic
function** is composed of multiple functions implementing the same
operation for different types. Which
implementation should be used during a call is determined by the
dispatch algorithm.
When the implementation is chosen based on the type of a single argument,
this is known as **single dispatch**.

Adding an implementation to a generic function is achieved by using the
:func:`register` attribute of the generic function. The
:func:`register` attribute is a decorator, taking a type paramater
and decorating a function implementing the operation for that type.

And replacing overload with implementation in the remainder of the
docs and code.

Last, but not least, there should be a stacking example somewhere in
the doc, as in the PEP, and perhaps the suggestion to name individual
implementations differently from each other and the main function --
perhaps as an adjunct to documenting that register() always returns
its argument unchanged.  (Currently, it doesn't mention what
register()'s return value is.)

(It may also be useful to note somewhere that, due to caching,
changing the base classes of an existing class may not change what
implementation is selected the next time the generic function is
invoked with an argument of that type or a subclass thereof.)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 443 - Single-dispatch generic functions

[PJ Eby]

On Thu, May 23, 2013 at 11:57 PM, Nick Coghlan ncogh...@gmail.com wrote:
 We should be able to use it to help deal with the every growing
 importer API problem, too. I know that's technically what pkgutil
 already uses it for, but elevating this from pkgutil implementation
 detail to official stdlib functionality should make it easier to
 document properly :)


Oh, that reminds me.  pprint() is actually an instance of a general
pattern that single dispatch GF's are good for: visitor pattern
algorithms.  There's a pretty good write-up on the general issues with
doing visitor pattern stuff in Python, and how single-dispatch GF's
can solve that problem, here:

http://peak.telecommunity.com/DevCenter/VisitorRevisited

The code samples use a somewhat different API from the PEP, but it's
pretty close.  The main issues solved are eliminating monkeypatching
and fixing the inheritance problems that occur when you use
'visit_foo' methods.  One of the samples actually comes from the old
'compiler' package in the stdlib...  which tells you how long ago I
did the write-up.  ;-)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 443 - Single-dispatch generic functions

[PJ Eby]

On Thu, May 23, 2013 at 11:11 AM, Paul Moore p.f.mo...@gmail.com wrote:
 Is the debate between 1 and 2, or 1 and 3? Is it even possible to implement
 3 without having 2 different names for register?

Yes.  You could do it as either:
@func.register
def doit(foo: int):
...

by checking for the first argument to register() being a function, or:

   @func.register()
def doit(foo: int):
...

by using a default None first argument.  In either case, you would
then raise a TypeError if there wasn't an annotation.

As to the ability to do multiple types registration, you could support
it only in type annotations, e.g.:

@func.register
def doit(foo: [int, float]):
...

without it being confused with being multiple dispatch.

One other thing about the register API that's currently unspecified in
the PEP: what does it return, exactly?

I generally lean towards returning the undecorated function, so that if you say:

@func.register
def do_int(foo: int):
...

You still have the option of calling it explicitly.  OTOH, some may
prefer to treat it like an overload and call it 'func' every time, in
which case register should return the generic function.  Some guidance
as to what should be the official One Obvious Way would be helpful
here.

(Personally, I usually name my methods explicitly because in debugging
it's a fast clue as to which piece of code I should be looking at.)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 443 - Single-dispatch generic functions

[PJ Eby]

On Thu, May 23, 2013 at 2:59 PM, PJ Eby p...@telecommunity.com wrote:
 I generally lean towards returning the undecorated function, so that if you 
 say:

 @func.register
 def do_int(foo: int):
 ...

Oops, forgot to mention:  one other advantage to returning the
undecorated function is that you can do this:

@func.register(int)
@func.register(float)
def do_num(foo):
   ...

Which neatly solves the multiple registration problem, even without
argument annotations.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 443 - Single-dispatch generic functions

[PJ Eby]

On Thu, May 23, 2013 at 6:58 PM, Ben Hoyt benh...@gmail.com wrote:
 It seems no one has provided
 decent use-case examples (apart from contrived ones)

Um, copy.copy(), pprint.pprint(), a bunch of functions in pkgutil
which are actually *based on this implementation already* and have
been since Python 2.5...  I don't see how any of those are contrived
examples.  If we'd had this in already, all the registration-based
functions for copying, pickling, etc. would likely have been
implemented this way, and the motivating example for the PEP is the
coming refactoring of pprint.pprint.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Fighting the theoretical randomness of is on immutables

[PJ Eby]

On Mon, May 6, 2013 at 4:46 AM, Armin Rigo ar...@tunes.org wrote:
 This is clearly a language design issue though.  I can't really think
 of a use case that would break if we relax the requirement, but I
 might be wrong.  It seems to me that at most some modules like pickle
 which use id()-keyed dictionaries will fail to find some
 otherwise-identical objects, but would still work (even if tuples are
 relaxed in this way, you can't have cycles with only tuples).

I don't know if I've precisely understood the change you're proposing,
but I do know that in PEAK-Rules I use id() as an approximation for
is in order to build indexes of various parameter is some_object
conditions, for various some_objects and a given parameter.  The
rule engine takes id(parameter) at call time and then looks it up to
obtain a subset of applicable rules.  IIUC, this would require that
either x is y equates to id(x)==id(y), or else that there be some
way to determine in advance all the possible id(y)s that are now or
would ever be is x, so they can be placed in the index.  Otherwise,
any use of an is condition would require a linear search of the
possibilities, as you could not rule out the possibility that a given
x was is to *some* y already in the index.

Of course, rules using is tend to be few and far between, outside of
some special cases, and their use with simple integers and strings
would be downright silly.  And on top of that, I'm not even sure
whether the a = b notation you used was meant to signify a implies
b or b implies a.  ;-)

But since you mentioned id()-keyed dictionaries and this is another
use of them that I know of, I figured I should at least throw it out
there for information's sake, regardless of which side of the issue it
lands on.  ;-)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] enum discussion: can someone please summarize open issues?

[PJ Eby]

On Sun, Apr 28, 2013 at 7:37 PM, Steven D'Aprano st...@pearwood.info wrote:
 I have also suggested that that the enum package provide a decorator
 which can be used to explicitly flag values to *not* be turned into
 enum values. See here:

 http://mail.python.org/pipermail/python-dev/2013-April/125641.html

In that example, 'food = property(lambda:skip)' would work in a
pinch. (Granted, it wouldn't be a *class* attribute, but you can make
a class attribute by assiging it after class creation is completed.)

And if you want to make your enum instances callable, ISTM the right
(or at least the One Obvious) way to do it is to add a __call__ method
to the class.


 Even if the Enum class doesn't support this feature, I ask that it be
 written in such a way that a subclass could add it (i.e. please expose
 a public method for deciding what to exclude).

Since you can exclude anything by it having a __get__ method, or
include it by making it *not* have a __get__ method, I'm not sure what
use case you're actually looking for.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] class name spaces inside an outer function

[PJ Eby]

On Sat, Apr 27, 2013 at 2:27 PM, Ethan Furman et...@stoneleaf.us wrote:
 I filed bug http://bugs.python.org/issue17853 last night.

 If somebody could point me in the right direction (mainly which files to
 look in), I'd be happy to attempt a patch.

Wow.  I had no idea Python actually did this (override class-local
references with ; I'd have expected your code to work.  I was even
more surprised to find that the same thing happens all the way back to
Python 2.3.  Guess I'm not nearly the wizard of abusing scope rules
that I thought I was.  ;-)

About the only workaround I can see is to put Season = Season at the
top of a class that uses this inside a function definition, or else to
define a special name 'enum' instead and hope that nobody ever tries
to define an enumeration inside a function with a local variable named
'enum'.  ;-)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] relative import circular problem

[PJ Eby]

On Thu, Apr 4, 2013 at 11:17 AM, Guido van Rossum gu...@python.org wrote:
 I don't really see what we could change to avoid breaking code in any
 particular case

Actually, the problem has *nothing* to do with circularity per se;
it's that import a.b and from a import b behave differently in
terms of how they obtain the module 'a.b'...

And from a import b will *always* fail if 'a.b' is part of a cycle
with the current module, whereas import a.b will *always* succeed.

The workaround is to tell people to always use import a.b in the
case of circular imports; it's practically a FAQ, at least to me.  ;-)

The problem with from import is that it always tries to
getattr(a,'b'), even if 'a.b' is in sys.modules.  In contrast, a plain
import will simply fetch a.b from sys.modules first.

In the case of a normal import, this is no problem, because a.b is set
to sys.modules['a.b'] at the end of the module loading process.  But
if the import is circular, then the module is in sys.modules['a.b'],
but *not* yet bound to a.b.  So the from import fails.

So, this is actually an implementation quirk that could be fixed in a
(somewhat) straightforward manner: by making from a import b succeed
if 'a.b' is in sys.modules, the same way import a.b does.  It would
require a little bit of discussion to hash out the exact way to do it,
but it could be done.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] relative import circular problem

[PJ Eby]

On Thu, Apr 4, 2013 at 4:42 PM, Guido van Rossum gu...@python.org wrote:
 I do think it would be fine if from a import b returned the
 attribute 'b' of module 'a' if it exists, and otherwise look for
 module 'a.b' in sys.modules.

Technically, it already does that -- but inside of __import__, not in
the IMPORT_FROM opcode.

But then *after* doing that check-and-fallback, __import__ doesn't
assign a.b, because it assumes the recursive import it called has
already done this...  which means that when __import__ returns, the
IMPORT_FROM opcode tries and fails to do the getattr.

This could be fixed in one of two ways.   Either:

1. Change importlib._bootstrap._handle_fromlist() to set a.b if it
successfully imports 'a.b' (inside its duplicate handling for what
IMPORT_FROM does), or
2. Change the IMPORT_FROM opcode to handle the fallback itself

While the latter involves a bit of C coding, it has fewer potential
side-effects on the import system as a whole, and simply ensures that
if import would succeed, then so would from...import targeting the
same module.

(There might be other fixes I haven't thought of, but really, changing
IMPORT_FROM to fallback to a sys.modules check is probably by far the
least-invasive way to handle it.)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Can I introspect/reflect to get arguments exec()?

[PJ Eby]

On Thu, Mar 28, 2013 at 6:43 AM, Rocky Bernstein ro...@gnu.org wrote:
 Of course the debugger uses sys.settrace too, so the evil-ness of that is
 definitely not a concern. But possibly I need to make sure that since the
 DecoratorTools and the debugger both hook into trace hooks they play nice
 together and fire in the right order.

DecoratorTools' trace hooking is unrelated to its linecache
functionality.  All you need from it is the cache_source() function;
you can pretty much ignore everything else for your purposes.  You'll
just need to give it a phony filename to work with, and the associated
string.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Safely importing zip files with C extensions

[PJ Eby]

On Wed, Mar 27, 2013 at 5:19 PM, Bradley M. Froehle
brad.froe...@gmail.com wrote:
 I implemented just such a path hook  zipimporter plus the magic required
 for C extensions --- as a challenge to myself to learn more about the Python
 import mechanisms.

 See https://github.com/bfroehle/pydzipimport.

FYI, there appears to be a bug for Windows with packages: you're using
'/__init__' in a couple places that should actually be
os.sep+'__init__'.

This does seem like a good way to address the issue, for those rare
situations where this would be a good idea.

The zipped .egg approach was originally intended for user-managed
plugin directories for certain types of extensible platforms, where
download a file and stick it in the plugins directory is a
low-effort way to install plugins, without having to build a lot of
specialized install capability.

As Jim has pointed out, though, this doesn't generalize well to a
full-blown packaging system.

Technically, you can blame Bob Ippolito for this, since he's the one
who talked me into using eggs to install Python libraries in general,
not just as a plugin packaging mechanism.  ;-)

That being said, *unpacked* egg, er, wheels, are still a great way to
meet all of the different apps needing different versions use cases
(without needing one venv per app), and nowadays the existence of
automated installer tools means that using one to install a plugin for
a low-tech plugin system is not a big deal, as long as that tool
supports the simple unpacked wheel scenario.  So I wholeheartedly
support some kind of mount/unmount or require-type mechanism for
finding plugins.  pkg_resources even has an API for handling simple
dynamic plugin dependency resolution scenarios:

  http://peak.telecommunity.com/DevCenter/PkgResources#locating-plugins

It'd be a good idea if distlib provides a similar feature, or at least
the APIs upon which apps or frameworks can implement such features.

(Historical note for those who weren't around back then: easy_install
wasn't even an *idea* until well after eggs were created; the original
idea was just that people would build plugins and libraries as eggs
and manually drop them in directories, where a plugin support library
would discover them and add them to sys.path as needed.  And Bob and I
also considered a sort of update site mechanism ala Eclipse, with a
library to let apps fetch plugins.  But as soon as eggs existed and
PyPI allowed uploads, it was kind of an obvious follow-up to make an
installation tool as a kind of technology demonstration  which
promptly became a monster.   The full story with all its twists and
turns can also be found here:
http://mail.python.org/pipermail/python-dev/2006-April/064145.html )
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Can I introspect/reflect to get arguments exec()?

[PJ Eby]

On Tue, Mar 26, 2013 at 11:00 PM, Rocky Bernstein ro...@gnu.org wrote:
 Okay. But is the string is still somewhere in the CPython VM stack? (The
 result of LOAD_CONST 4 above). Is there a way to pick it up from there?

Maybe using C you could peek into the frame's value stack, but that's
not exposed to any Python API I know of.  But that still doesn't help
you, because the value will be removed from the stack before exec() is
actually called, which means if you go looking for it in code called
from the exec (e.g. the call event itself), you aren't going to see
the data.

 At the point that we are stopped the exec action hasn't taken place yet.

That doesn't help if you're using line-level tracing events.  At the
beginning of the line, the data's not on the call stack yet, and by
the time you enter the frame of the code being exec'd, it'll be off
the stack again.

Basically, there is no way to do what you're asking for, short of
replacing the built-in exec function with your own version.  And it
still won't help you with stepping through the source of functions
that are *compiled* using an exec() or compile(), or any other way of
ending up with dynamically-generated code you want to debug.

(Unless you use something like DecoratorTools to generate it, that is
-- DecoratorTools has some facilities for caching
dynamically-generated code so that it works properly with debuggers.
But that has to be done by the code doing the generation, not the
debugger.  If the code generator uses DecoratorTools' caching support,
then any debugger that uses the linecache module will Just Work.  It
might be nice for  the stdlib should have something like this, but you
could also potentially fake it by replacing the builtin eval, exec,
compile, etc. functions w/versions that cache the source.)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Planning on removing cache invalidation for file finders

[PJ Eby]

On Sun, Mar 3, 2013 at 12:31 PM, Brett Cannon br...@python.org wrote:
 But how about this as a compromise over introducing write_module():
 invalidate_caches() can take a path for something to specifically
 invalidate. The path can then be passed to the invalidate_caches() on
 sys.meta_path. In the case of PathFinder it would take that path, try to
 find the directory in sys.path_importer_cache, and then invalidate the most
 specific finder for that path (if there is one that has any directory prefix
 match).

 Lots of little details to specify (e.g. absolute path forced anywhere in
 case a relative path is passed in by sys.path is all absolute paths? How do
 we know something is a file if it has not been written yet?), but this would
 prevent importlib from subsuming file writing specifically for source files
 and minimize performance overhead of invalidating all caches for a single
 file.

ISTR that when we were first discussing caching, I'd proposed a
TTL-based workaround for the timestamp granularity problem, and it was
mooted because somebody already proposed and implemented a similar
idea.  But my approach -- or at least the one I have in mind now --
would provide an eventual consistency guarantee, while still
allowing fast startup times.

However I think the experience with this heuristic so far shows that
the real problem isn't that the heuristic doesn't work for the normal
case; it works fine for that.  Instead, what happens is that *it
doesn't work when you generate modules*.

And *that* problem can be fixed without even invalidating the caches:
it can be fixed by doing some extra work when writing a module - e.g.
by making sure the directory mtime changes again after the module is
written.

For example, create the module under a temporary name, verify that the
directory mtime is different than it was before, then keep renaming it
to different temporary names until the mtime changes again, then
rename it to the final name.  (This would be very fast on some
platforms, much slower on others, but the OS itself would tell you
when it had worked.)  A utility routine to write_module() or
write_package() would be easier to find than advice that says to
invalidate the cache under thus-and-such conditions, as it would show
up in searches for writing modules dynamically or creating modules
dynamically, where you could only search for info about the cache if
you knew the cache existed.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0

[PJ Eby]

On Wed, Feb 20, 2013 at 5:30 AM, M.-A. Lemburg m...@egenix.com wrote:
 The wording in the PEP alienates the egg format by defining
 an incompatible new standard for the location of the metadata
 file:

This isn't a problem, because there's not really a use case at the
moment for eggs to include a PEP 426-format metadata file, and if they
ever do, it ought to be called METADATA, so that pkg_resources will
know to read it if there are no egg-format dependencies listed.
Setuptools also doesn't care what format PKG-INFO is in, as it only
ever reads the Version: field, and that only in the case of
in-development source packages.

 It's easy to upgrade distribute and distutils to write
 metadata 1.2 format, simply by changing the version in the
 PKG-INFO files.

As soon as distutils does it, setuptools will do it, because
setuptools delegates metadata writing to distutils.  So there's no
alienation here.

What will need to happen at some point is for pkg_resources to
implement support for PEP 426-style version requirements, which I
haven't tried to fully wrap my head around as yet.  I'm hoping that
there are simple textual substitutions (e.g. regexes) that can be done
to convert them to pkg_resources requirements.  If need be, I'll swipe
whatever's needed from distlib.  ;-)

In the meantime, eggs aren't actually going anywhere, since wheels
aren't actually trying to replace all of their use cases.  And since
the new metadata and binary formats don't actually add much new
functionality over what eggs already do, eggs wouldn't lose much by
not being able to use the same metadata, anyway.  ;-)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Distutils] PEP 426 is now the draft spec for distribution metadata 2.0

[PJ Eby]

On Tue, Feb 19, 2013 at 6:42 AM, Nick Coghlan ncogh...@gmail.com wrote:
 Nothing in the PEP is particularly original - almost all of it is
 either stolen from other build and packaging systems, or is designed
 to provide a *discoverable* alternative to existing
 setuptools/distribute/pip practices (specifically, the new extension
 mechanism means things like entry points can be defined in the
 standard metadata files without python-dev needing to get involved).

FWIW, I actually think this is a step in the wrong direction relative
to eggs; the ability to add new metadata files is a useful feature for
application frameworks.  For example, the EggTranslations project uses
egg metadata to implement resource localization plugins.  It lets you
have an application with plugins that either contain their own
translations, contain multiple translations for another plugin, a
single language translation for an assortment of plugins, etc.

These kinds of runtime-discovery use cases haven't seen much attention
in the metadata standard discussion.  On one level, that's fine,
because it makes sense that distribution-provided metadata should be
parseable by all tools, and that at build/download/install time the
performance and ease-of-use favor a single file approach.  That does
not mean, however, that the presence of other files is bad or should
be deprecated.  IMO, metadata that see significant runtime use
independent of the core metadata *should* appear in their own files,
even if redundant.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Submitting PEP 422 (Simple class initialization hook) for pronouncement

[PJ Eby]

On Mon, Feb 11, 2013 at 12:44 PM, Guido van Rossum gu...@python.org wrote:
 Hi Nick,

 I think this will make a fine addition to the language. I agree that
 it is superior to the alternatives and fulfills a real (if rare) need.

 I only have a few nits/questions/suggestions.

 - With PJE, I think __init_class__ should automatically be a class
 method.

Actually, I didn't say that as such, because I'm not sure how the heck
we'd implement that.  ;-)

For example, at what point is it converted to a classmethod?  Is it
going to be a slot method with special C-level handling?  Handled by
the compiler?  What happens if somebody makes it a

 The same way that __new__ is automatically a class method.

Actually, isn't it automatically a staticmethod?  Oh crap.  Now that
I'm thinking about it, doesn't this *have* to be a static method,
explicitly passing in the class?  I mean, otherwise, won't calling
super().__init_class__() invoke it on the base class, rather than the
current class?

ISTM that EIBTI argues for the __new__/staticmethod approach,
especially if you're returning the class (per below)


 - Would it make any sense to require that __init_class__ *returns* the
 new class object (to complete the similarity with class decorators)?

It would certainly be quite useful to do so, but in that case, perhaps
the method should be named __decorate_class__?  And in that event the
standard usage would look like:

def __decorate_class__(cls):
cls = super().__decorate_class__(cls)
# do stuff
return cls

On the other hand, one could just drop the super() requirement and
make the usage even simpler by having the class machinery walk the MRO
and pass each method the result of invoking the previous one.  Then
the methods are short and sweet, and super() and __class__ don't come
into it.  (Though I guess the class machinery could keep setting
__class__ to whatever the last-returned class was.)

In his first draft, Nick implemented inheritable decorators instead,
using a __decorators__ attribute in the class body, or something like
that.  While that approach had an issue or two of its own, it's
possible that just going with a single __decorate_class__ method would
work out better.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Submitting PEP 422 (Simple class initialization hook) for pronouncement

[PJ Eby]

On Sun, Feb 10, 2013 at 7:32 AM, Nick Coghlan ncogh...@gmail.com wrote:
class Example:
@classmethod
def __init_class__(cls):

Is the @classmethod required?  What happens if it's not present?

Second, will type have a default __init_class__?  (IMO, it should,
otherwise it will be impossible to write co-operative __init_class__
functions.)

Only other comment is that the PEP could use a more concrete use case, e.g.:

class Record:
__fields = {}

@classmethod
def __init_class__(cls):
 cls.__fields = dict(cls.__fields)  # inherited fields
 cls.__fields.update({attr:val for attr, val in
cls.__dict__.iteritems() if isinstance(val, Field)})
 super().__init_class__()   # be co-operative

 # ...other methods that use the __fields attribute

class SomeRecord(Record):
  foo = Field(int)
  bar = Field(str)

Putting something like this early on might help to demonstrate the
usefulness of the feature on its own merits, independent of the
porting issue, etc.  ;-)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Submitting PEP 422 (Simple class initialization hook) for pronouncement

[PJ Eby]

On Sun, Feb 10, 2013 at 11:48 AM, Stefan Behnel stefan...@behnel.de wrote:
 So, the way to explain it to users would be 1) don't use it, 2) if you
 really need to do something to a class, use a decorator, 3) if you need to
 decide dynamically what to do, define __init_class__() and 4) don't forget
 to call super's __init_class__() in that case, and 5) only if you need to
 do something substantially more involved and know what you're doing, use a
 metaclass.

I'd revise that to:

1) if there's no harm in forgetting to decorate a subclass, use a
class decorator
2) if you want to ensure that a modification is applied to every
subclass of a single common base class, define __init_class__ (and
always call its super)
3) If you need to make the class object *act* differently (not just
initialize it or trigger some other side-effect at creation time), or
if you want the class suite to return some other kind of object,
you'll need a metaclass.

Essentially, this change fixes a hole in class decorators that doesn't
exist with function decorators: if you need the decoration applied to
subclasses, you can end up with silent failures right now.
Conversely, if you try prevent such failures using a metaclass, you
not only have a big hill to climb, but the resulting code will be
vulnerable to metaclass conflicts.

The proposed solution neatly fixes both of these problems, providing
One Obvious Way to do subclass initialization.

(An alternative, I suppose, would be to let you do something like
@@someClassDecorator to have inheritable class decorators, but as Nick
has pointed out, implementing inheritable decorators is a lot more
complicated.)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Why does Signature.from_function() have to check the type of its argument?

[PJ Eby]

On Fri, Feb 8, 2013 at 5:44 PM, Stefan Behnel stefan...@behnel.de wrote:
 Argh - sorry, got it all wrong. __instancecheck__() works exactly the
 other way round. In the type check above, it's the FunctionType type that
 gets asked for an instance check, which doesn't help at all in this case
 because it simply doesn't know about the desired subtype relation. It would
 work if type(func).__instancecheck__() was used, but that doesn't happen.

 So, no help from that side, sadly.

How about you return FunctionType as your __class__ attribute?  ;-)

Your type() will still be different, but isinstance() also considers
the __class__ if it's different from the type().  (At least it does in
2.x, I've not tried it in any 3.x versions yet...)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3)

[PJ Eby]

On Sun, Feb 3, 2013 at 8:08 AM, Nick Coghlan ncogh...@gmail.com wrote:
 The rationale for the distutils freeze is don't break setuptools.
 That rationale still holds.

IIRC, the historical issue that triggered the freeze was not that the
distutils refactoring broke setuptools, but that it did so in what was
supposed to be a bugfix-only release of Python, via a change to what
appeared to be a public/documented method.

This rationale doesn't apply to major/minor releases of Python, and
the freeze was only supposed to apply to 2.x in any event.  (IIRC,
some of the breakage was the result of backporting 3.x distutils
changes to 2.x.)

Anyway, adding features or refactoring distutils is not the problem:
doing it in bugfix releases is.  With normal releases, third-party
extenders of distutils at least have the opportunity to test and issue
updates in a timely fashion.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Accessing value stack

[PJ Eby]

On Mon, Jan 7, 2013 at 11:32 AM, Brett Cannon br...@python.org wrote:
 On Mon, Jan 7, 2013 at 10:46 AM, Maciej Fijalkowski fij...@gmail.com wrote:
 On Mon, Jan 7, 2013 at 4:22 PM, Oleg Broytman p...@phdru.name wrote:
 On Mon, Jan 07, 2013 at 03:06:51PM +0100, Dima Tisnek dim...@gmail.com 
 wrote:
 Hi, is it possible to access the values stored on the stack in Python stack
 machine from Python?

In short: it's possible but very much discouraged. Don't hack into
 python internals.

 Is it possible? I claim it's not (from *Python* because obviously data
 is in memory).

 Nope, it's not.

I took that as a challenge, and just tried to do it using
gc.get_referents().  ;-)

Didn't work though...  which actually makes me wonder if that's a bug
in gc.get_referents(), or whether I'm making a bad assumption about
when you'd have to run gc.get_referents() on a frame in order to see
items from the value stack included.  Could this actually be a bug in
frames' GC implementation, or are value stack items not supposed to
count for GC purposes?
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] More compact dictionaries with faster iteration

[PJ Eby]

On Wed, Dec 12, 2012 at 3:37 PM, Dag Sverre Seljebotn
d.s.seljeb...@astro.uio.no wrote:
 On 12/12/2012 01:15 AM, Nick Coghlan wrote:

 On Wed, Dec 12, 2012 at 5:37 AM, Dino Viehland di...@microsoft.com
 mailto:di...@microsoft.com wrote:

 OTOH changing certain dictionaries in IronPython (such as keyword
 args) to be
 ordered would certainly be possible.  Personally I just wouldn't
 want to see it
 be the default as that seems like unnecessary overhead when the
 specialized
 class exists.


 Which reminds me, I was going to note that one of the main gains with
 ordered keyword arguments, is their use in the construction of
 string-keyed objects where you want to be able to control the order of
 iteration (e.g. for serialisation or display purposes). Currently you
 have to go the path of something like namedtuple where you define the
 order of iteration in one operation, and set the values in another.


 So here's a brand new argument against ordered dicts: The existence of
 perfect hashing schemes. They fundamentally conflict with ordered dicts.

If I understand your explanation, then they don't conflict with the
type of ordering described in this thread.  Raymond's optimization
separates the hash table part from the contents part of a
dictionary, and there is no requirement that these two parts be in the
same size or the same order.

Indeed, Raymond's split design lets you re-parameterize the hashing
all you want, without perturbing the iteration order at all.  You
would in fact be able to take a dictionary at any moment, and say,
optimize the 'hash table' part to a non-colliding state based on the
current contents, without touching the 'contents' part at all.

(One could do this at class creation time on a class dictionary, and
just after importing on a module dictionary, for example.)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] More compact dictionaries with faster iteration

[PJ Eby]

On Wed, Dec 12, 2012 at 5:06 PM, Dag Sverre Seljebotn
d.s.seljeb...@astro.uio.no wrote:
 Perfect hashing already separates hash table from contents (sort of),
 and saves the memory in much the same way.

 Yes, you can repeat the trick and have 2 levels of indirection, but that
 then requires an additional table of small ints which is pure overhead
 present for the sorting; in short, it's no longer an optimization but just
 overhead for the sortability.

I'm confused.  I understood your algorithm to require repacking,
rather than it being a suitable algorithm for incremental change to an
existing dictionary.  ISTM that that would mean you still pay some
sort of overhead (either in time or space) while the dictionary is
still being mutated.

Also, I'm not sure how 2 levels of indirection come into it.   The
algorithm you describe has, as I understand it, only up to 12
perturbation values (bins), so it's a constant space overhead, not a
variable one.  What's more, you can possibly avoid the extra memory
access by using a different perfect hashing algorithm, at the cost of
a slower optimization step or using a little more memory.

 Note: I'm NOT suggesting the use of perfect hashing, just making
 sure it's existence is mentioned and that one is aware that if
 always-ordered dicts become the language standard it precludes
 this option far off in the future.

Not really.  It means that some forms of perfect hashing might require
adding a few more ints worth of overhead for the dictionaries that use
it.  If it's really a performance benefit for very-frequently-used
dictionaries, that might still be worthwhile.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] More compact dictionaries with faster iteration

[PJ Eby]

On Mon, Dec 10, 2012 at 10:48 AM, Antoine Pitrou solip...@pitrou.net wrote:
 Le Mon, 10 Dec 2012 10:40:30 +0100,
 Armin Rigo ar...@tunes.org a écrit :
 Hi Raymond,

 On Mon, Dec 10, 2012 at 2:44 AM, Raymond Hettinger
 raymond.hettin...@gmail.com wrote:
  Instead, the data should be organized as follows:
 
  indices =  [None, 1, None, None, None, 0, None, 2]
  entries =  [[-9092791511155847987, 'timmy', 'red'],
  [-8522787127447073495, 'barry', 'green'],
  [-6480567542315338377, 'guido', 'blue']]

 As a side note, your suggestion also enables order-preserving
 dictionaries: iter() would automatically yield items in the order they
 were inserted, as long as there was no deletion.  People will
 immediately start relying on this feature...  and be confused by the
 behavior of deletion. :-/

 If that's really an issue, we can deliberately scramble the iteration
 order a bit :-) (of course it might negatively impact HW prefetching)

On the other hand, this would also make a fast ordered dictionary
subclass possible, just by not using the free list for additions,
combined with periodic compaction before adds or after deletes.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Keyword meanings [was: Accept just PEP-0426]

[PJ Eby]

On Sun, Dec 9, 2012 at 10:38 PM, Andrew McNabb amcn...@mcnabbs.org wrote:
 On Fri, Dec 07, 2012 at 05:02:26PM -0500, PJ Eby wrote:
 If the packages have files in conflict, they won't be both installed.
 If they don't have files in conflict, there's nothing important to be
 informed of.  If one is installing pexpect-u, then one does not need
 to discover that it is a successor of pexpect.

 In the specific case of pexpect and pexpect-u, the files don't actually
 conflict.  The pexpect package includes a pexpect.py file, while
 pexpect-u includes a pexpect/ directory.  These conflict, but not in
 the easily detectable sense.

Excellent!  A concrete non-file use case.  Setuptools handles this
particular scenario by including a list of top-level module or package
names, but newer tools ought to look out for this scenario, too.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Keyword meanings [was: Accept just PEP-0426]

[PJ Eby]

On Mon, Dec 10, 2012 at 3:27 AM, Stephen J. Turnbull step...@xemacs.org wrote:
 PJ Eby writes:

   By clear, I mean free of prior assumptions.

 Ah, well, I guess I've just run into a personal limitation.  I can't
 imagine thinking that is free of prior assumptions.  Not my
 ownwink/, and not by others, either.

I suppose I should have said, free of *known* prior assumptions,
since the trick to suspending assumptions is to find the ones you
*have*.

The deeper assumptions, alas, can usually only be found by clashing
opinions with others...  then stepping back and going, wait...  what
does he/she believe that's *different* from what I believe, that
allows them to have that differing opinion?

And then that's how you find out what it is that *you're* assuming,
that you didn't know you were assuming.  ;-)  (Not to mention what the
other person is.)


   Put together, the phrase clear thinking on concrete use cases means
   (at least to me), dropping all preconceptions of the existing design
   and starting over from square one, to ask how best the problem may be
   solved, using specific examples as a guide rather than using
   generalities.

 Sure, but ISTM that's the opposite of what you've actually been doing,
 at least in terms of contributing to my understanding.  One obstacle
 to discussion you have contributed to overcoming in my thinking is the
 big generality that the packager (ie, the person writing the metadata)
 is in a position to recommend good behavior to the installation
 tool, vs. being in a position to point out relevant considerations
 for users and tools installing the packager's product.

Right, but I started from a concrete scenario I wanted to avoid, which
led me to question the assumption that those fields were actually
useful.  As soon as I began questioning *that* assumption and asking
for use cases (2 years ago, in the last PEP 345 revision discussion),
it became apparent to me that there was something seriously wrong with
the conflicts and obsoletes fields, as they had almost no real utility
as they were defined and understood at that point.

 Until that generality is formulated and expressed, it's very difficult
 to see why the examples and particular solutions to use cases that
 various proponents have described fail to address some real problems.

Unfortunately, it's a chicken-and-egg problem: until you know what
assumptions are being made, you can't formulate them.  It's an
iterative process of exposing assumptions, until you succeed in
actually communicating.  ;-)

Heck, even something as simple as my assumptions about what clear
thinking meant and what I was trying to say has taken some back and
forth to clarify.  ;-)

 It was difficult for me to see, at first, what distinction was
 actually being made.

 Specifically, I thought that the question about Obsoletes vs.
 Obsoleted-By was about which package should be considered
 authoritative about obsolescence.  That is a reasonable distinction
 for that particular discussion, but there is a deeper, and general,
 principle behind that.  Namely, metadata is descriptive, not
 prescriptive.

Actually, the principle I was clinging to for *both* fields was not
giving project authors authority over other people's projects.  It's
fine for metadata to be prescriptive (e.g. requirements), it's just
that it should be prescriptive *only* for that project in isolation.
(In the broader sense, it also applies to the distro situation: the
project author doesn't really have authority over the distro, either,
so it can only be a suggestion there, as well.)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] More compact dictionaries with faster iteration

[PJ Eby]

On Mon, Dec 10, 2012 at 1:01 PM, Armin Rigo ar...@tunes.org wrote:
 On Mon, Dec 10, 2012 at 5:16 PM, PJ Eby p...@telecommunity.com wrote:
 On the other hand, this would also make a fast ordered dictionary
 subclass possible, just by not using the free list for additions,
 combined with periodic compaction before adds or after deletes.

 Technically, I could see Python switching to ordered dictionaries
 everywhere.  Raymond's insight suddenly makes it easy for CPython and
 PyPy, and at least Jython could use the LinkedHashMap class (although
 this would need checking with Jython guys).

What about IronPython?

Also, note that using ordered dictionaries carries a performance cost
for dictionaries whose keys change a lot.  This probably wouldn't
affect most dictionaries in most programs, because module and object
dictionaries generally don't delete and re-add a lot of keys very
often. But in cases where a dictionary is used as a queue or stack or
something of that sort, the packing costs could add up.  Under the
current scheme, as long as collisions were minimal, the contents
wouldn't be repacked very often.

Without numbers it's hard to say for certain, but the advantage to
keeping ordered dictionaries a distinct type is that the standard
dictionary type can then get that extra bit of speed in exchange for
dropping the ordering requirement.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] More compact dictionaries with faster iteration

[PJ Eby]

On Mon, Dec 10, 2012 at 4:29 PM, Tim Delaney tim.dela...@aptare.com wrote:
 Whilst I think Python should not move to ordered dictionaries everywhere, I
 would say there is an argument (no pun intended) for making **kwargs a
 dictionary that maintains insertion order *if there are no deletions*.

Oooh.  Me likey.  There have been many times where I've wished kwargs
were ordered when designing an API.

(Oddly, I don't remember any one of the APIs specifically, so don't
ask me for a good example.  I just remember a bunch of different
physical locations where I was when I thought, Ooh, what if I
could...  no, that's not going to work.)

One other useful place for ordered dictionaries is class definitions
processed by class decorators: no need to write a metaclass just to
know what order stuff was defined in.

  It sounds like we could get that for free with this implementation, although
 from another post IronPython might not have something suitable.

Actually, IronPython may already have ordered dictionaries by default; see:

  http://mail.python.org/pipermail/ironpython-users/2006-May/002319.html

It's described as an implementation detail that may change, perhaps
that could be changed to being unchanging.  ;-)


 I think there are real advantages to doing so - a trivial one being the 
 ability
 to easily initialise an ordered dictionary from another ordered dictionary.

Or to merge two of them together, either at creation or .update().

I'm really starting to wonder if it might not be worth paying the
compaction overhead to just make all dictionaries ordered, all the
time.  The problem is that if you are always adding new keys and
deleting old ones (as might be done in a LRU cache, a queue, or other
things like that) you'll probably see a lot of compaction overhead
compared to today's dicts.

OTOH...  with a good algorithm for deciding *when* to compact, we can
actually make the amortized cost O(1) or so, so maybe that's not a big
deal.  The cost to do a compaction is at worst, the current size of
the table.  So if you wait until a table has twice as many entries
(more precisely, until the index of the last entry is twice what it
was at last compaction), you will amortize the compaction cost down to
one entry move per add, or O(1).

That would handle the case of a cache or queue, but I'm not sure how
it would work with supersized dictionaries that are then deleted down
to a fraction of their original size.  I suppose if you delete your
way down to half the entries being populated, then you end up with two
moves per delete, or O(2).  (Yes, I know that's not a valid O number.)

So, offhand, it seems pretty doable, and unlikely to significantly
change worst-case performance even for pathological use cases.  (Like
using a dict when you should be using a deque.)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] More compact dictionaries with faster iteration

[PJ Eby]

On Mon, Dec 10, 2012 at 5:14 PM, Andrew Svetlov
andrew.svet...@gmail.com wrote:
 Please, no. dict and kwargs should be unordered without any assumptions.

Why?
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Keyword meanings [was: Accept just PEP-0426]

[PJ Eby]

On Sun, Dec 9, 2012 at 12:54 AM, Nick Coghlan ncogh...@gmail.com wrote:
 On Sun, Dec 9, 2012 at 6:18 AM, PJ Eby p...@telecommunity.com wrote:

 On Sat, Dec 8, 2012 at 5:06 AM, Nick Coghlan ncogh...@gmail.com wrote:
  On Sat, Dec 8, 2012 at 4:46 PM, PJ Eby p...@telecommunity.com wrote:
 
  So if package A includes a Conflicts: B declaration, I recommend the
  following:
 
  * An attempt to install A with B already present refuses to install A
  without a warning and confirmation
  * An attempt to install B informs the user of the conflict, and
  optionally offers to uninstall A
 
  In this way, any collateral damage to B is avoided, while still making
  the intended lack of support declaration clear.
 
  How does that sound?
 
 
  No, that's not the way it works. A conflict is always symmetric, no
  matter
  who declares it.

 But that *precisely contradicts* what you said in your previous email:

  It's to allow a project to say
  *they don't support* installing in parallel with another package.

 Just because A doesn't support being installed next to B, doesn't mean
 B doesn't support being installed next to A.  B might work just fine
 with A installed, and even be explicitly supported by the author of B.
  Why should the author of A get to decide what happens to B?  Just
 because I trust A about A, doesn't mean I should have to trust them
 about B.


 If I'm installing both A *and* B, I want to know if *either* project doesn't
 support that configuration. The order in which they get installed should
 *not* have any impact on my finding out that I am using one of my
 dependencies in an unsupported way that may cause me unanticipated problems
 further down the line.

This is probably moot now, but I didn't propose that installation
order matter -- in both scenarios I described, you end up with a
warning and A not installed, regardless of whether A or B were
installed first.

 The author of A *doesn't* get to decide what happens to B, *I* do.

The reason I said, (de facto), is because the default behavior of
whatever the next big installation tool is, would be what most users
would've gotten by default.

 They're
 merely providing a heads up that they believe there are problems when using
 their project in conjunction with B. My options will be:
 - use them both anyway (e.g. perhaps after doing some research, I may find
 out the conflict relates solely to a feature of B that I'm not using, so I
 simply update my project documentation to say do not use feature X from
 project B, as it conflicts with dependency A)
 - choose to continue using A, find another solution for B
 - choose to continue using B, find another solution for A

 As a concrete example, there are projects out there that are known not to
 work with gevent's socket monkeypatching, but people don't know that until
 they try it and it blows up in their face.

Here's the question, though: who's going to maintain that list?

I can see gevent wanting to have a compatibility chart page in their
docs, but it seems unlikely they'd want to block installation of
non-gevent-compatible projects or vice versa.  Similarly, I can't see
why any of those other projects would want to block installation of
gevent, or vice versa.

That being said, I don't object to having the ability for either of
them to do so: the utility of the field is *much* enhanced once its
connection to installation tools is gone, since a wider variety of
issues can be described without inconveniencing users.


 I now agree that *enforcing* a conflicts field at install time in a Python
 installer doesn't make any sense, since the nature of Python means it will
 often be easy to sidestep any such issues once you're aware of their
 existence (e.g. by avoiding gevent's monkeypatching features and using
 threads to interact with the uncooperative synchronous library, or by
 splitting your application into multiple processes, some using gevent and
 others synchronous sockets). I also believe that *any* Conflicts declaration
 *should* be backed up with an explicit explanation and rationale for that
 conflict declaration in the project documentation.

Beyond that, I think a reference URL should be included *in the field
itself*, e.g. to a bug report, support ticket, or other page that
documents the incompatibility and will be updated as the situation
changes.  The actual usefulness of the field to anyone downstream
seems greatly reduced if they have to go hunting for the information
explaining the compatibility issue(s).

This is a good example of what I meant about clear thinking on
concrete use cases, vs. simply copying fields from distro tools.  In
the distro world, these kinds of fields reflect the *results* of
research and decision-making about compatibility.  Whereas, in our
upstream world, the purpose of the fields is to provide downstream
repackagers and integrators with the source materials for such
research.


 So, I still like the idea of including a Conflicts field, but think a few

Re: [Python-Dev] Keyword meanings [was: Accept just PEP-0426]

[PJ Eby]

On Sun, Dec 9, 2012 at 8:48 PM, Stephen J. Turnbull step...@xemacs.org wrote:
 PJ Eby writes:
   This is a good example of what I meant about clear thinking on
   concrete use cases, vs. simply copying fields from distro tools.  In
   the distro world, these kinds of fields reflect the *results* of
   research and decision-making about compatibility.  Whereas, in our
   upstream world, the purpose of the fields is to provide downstream
   repackagers and integrators with the source materials for such
   research.

 I agree with the meaning of the above paragraph, but would like to
 dissociate myself from the comparison implied by the expression clear
 thinking.

What comparison is that?

By clear, I mean free of prior assumptions.   The assumptions that
made the discussion difficult weren't just about the use cases
themselves, but about the environments, tools, organizations,
concepts, etc. surrounding those use cases.  Indeed, even the
assumption of what should *qualify* as a use case was a stumbling
block on occasion.  ;-)

And by thinking, I mean, considering alternatives and
consequences, as distinct from debating the merits of a specific
position.

Put together, the phrase clear thinking on concrete use cases means
(at least to me), dropping all preconceptions of the existing design
and starting over from square one, to ask how best the problem may be
solved, using specific examples as a guide rather than using
generalities.  Generalities not rooted in concrete examples have a
way of leading to non-terminating discussions.  ;-)

Starting over a discussion in this fashion isn't easy, but the results
are usually worth it.  I appreciate Nick and Daniel's patience in
particular.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Keyword meanings [was: Accept just PEP-0426]

[PJ Eby]

On Sat, Dec 8, 2012 at 5:06 AM, Nick Coghlan ncogh...@gmail.com wrote:
 On Sat, Dec 8, 2012 at 4:46 PM, PJ Eby p...@telecommunity.com wrote:

 So if package A includes a Conflicts: B declaration, I recommend the
 following:

 * An attempt to install A with B already present refuses to install A
 without a warning and confirmation
 * An attempt to install B informs the user of the conflict, and
 optionally offers to uninstall A

 In this way, any collateral damage to B is avoided, while still making
 the intended lack of support declaration clear.

 How does that sound?


 No, that's not the way it works. A conflict is always symmetric, no matter
 who declares it.

But that *precisely contradicts* what you said in your previous email:

 It's to allow a project to say
 *they don't support* installing in parallel with another package.

Just because A doesn't support being installed next to B, doesn't mean
B doesn't support being installed next to A.  B might work just fine
with A installed, and even be explicitly supported by the author of B.
 Why should the author of A get to decide what happens to B?  Just
because I trust A about A, doesn't mean I should have to trust them
about B.

Look, I really don't care about the individual fields' definitions
that much.  I care about only one thing: A shouldn't get to (de facto)
dictate what happens to B.  If you *really* want the behavior to be
symmetrical, then it should *only* be symmetrical if both A and B
*agree* they are in conflict.  (i.e., both refer to the other in their
conflict fields).  Otherwise, it should only be a warning.

There are tons of other things that I could argue here about the
positions you've laid out.  But all I *really* care about is that we
not define fields in such a way as to permit or encourage
inter-package warfare -- intentional or not.  Solutions acceptable to
me include (in no particular order):

* Make declarations affect only the declarer (as with Obsoleted-By)
* Make declarations only warn users, not block installation or result
in uninstallation
* Have no automated action at all, and document them as intended for
downstream repackagers only
* Toss the field entirely
* Make the field include a context (e.g. a distro name), so that only
tools explicitly told you're operating in that context pay attention
* Use the new metadata extension vocabularies to define hints for
specific downstream packaging tools and systems
* Replace conflicts with a specification of resources actually used
by the project, so that such conflicts can be automatically detected
without needing to target a specific project

And there are probably others I haven't thought of yet.  If you can be
clearer about what it is you want from the Conflicts field *other*
than just wanting it to stay as is (or perhaps *why* you would like to
have the Python infrastructure side with project A over project B,
irrespective of which project is A and which one is B), then perhaps I
can come up with others.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Conflicts [was Re: Keyword meanings [was: Accept just PEP-0426]]

[PJ Eby]

On Sat, Dec 8, 2012 at 10:22 PM, MRAB pyt...@mrabarnett.plus.com wrote:
 On 2012-12-09 01:15, Steven D'Aprano wrote:

 On 09/12/12 08:14, MRAB wrote:

 If package A says that it conflicts with package B, it may or may not
 be symmetrical, because it's possible that package B has been updated
 since the author of package A discovered the conflict, so it's
 important that the user is told which package is complaining about the
 conflict, the one that is being installed or the one that is already
 installed.


 I must admit than in reading this thread, I'm having a bit of trouble
 understanding why merely *installing* packages should lead to conflicts.

 [snip]
 Personally speaking, I was thinking more about possible problems at
 runtime due to functional conflicts, but it could apply to any
 (undefined) conflict.

If it's for a runtime functional conflict, there's no need for
installation tools to worry about it, except perhaps in the case where
a single project C depends on *both* A and B, where A and B conflict
with each other.  Apart from that piece of information, there is no
way to know that the code will ever even be imported at the same time.
 (And even then, it's just a hint of the possibility, not a
guarantee.)

Nick, OTOH, says that the purpose of the field is to declare that mere
side-by-side installation invalidates developer support for the
configuration.

However, the widespread confusion (conflicts?) over what exactly the
field is supposed to mean and when it should be used suggests that its
charter is not nearly as clear as it should be.

It seems perhaps it is suffering from the so-called Illusion of
Transparency, wherein everybody looks at it and thinks that it
*obviously* means X, and only a fool could think otherwise...  except
that everyone has a *different* value of X in mind.

That's why I keep asking for specific, concrete use cases.  At this
point, for the field to make any sense, there needs to be some better
idea of what a runtime or undefined conflict is.  Apart from file
conflicts, has anybody identified a single PyPI package that would
make use of this field?  If so, what *is* that example, and what is
the nature of the conflict?

Do any of the distro folks know of a Python project tagged as
conflicting with another for their distro, where the conflict does
*not* involve any files in conflict?

(And the conflict is not specific to the distro's packaging of that
project and the project in conflict?  i.e., that it would have
actually been possible and/or meaningful for the upstream developer to
have flagged the conflict in the project's metadata, given the
proposed metadata standard?)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Keyword meanings [was: Accept just PEP-0426]

[PJ Eby]

On Fri, Dec 7, 2012 at 12:01 PM, Toshio Kuratomi a.bad...@gmail.com wrote:
 On Fri, Dec 07, 2012 at 01:18:40AM -0500, PJ Eby wrote:
 On Thu, Dec 6, 2012 at 1:49 AM, Toshio Kuratomi a.bad...@gmail.com wrote:
  On Wed, Dec 05, 2012 at 07:34:41PM -0500, PJ Eby wrote:
  Nobody has actually proposed a better one, outside of package renaming
  -- and that example featured an author who could just as easily have
  used an obsoleted-by field.
 
  How about pexpect and pextpect-u as a better example?

 Perhaps you could explain?  I'm not familiar with those projects.


 pexepect was last released in 2008.  Upstream went silent with unanswered
 bugs in its tracker and no mailing list.  A fork of pexpect was created that
 addressed the issue of unicode type in python2, a python3 port, and has
 slowly evolvd since then.

 I see that the original upstream has made some commits to their source
 repository since the fork was created although there has still been no new
 release.

And what problem are you saying which fields would have solved (or
which benefits they would have provided), for whom?

If the packages have files in conflict, they won't be both installed.
If they don't have files in conflict, there's nothing important to be
informed of.  If one is installing pexpect-u, then one does not need
to discover that it is a successor of pexpect.  If one is installing
pexpect, it might be useful to know that pexpect-u exists, but one
can't simply discover that from an Obsoletes field on pexpect-u.
However, even if one did discover it, this would merely constitute an
*advertisement* of pexpect-u's existence, not a *requirement* that it
be used in place.  A tool cannot know, without other affirmative user
action, that it is actually a good assumption to use the advertised
replacement.

In the distro world, a user has *already* taken this affirmative
action by choosing which repository to source packages from, on an
implicit contract that this source is up to the job of managing his
needs across multiple packages.  Or, if they choose to source an
off-brand or upstream package, they are taking affirmative action to
risk it.

In the Python world, there is no notion of a repository, aside from
a handful of managed Python distros, which have their own, distinct
packaging methods and distribution tools.  So there is no affirmative
contract of trust regarding *inter-project* relationships.

It is precisely this lack that is why the metadata spec has gone
mostly unused since its inception about a decade ago.  Nobody really
knows what to provide or require, or in what context they would
actually be obsoleting anything that isn't their own package, or a
package they've forked.

But if you live mainly in the distro world, this concept seems absurd,
and the fields *obviously* useful.  But that's because you're swimming
in an ocean of context that doesn't exist on dry land.  You're saying
that *of course* swimming fins are useful...  if you live in the
ocean.

And I, living on dry land, am saying that *sure* they are...  but only
in a swimming pool or a pond, and we don't have very many of those
here in dry Python-land.  And the people who run the swimming pools
have thoughtfully already provided their own.  Do we need to
standardize swim fin sizes for people who mostly live on dry land?

The flip side of this, btw, is that there's an implicit contract in
the Python world that there is generally only the package - not the
package as patched and re-packaged by vendors X, Y, and Z.  If I
install python project foo, version 1.2, I expect it to be the *same*
foo-1.2, with the *same metadata*, *no matter where I got it from*.

And so, this assumption is our air to your water.  We know that
pools and ponds (curated Python distros) are different, as an
exception to this rule, just as you know that reefs and islands
(uncurated repositories, search engines, and upstream-built packages)
are different, as an exception to your assumption that the package I
get is intended to play well with everything else in my system.

(This of course is why many distro managers are suspicious of
language-specific or other sorts of vertical package management tools
- they seem as pointless as wheels in the water, solving problems you
don't have, and creating new problems for you at the same time.
Unfortunately, people on land will keep inventing them, because they
have a different set of problems to solve -- some of which are
actually created by the ocean-oriented tools.  For example, virtualenv
and its predecessors were developed to solve the problem of a single
integrated environment, even though that integrated environment is the
solution from a distro perspective.)


 *) Not all packages built build on top of that system.  There are rpm
 packages provided by upstreams that users attempt (to greater and lesser
 degrees of success) to install on SuSE, RHEL, Fedora, Mandriva, etc.  There
 are debs built for Ubuntu that people attempt to install onto Debian

Re: [Python-Dev] Keyword meanings [was: Accept just PEP-0426]

[PJ Eby]

On Fri, Dec 7, 2012 at 8:33 PM, Nick Coghlan ncogh...@gmail.com wrote:
 That's not what a Conflicts field is for. It's to allow a project to say
 *they don't support* installing in parallel with another package.

If that's the actual intended use case, the PEP needs some revision.
In particular, if there's a behavioral recommendation for installer
tools, it should be to avoid installing the project that *declares*
the conflict, rather than the one that is the object of that
declaration.  ;-)

In any case, as I said before, I don't have an issue with the fields
all being declared as being for informational purposes only.  My issue
is only with recommendations for automated tool behavior that permit
one project's author to exercise authority over another project's
installation.  If the fields are defined in such a way that an author
can only shoot *themselves* in the foot with a bad declaration, that's
fine by me.

So if package A includes a Conflicts: B declaration, I recommend the
following:

* An attempt to install A with B already present refuses to install A
without a warning and confirmation
* An attempt to install B informs the user of the conflict, and
optionally offers to uninstall A

In this way, any collateral damage to B is avoided, while still making
the intended lack of support declaration clear.

How does that sound?
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Keyword meanings [was: Accept just PEP-0426]

[PJ Eby]

On Thu, Dec 6, 2012 at 8:39 AM, Daniel Holth dho...@gmail.com wrote:
 It will be Obsoleted-By:. The drop in replacement requirement will be
 removed. The package manager will say you are using these obsolete
 packages; check out these non-obsolete ones but will not automatically pull
 the replacement without a Requires tag.

Sounds fine to me.

 I will probably add the unambiguous Conflicts: tag uninstall this other
 package if I am installed.

Please don't.  See my lengthy posts from the previous PEP 345 retread
discussion for why, or ask MRAB to succinctly summarize them as he did
so brilliantly with the obsoletes/obsoleted-by issue.  ;-)

I'll take a stab at a short version, though: a conflict (other than
filename conflict) is not an installation-time property of a single
project, but rather a *runtime* property of an overall system to which
the projects are being installed, including configuration that is out
of scope for a Python-specific installation tool to manage.  In
addition, even declaring overall conflicts as a *mere shorthand* for
an existing file conflict creates the possibility of stale conflict
information!

For example, RuleDispatch vs. PyDispatcher: at one time both provided
a dispatch package, but if RuleDispatch declared PyDispatcher
conflicting, the declaration would quickly have become outdated:
PyDispatcher soon renamed its provided package to resolve the
conflict.  A file-based system can both detect and resolve this
conflict (or lack thereof) automatically, whereas a manual Conflicts
notation must be maintained by the author(s) of one or both packages
and removed when out of date.

In effect, a conflicts field actually *creates* conflicts and
maintenance burdens where they did not previously exist, because even
after the conflict no longer really existed, an automated tool would
have prevented PyDispatch from being installed, or, per your
suggestion above, unnecessarily *uninstalled* it after a user
installed RuleDispatch.

And unlike the Obsoletes-Obsoleted-By change, I do not know of any
similar way to salvage the idea of a Conflicts field, without
reference to some mediating authority that manages the information on
behalf of an overall system into which the projects are being fitted.
But in that case, neither of the projects really owns the declaration
- it's more like Zope (say) would need a list of plugins that conflict
with each other, or they could declare that they conflict when
activated in the same instance.

A generic Python installer, however, that doesn't know about Zope
instances or Apache vhosts or Django apps or any other environment of
conflict, can't assume that *mere installation* constitutes a
conflict!  It doesn't know, for example, whether code from two
simultaneously-installed packages will ever even be *imported* in the
same process, let alone whether their specific conflicting features
will be used in that process.

This effectively ensures that in general, Python installation tools
can *only* rely on file-based conflicts as being denotable by project
metadata -- and even then, it's better to stick with *actual* file
conflicts rather than predicted ones, to avoid the type of logjam
described above.


P.S. Sorry once again to drag you through all this at the last minute;
I just sort of assumed you picked up where Alexis left off on the
previous attempt at an update to PEP 345 and didn't pay close enough
attention to earlier drafts.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Keyword meanings [was: Accept just PEP-0426]

[PJ Eby]

On Thu, Dec 6, 2012 at 9:58 AM, Vinay Sajip vinay_sa...@yahoo.co.uk wrote:
 Daniel Holth dholth at gmail.com writes:

 The wheel implementation makes sure all the metadata (the .dist-info 
 directory)
 is at the end of the .zip archive. It's possible to read the metadata with a
 single HTTP partial request for the end of the archive without downloading 
 the
 entire archive.

 Sounds good, but can you point to any example code which does this? As I
 understand it, for .zip files you have to read the last part of the file to 
 get a
 pointer to the directory, then read that to find where each file in the 
 archive
 is, then seek to a specific position to read the file contents.

ISTR that this is especially true for zipimport: I think it depends on
a zipfile signature being present at the *end* of the file.

Certainly, the standard for .exe and shell wrappers for zipfiles is to
place them at the beginning of the file, rather than the end.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Keyword meanings [was: Accept just PEP-0426]

[PJ Eby]

On Thu, Dec 6, 2012 at 1:49 AM, Toshio Kuratomi a.bad...@gmail.com wrote:
 On Wed, Dec 05, 2012 at 07:34:41PM -0500, PJ Eby wrote:
 On Wed, Dec 5, 2012 at 6:07 PM, Donald Stufft donald.stu...@gmail.com 
 wrote:

 Nobody has actually proposed a better one, outside of package renaming
 -- and that example featured an author who could just as easily have
 used an obsoleted-by field.

 How about pexpect and pextpect-u as a better example?

Perhaps you could explain?  I'm not familiar with those projects.

 Note that although well-managed Linux distros attempt to control random
 forking internally, the distro package managers don't prevent people from
 installing from third parties.  So Ubuntu PPAs, upstreams that provide their
 own rpms/debs, and major third party repos (for instance, rpmfusion as
 an add-on repo to Fedora) all have and sometimes (mis)use the ability to
 Obsolete packages in the base repository.

But in each of these cases, the packages are being defined *with
reference to* some underlying vision of what the distro (or even a
distro) is.  An Ubuntu PPA, if I understand correctly, is still
*building an Ubuntu system*.  Python packaging as a whole lacks such
frames of reference.  A forked distro is still a distro, and it's a
fork *of something*.  Rpmfusion is defining an enhanced Fedora, not
slinging random unrelated packages about.

If there's a distro analogy to PyPI, it seems to me that something
like RpmFind would be closer: it's just a free-for-all of packages,
with the user needing to decide for themselves whether installing
something from a foreign distro will or won't blow up their system.
(E.g., because their native distro and the foreign one use a different
provides taxonomy.)

RpmFind itself can't solve anybody's issues with conflicts or
obsoletes; all it can do is search the data that's there.

But unlike PyPI, RpmFind can at least tell you which vision of a
distro a particular package was intended for.  ;-)


 The ability for this class of fields to cause harm is not, to me,
 a compelling argument not to include them.

But it is absolutely not a compelling argument *to* include them, and
the actual arguments for them are pretty thin on the ground.

The real knockdown is that in the PyPI environment, there aren't any
automated use cases that don't produce collateral damage (outside of
advisories about Obsoleted-By projects).


 It could be an argument to
 explicitly tell implementers of install tools that they all have caveats
 when used with pypi and similar unpoliced community package repositories.

AFAIK, there are only a handful of curated repositories: Scipy,
Enthought, and ActiveState come to mind.  These are essentially
python distros, and they might certainly have reason to build policy
into their metadata.  I expect, however, that they would not want the
*package* authors declaring their own conflicts or obsolescence, so
I'm not sure how the metadata spec will help them.  Has anyone asked
for their input or experience?  It seems pointless to speculate on
what they might or might not need for curated distribution.  (I'm
pretty sure Enthought has their own install tools, not sure about the
other two.)

 The install tools can then choose how they wish to deal with those caveats.
 Some example strategies: choose to prompt the user as to which to install,
 choose to always treat the fields as human-informational only, mark some
 repositories as being trusted to contain packages where these fields are
 active and other repositories where the fields are ignored.

A peculiar phenomenon: every defense of these fields seems to refer
almost exclusively to how the problems could be fixed or why the
problems aren't that bad, rather than *how useful the fields would be*
in real-world scenarios.  In some cases, the argument for the fields'
safety actually runs *counter* to their usefulness, e.g., the fields
aren't that bad because we could make them have a limited function or
no function at all.  Isn't lack of usefulness generally considered an
argument for *not* including a feature?  ;-)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Keyword meanings [was: Accept just PEP-0426]

[PJ Eby]

On Wed, Dec 5, 2012 at 2:46 AM, Donald Stufft donald.stu...@gmail.com wrote:
 There's nothing preventing an installer from, during it's attempt to
 install B, see it Obsoletes A, looking at what depends on A and
 warning the user what is going to happen and prompt it.

Unless the user wrote those things that depend on A, they aren't going
to be in a position to do anything about it.  (Contrast with a distro,
where dependencies are indirect - the other package will depend on an
abstraction provided by both A and B, rather than directly depending
on A *or* B.)

(Also note that all the user knows at this point is that the author of
B *claims* to obsolete A, not that the authority managing the
repository as a whole has decreed B to obsolete A.)


 You can automatically uninstall A from B in an automatic dependency
management system

My point is that this can only work if the obsoleting is effectively
just a rename, in which case the field should be renames, or better
still, renamed-to on the originating package.

As I've mentioned repeatedly, Obsoleted-By handles more use cases than
Obsoletes, and has at least one practical automated use case
(notifying a developer that their project is depending on something
that's obsolete).

Also, the example given as a use case in the PEP (Gorgon to Torqued)
is not just wrong, it's *actively misleading*.  Gorgon and Torqued are
transparent renames of Medusa and Twisted, which do not share a common
API and thus cannot be used as the subject of any automated processing
(in the case of Obsoletes) without doing some kind of PyPI metadata
search for every package installed every time a package is installed.


 I think Obsoletes as is an alright bit of information.

1. It cannot be used to prevent the installation of an obsolete
package without a PyPI metadata search, since you must examine every
*other* package on PyPI to find out whether some package obsoletes the
one you're trying to install.

2. Unlike RPM, where metadata is provided by a trusted third party,
Obsoletes can be specified by any random forker (no pun intended),
which makes this information a mere advertisement... and an
advertisement to the wrong audience at that, because they must have
*already* found B in order to discover that it replaces A!

3. Nobody has yet supplied a use case where Obsoletes would not be
strictly improved upon by Obsoleted-By.  (Note that the author of
package X no longer maintains it does not equal package Y is
entitled to name itself the successor and enforce this upon all users
-- this can work in RPM only because it is a third party Z who
declares Y the successor to X, and there is no such party Z in the
Python world.)


 I don't see this in this thread, could you link it again?

http://mail.python.org/pipermail/catalog-sig/2010-October/003368.html
http://mail.python.org/pipermail/catalog-sig/2010-October/003364.html

These posts also address why a Conflicts field is *also* unlikely to
be particularly useful in practice, in part for reasons that relate to
differences between RPM-land and Python-land.  (For example, RPMs can
conflict over things besides files, due to runtime and configuration
issues that are out-of-scope for a Python installer tool.)

While it's certainly desirable to not invent wheels, it's important to
understand that the Python community does not work the same way as a
Linux distribution.  We are not a single organization shipping a
fully-functional and configured machine, we are hundreds of individual
authors shipping our own stuff.  Conflict resolution and package
replacement (and even deciding what it is that things provide or
require) are primarily *human* processes, not technical ones.
Relationship and support contracts, IOW, rather than software
contracts.

That's why, in the distro world, a package manager can use simple
fields to carry out the will of the human organization that made those
support and compatibility decisions.  For Python, the situation is a
bit more complicated, which is why clear thinking is needed.  Simply
copying fields blindly from other packaging systems just isn't going
to cut it.

Now, if the will of the community is to turn PyPI into a distro-style
repository, that's fine... but even if you completely ignore the human
issues, there are still technical ones.  Generally, distro-style
repositories work by downloading the full metadata set (or at least an
index) to a user's machine.  And that's the sort of architecture you'd
need in order for these type of fields to be technically feasible
(e.g., doing an index search for Obsoletes), without grinding the PyPI
servers into dust.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Keyword meanings [was: Accept just PEP-0426]

[PJ Eby]

On Wed, Dec 5, 2012 at 5:30 PM, Daniel Holth dho...@gmail.com wrote:
 My desire is to invent the useful wheel binary package format in a
 reasonable and limited amount of time by making changes to Metadata 1.2 and
 implementing the new metadata format and wheel in distribute and pip. Help
 me out by allowing useless but un-changed fields to remain in this version
 of the PEP. I am done with the PEP and submit that it is not worse than its
 predecessor.

You could just mark those fields as deprecated and that they should
not be used to delete packages or block packages from installation.

Justification: nobody has managed to make them work in an automated
tool yet, and their use in same is controversial, so they are
downgraded to human-informational only.

Please, let's not have yet *another* metadata spec that advertises
these attractive nuisance[1] fields.  I do not want us to be having
this same conversation AGAIN the next time any metadata changes are
being considered.  We've already had it too many times already.  PEPs
are supposed to summarize these discussions for that very reason.

---
[1] For non-native speakers, an attractive nuisance is a dangerous
thing that entices unsuspecting persons to play with it;
http://en.wikipedia.org/wiki/Attractive_nuisance_doctrine has more
details.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Keyword meanings [was: Accept just PEP-0426]

[PJ Eby]

On Wed, Dec 5, 2012 at 6:07 PM, Donald Stufft donald.stu...@gmail.com wrote:
 Arguing over Obsoletes vs Renames is a massive bikeshedding argument.

And is entirely beside the point.  The substantive question is whether
it's Obsoletes or Obsoleted-By - i.e., which side is it declared on.

 So it's a bad example. Hardly an argument against it.

Nobody has actually proposed a better one, outside of package renaming
-- and that example featured an author who could just as easily have
used an obsoleted-by field.


 Will require support from PyPI but this ultimately isn't a big deal.

...and every PyPI clone.  And of course the performance issues.


 If you're installing B you've prescribed trust to that author. If you don't
 trust the author then why are you installing (and then executing) code
 they wrote.

Trusting their code is one thing; trusting whether they understood a
PEP (and its interactions with various installation tools) well enough
to not accidentally delete *somebody else's code* out of my system is
another thing altogether.

OTOH, trusting an author to tell me (in an automated fashion), hey,
you should switch to this other thing as soon as you can is a FAR
smaller amount of required trust.

Arguing that because I have to trust one thing, means I must trust
another, is a Fallacy of Gray argument.


 Very convenient to declare that one of the major use cases for
 Obsoletes over Obsoleted-By is not valid because of your own
 personal opinions.

I didn't say it was invalid, I said:

Note that the author of package X no longer maintains it does not
equal package Y is entitled to name itself the successor and enforce
this upon all users

These things are not equal.  AFAIK, well-managed Linux distros do not
allow random forkers to declare themselves the official successor to a
defunct package, so any analogy between this use case in the Python
world and the distro world is strained at *best*.


 Unless you think there are
 no cases where two packages can conflict in more than what files
 are going to be installed

The rationale for that is laid out in the posts I linked.

 then there are cases where it would be helpful

Please, present a *real-life instance* where it would have been helpful to you.

 and merely having the ability to use it when it is the best tool for the job
 isn't going to cause any great issue.

One of the posts I linked presents an instance where it would have
actually *harmed* things to specify it, and it's quite easy to see how
the same problem would arise if used for non-file-related conflicts...

And the problem present is *directly* tied to the lack of a
third-party Z who decides whether X and Y, as configured for release Q
of distro P, conflict.

This is not a problem that is solvable even in *principle* for an
automated tool in the absence of party Z, which means that any such
field's actual function is limited to a heads-up to a human user.


 This is insane. A fairly simple database query is going to grind the PyPI
 servers into dust?  You're going to need to back up this FUD or please
 refrain from spouting it.

I take it you're not familiar with PyPI's history of performance and
scaling problems over the last several years, then.  The statically
cached /simple index was developed precisely to stop *today's* class
of installation tools from killing the servers...  and then mirroring
PyPI was still required to scale.  Any proposal that calls for
encouraging tools to query a metadata field *every time* a package is
installed (or even just downloaded) almost certainly needs to be
vetted with the PyPI admin team.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Keyword meanings [was: Accept just PEP-0426]

[PJ Eby]

On Mon, Dec 3, 2012 at 2:43 PM, Daniel Holth dho...@gmail.com wrote:
 How to use Obsoletes:

 The author of B decides A is obsolete.

 A releases an empty version of itself that Requires: B

 B Obsoletes: A

 The package manager says These packages are obsolete: A. Would you like to
 remove them?

 User says OK.


Um, no.  Even if the the author of A and B are the same person, you
can't remove A if there are other things on the user's system using
it.  The above scenario does not work *at all*, ever, except in the
case where B is simply an updated version of A (i.e. identical API) --
in which case, why bother?  To change the project name?  (Then it
should be Formerly-named or something like that, not Obsoletes.)

Please, *please* see the previous Catalog-SIG discussion I linked:
this is only one of multiple metadata fields that were thoroughly
debunked in that discussion as completely useless for automated
dependency management.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Accept just PEP-0426

[PJ Eby]

On Tue, Nov 20, 2012 at 11:49 AM, Vinay Sajip vinay_sa...@yahoo.co.ukwrote:

 Also: what happens when a requirement is for setuptools (= X.Y), but the
 distribute fork hasn't kept pace, and so only supports setuptools at a
 lower
 version than X.Y? I take it we're entirely comfortable with installing
 setuptools X.Y in that case? How would you ensure the right setuptools is
 always loaded, since presumably both are on sys.path?


Egg-based tools don't have any problem with this, since they set sys.path
to include the eggs needed for the running program.  Other tools will have
to tell the user and let them work it out, e.g. by using a different
virtualenv.

I personally don't think that forks claiming to provide something is
really a good thing to encourage; ISTM that saying a package *conflicts*
with another is more accurate, e.g. distribute Conflicts-Dist setuptools.
I also think distributions should say they are obsoleted, rather than
allowing other distributions to obsolete them.

That is, centralized packaging systems rely on a central authority to
resolve issues of who provides what and obsoletes what; there's an implicit
x obsoletes y [by decree of semi-independent third-party z].

However, in Python package metadata, it's x obsoletes y [by decree of
x].  IMO, this should be reversed to, Y is obsoleted by x [by decree of
y], and installing Y will conflict with X [by decree of X], so that in
each case the scope of authority for the statement is clear.

That is, in each case (conflict or obsolescence), the project's developers
are declaring under what conditions they will not be supporting an
installation.  In the case of obsolescence, the developer is saying, this
is being phased out, you should use that other thing instead.  In the case
of forks, the developer is saying, If you install both versions,
something's gonna break.

Note that installation conflict is a more conservative claim anyway: a
conflict between forked foobar packages is permanent, in the sense that
it doesn't matter what versions of both packages you're interested in: they
both want to install a foobar/__init__.py.  (Of course, installers can and
should detect that condition automatically, but not until they download the
package first.)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Accept just PEP-0426

[PJ Eby]

On Tue, Nov 20, 2012 at 4:07 PM, Glenn Linderman v+pyt...@g.nevcal.comwrote:

  On 11/20/2012 12:46 PM, PJ Eby wrote:

  I personally don't think that forks claiming to provide something is
 really a good thing to encourage; ISTM that saying a package *conflicts*
 with another is more accurate, e.g. distribute Conflicts-Dist setuptools.
 I also think distributions should say they are obsoleted, rather than
 allowing other distributions to obsolete them.

  Obsolete distributions won't say they are obsoleted, unless they receive
 further maintenance. However, if the distribution is obsolete because the
 maintainer has lost interest, they won't receive further maintenance.


(We've been over this before, the last time this discussion came up on the
Distutils-SIG for a previous Metadata PEP a year or two back, but here
goes)

Obsoleting a package is for handling renames and support transitions.  For
example, if it actually did anything to do so, I'd mark RuleDispatch as
obsoleted-by PEAK, the Pylons folks might mark some version of that as
obsoleted-by Pyramid, etc.  To put it another way, marking a package
obsolete is part of deprecation and replacement, not an unsubstantiated
third-party claim about the maintenance status of an unrelated project.

If a package is *actually* dead, there's no real point to declaring that
something else obsoletes it, and certainly no reason to put it in metadata
form.  Otherwise, we could have Twisted claiming to obsolete GEvent and
vice-versa at the same time.  Which one should an installer believe?  It
makes no sense in a standard where the project's maintainers can say
whatever they want about somebody else's project.  The scope of authority
for automatically-consumed metadata should *only* encompass the project
that provided the metadata.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Accept just PEP-0426

[PJ Eby]

On Tue, Nov 20, 2012 at 5:01 PM, Daniel Holth dho...@gmail.com wrote:

 http://www.python.org/dev/peps/pep-0314/ says:

   The most common use of this field will be in case a package name
   changes, e.g. Gorgon 2.3 gets subsumed into Torqued Python 1.0.
   When you install Torqued Python, the Gorgon package should be
   removed.

   Example:

   Obsoletes: Gorgon


 They mean pretty much what the same words mean in RPM and do not need
 further bikeshedding.


The problem is that the above *makes no sense*.  Torqued Python and
Gorgon are veiled pseudonyms for Twisted and Medusa  and Twisted is
not actually a plug-and-play substitute for Medusa, AFAIK.

Can anybody suggest an *actual* use case for Obsoletes, and explain how
it is supposed to work in software?  The last time this discussion came up,
nobody had any use cases that stood up to the how's that actually going to
work and/or help? test.  Here's a post of mine summarizing this and
related points in the previous thread:

  http://mail.python.org/pipermail/catalog-sig/2010-October/003364.html
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Accept just PEP-0426

[PJ Eby]

On Tue, Nov 20, 2012 at 6:43 PM, Daniel Holth dho...@gmail.com wrote:

 No. We trust the packages we install, including the way they decide to use
 the metadata. A bad package could delete all our files or cause dependency
 resolution to fail. Mostly they won't.


That's sort of beside the point.  The *only* use case which Obsoletes
provides over Obsoleted-By is that it allows third parties to unilaterally
advertise their forked project as a substitute for the original, and maybe
block users from switching back to the un-forked project -- regardless of
the status of the original project or the consent of the original project's
maintainer.

This use case, however, benefits nobody besides the forkers.  There are
many other legitimate channels by which the forkers can advertise
themselves as a replacement for their parent project, and no reason for the
installing end user to be bothered with the subject, except in case of a
conflict.

For somebody obsoleting their own package, on the other hand, it's likely
well worth the effort to at least update their PyPI metadata to reflect the
change in status -- especially if this can be done through the web
interface.   It's likely they would wish to update their description as
well, to notify human beings of the change.

But here's the thing that kills Obsoletes dead in the first place as a
practical tool: unless installers use a PyPI search before installing
*every single project*, there is no way for them to realize that the
obsoleting package exists!

By contrast, if a package is Obsoleted-By, then installing that package
(or declaring a dependency on it) provides an opportunity to inform the
user of the need to make a transition.  This can't be done with an
Obsoletes field.

Conversely, if you have already installed a package that says it
Obsoletes another package, this does *not* tell you that the obsolete
package shouldn't still be installed!  A replacement project doesn't
necessarily share the same API, and may exist in a different package
namespace altogether.

In short, Obsoletes is virtually *useless* as a machine-consumed metadata
field, because there is nothing you can actually do with it in a practical
installer.

I'm against adding more fields to the metadata which do not have a
specification for how they should be used in practice; the presence of such
fields has been a problem with most of the preceding metadata specs, IMO.


 I re-named a package once just because I did not like the name. I used
Obsoletes for that. It is documentation.

Note that Obsoleted-By would also serve that use case, and have the
additional benefit of being able to notify people who install new copies of
the replaced project.

(By the way Daniel, I'm sorry I didn't comment on this PEP sooner; I'd
forgotten about the previous PEP 345 rehashing in 2010, or rather, I just
sort of assumed that the results of that discussion had been incorporated
into the newer PEP, and didn't notice the reappearance of the noise fields
until your call for approval just now.  Sorry!)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Keyword meanings [was: Accept just PEP-0426]

[PJ Eby]

On Wed, Nov 21, 2012 at 12:00 AM, Stephen J. Turnbull step...@xemacs.orgwrote:

 Daniel Holth writes:

   When I used Obsoletes, it meant I am no longer developing this other
   package that is identical to this re-named package.

 But as a user I could care less!  The authors may care, but I don't
 care if Torqued obsoletes Gorgon, because in using Torqued I'm
 DTRT'ing even though I don't know it.

 What I care about is when I'm using Gorgon, and there's something
 better (or worse, correct) to use in my application.


Hence my suggestion for an Obsoleted-By field, in which Gorgon would be
able to suggest alternatives.



 It might be a good idea to have a just-like-Amazon

 While-This-Package-Is-Great-You-Might-Also-Consider:

 field.


Yeah, that's basically what Obsoleted-By is for.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Accept just PEP-0426

[PJ Eby]

On Mon, Nov 19, 2012 at 6:53 PM, Daniel Holth dho...@gmail.com wrote:

 I think this PEP is a significant improvement from its predecessor. It
 represents features like extras (provides-extra) and build requirements
 (setup-requires-dist) that are in use in the Python community but cannot be
 represented in older versions of the format, it finally specifies a UTF-8
 encoding, removes RFC 822, provides an extension mechanism, and allows the
 description to be placed in the document payload.


Can we maybe kill Provides-Dist and its associated baggage first, though?
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] AST optimizer implemented in Python

[PJ Eby]

On Sat, Aug 11, 2012 at 8:03 PM, Brett Cannon br...@python.org wrote:

 It would also be very easy to expand importlib.abc.SourceLoader to add a
 method which is called with source and returns the bytecode to be written
 out which people could override with AST optimizations before sending the
 bytecode back. That way we don't have to get into the whole business of AST
 transformations if we don't want to (although, as Victor pointed out, there
 are some people who do want this formally supported).

I'm not sure if this is directly related or not, but making this
mechanism support custom compilation for new filename suffixes would
be nice, especially for various e.g. HTML/XML templating systems that
compile to Python or bytecode.

Specifically, having a way to add a new source suffix (e.g. .kid,
.zpt, etc.) and a matching compilation function, such that it's
automatically picked up for compilation by both the filesystem and zip
importers would be awesome.  It'd also allow for DSLs and syntax
experiments using alternative filename extensions.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Requesting pronouncement on PEP 0424

[PJ Eby]

On Mon, Jul 30, 2012 at 12:51 PM, Guido van Rossum gu...@python.org wrote:
 - Most importantly: calling len(obj) and catching TypeError can only
 be a substitute for the real implementation, which IMO ought to check
 for the presence of a tp_len slot. Alas, checking hasattr(obj,
 '__len__') doesn't quite cut it either, since this returns true for a
 class object that defines a __len__ method for its instances (the
 class itself doesn't have a length).

This isn't the only place this pattern comes up; maybe a hasmethod()
function somewhere (builtin, operator, inspect?) for this would be a
good idea.  (i.e., something that returns true only if the method is
for the instance.)

(But perhaps that's a python-ideas topic, since it raises the question
of whether it should really be something more like instancehasattr(),
or whether it should be limited to special slots or something else.)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 423 : naming conventions and recipes related to packaging

[PJ Eby]

On Wed, Jun 27, 2012 at 10:57 AM, Paul Moore p.f.mo...@gmail.com wrote:

 For complex stuff, subpackages
 (import X.Y) might be needed, but that's rare (and even then, key
 names should be exposed directly from X).

 Paul.

 PS Having said all this, I don't maintain any code on PyPI - I'm a
 user not a producer. That may affect my perspective...


That, and if you don't work with web stuff or networking stuff.  Things
having lots of subpackages are quite the rule there.

Also, functional naming for top-level modules is actually an anti-pattern:
an invitation to naming conflicts, especially with future stdlib contents.
Suppose two people want to write an email package?  Unless you jam the
ownership into the name (e.g. joes_email and bobs_email), what are you
supposed to do?

This is why we have popular packages with names like nose and celery and
django and pyramid and lamson: because unique memorable names 
functionally descriptive names.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Empty directory is a namespace?

[PJ Eby]

On Sun, Jun 24, 2012 at 3:51 AM, Martin v. Löwis mar...@v.loewis.dewrote:

 On 23.06.2012 17:58, Antoine Pitrou wrote:
  On Sat, 23 Jun 2012 17:55:24 +0200
  mar...@v.loewis.de wrote:
  That's true. I would have hoped for it to be recognized only when
  there's at least one module or package inside, but it doesn't sound
  easy to check for (especially in the recursive namespace packages case
  - is that possible?).
 
  Yes - a directory becomes a namespace package by not having an
 __init__.py,
  so the namespace package case will likely become the default, and
 people
  will start removing the empty __init__.pys when they don't need to
 support
  3.2- anymore.
 
  Have you tested the performance of namespace packages compared to
  normal packages?

 No, I haven't.


It's probably not worthwhile; any performance cost increase due to looking
at more sys.path entries should be offset by the speedup of any subsequent
imports from later sys.path entries.

Or, to put it another way, almost all the extra I/O cost of namespace
packages is paid only once, for the *first* namespace package imported.  In
effect, this means that the amortized cost of using namespace packages
actually *decreases* as namespace packages become more popular.  Also, the
total extra overhead equals the cost of a listdir() for each directory on
sys.path that would otherwise not have been checked for an import.  (So,
for example, if even one import fails over the life of a program's
execution, or it performs even one import from the last directory on
sys.path, then there is no actual extra overhead.)

Of course, there are still cache validation stat() calls, and they make the
cost of an initial import of a namespace package (vs. a self-contained
package with __init__.py) to be an extra N stat() calls, where N is the
number of sys.path entries that appear *after* the sys.path directory where
the package is found.  (This cost of course must still be compared against
the costs of finding, opening, and running an empty __init__.py[co] file,
so it may actually still be quite competitive in many cases.)

For imports *within* a namespace package, similar considerations apply,
except that N is smaller, and in the simple case of replacing a
self-contained package with a namespace (but not adding any additional path
locations), N will be zero, making imports from inside the namespace run
exactly as quickly as normal imports.

In short, it's not worth worrying about, and definitely nothing that should
cause people to spread an idea that __init__.py somehow speeds things up.
If there's a difference, it'll likely be lost in measurement noise, due to
importlib's new directory caching mechanism.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Empty directory is a namespace?

[PJ Eby]

On Sun, Jun 24, 2012 at 3:27 PM, Martin v. Löwis mar...@v.loewis.dewrote:

  In short, it's not worth worrying about, and definitely nothing that
  should cause people to spread an idea that __init__.py somehow speeds
  things up.
 
  The best way to avoid people spreading that idea would be to show hard
  measurements.

 PJE wants people to spread an idea, not to avoid them doing so.

 In any case, hard measurements might help to spread the idea, here are
 mine. For the attached project, ec656d79b8ac gives, on my system

 import time for a namespace package: 113盜 (fastest run, hot caches)
 import time for a regular package:   128盜 (  --)
 first-time import of regular package: 1859盜 (due to pyc generation)
 (remove __init__.py and __pycache__ to construct the first setup)

 So namespace packages are indeed faster than regular packages, at least
 in some cases.


I don't really want to spread the idea that they're faster, either: the
exact same benchmark can probably be made to turn out differently if you
have, say, a hundred unzipped eggs on sys.path after the benchmark
directory.  A more realistic benchmark would import more than one module,
though...  and then it goes back and forth, dueling benchmarks that can
always be argued against with a different benchmark measuring different
things with other setup conditions.

That's what I meant by lost in the noise: the outcome of the benchmark
depends on which of many potentially-plausible setups and applications you
choose to use as your basis for measurement, so it's silly to think that
either omitting or including __init__.py should be done for performance
reasons.  Do whatever your application needs, because it's not going to
make much difference either way in any realistic program.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of packaging in 3.3

[PJ Eby]

On Fri, Jun 22, 2012 at 5:22 AM, Dag Sverre Seljebotn 
d.s.seljeb...@astro.uio.no wrote:

 On 06/22/2012 10:40 AM, Paul Moore wrote:

 On 22 June 2012 06:05, Nick Coghlanncogh...@gmail.com  wrote:

 distutils really only plays at the SRPM level - there is no defined OS
 neutral RPM equivalent. That's why I brought up the bdist_simple
 discussion earlier in the thread - if we can agree on a standard
 bdist_simple format, then we can more cleanly decouple the build
 step from the install step.


 That was essentially the key insight I was trying to communicate in my
 think about the end users comment. Thanks, Nick!


 The subtlety here is that there's no way to know before building the
 package what files should be installed. (For simple extensions, and perhaps
 documentation, you could get away with ad-hoc rules or special support for
 Sphinx and what-not, but there's no general solution that works in all
 cases.)

 What Bento does is have one metadata file for the source-package, and
 another metadata file (manifest) for the built-package. The latter is
 normally generated by the build process (but follows a standard
 nevertheless). Then that manifest is used for installation (through several
 available methods).


This is the right thing to do, IMO.

Also, I think rather than bikeshedding the One Serialization To Rule Them
All, it should only be the *built* manifest that is standardized for tool
consumption, and leave source descriptions to end-user tools.  setup.cfg,
bento.info, or whatever...  that part should NOT be the first thing
designed, and should not be the part that's frozen in a spec, since it
otherwise locks out the ability to enhance that format.

There's also been a huge amount of previous discussion regarding setup.cfg,
which anyone proposing to alter it should probably read.  setup.cfg allows
hooks to external systems, so IIUC,  you should be able to write a
setup.cfg file that contains little besides your publication metadata
(name, version, dependencies) and a hook to invoke whatever build tools you
want, as long as you're willing to write a Python hook.

This means that bikeshedding the build process is totally beside the
point.  If people want to use distutils, bento, SCons, ...  it really
doesn't matter, as long as they're willing to write a hook.  This is a
killer app for packaging, as it frees up the stdlib from having to do
every bloody thing itself and create One Build Process To Rule Them All.

I didn't invent setup.cfg or write the packaging code, but I support this
design approach wholeheartedly.  I have only the smallest of quibbles and
questions with it, but they aren't showstoppers.  I've already had some
discussion about these points on Distutils-SIG, and I think that should be
continued.

If there *is* to be any major discussion about switching directions in
packaging, the place to start should be *use cases* rather than file
formats.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of packaging in 3.3

[PJ Eby]

On Wed, Jun 20, 2012 at 11:57 PM, Nick Coghlan ncogh...@gmail.com wrote:

 Right - clearly enumerating the features that draw people to use
 setuptools over just using distutils should be a key element in any
 PEP for 3.4

 I honestly think a big part of why packaging ended up being incomplete
 for 3.3 is that we still don't have a clearly documented answer to two
 critical questions:
 1. Why do people choose setuptools over distutils?

Some of the reasons:

* Dependencies
* Namespace packages
* Less boilerplate in setup.py (revision control, data files support,
find_packages(), etc.)
* Entry points system for creating extensible applications and frameworks
that need runtime plugin discovery
* Command-line script wrappers
* Binary plugin installation system for apps (i.e. dump eggs in a directory
and let pkg_resources figure out what to put on sys.path)
* Test command
* Easy distribution of (and runtime access to) static data resources

Of these, automatic dependency resolution with as close to 100% backward
compatibility for installing other projects on PyPI was almost certainly
the #1 factor driving setuptools' initial adoption.  The 20% that drives
the 80%, as it were.  The rest are the 80% that brings in the remaining 20%.


 2. What's wrong with setuptools that meant the idea of including it
 directly in the stdlib was ultimately dropped and eventually replaced
 with the goal of incorporating distutils2?

Based on the feedback from Python-Dev, I withdrew setuptools from 2.5
because of what I considered valid concerns raised regarding:

1. Lack of available persons besides myself familiar with the code base and
design
2. Lack of design documents to remedy #1
3. Lack of unified end-user documentation

And there was no time for me to fix all of that before 2.5 came out,
although I did throw together the EggFormats documentation.  After that,
the time window where I was being paid (by OSAF) for setuptools
improvements came to an end, and other projects started taking precedence.

Since then, setuptools *itself* has become stable legacy code in much the
same way that the distutils has: pip, buildout, and virtualenv all built on
top of it, as it built on top of the distutils.  Problem #3 remains, but at
least now there are other people working on the codebase.

   If the end goal is the bulk of the setuptools feature set
 without the problematic features and default behaviours that make
 system administrators break out the torches and pitchforks, then we
 should *write that down* (and spell out the implications) rather than
 assuming that everyone knows the purpose of the exercise.

That's why I brought this up.  ISTM that far too much of the knowledge of
what those use cases and implications are, has been either buried in my
head or spread out among diverse user communities in the past.

Luckily, a lot of people from those communities are now getting
considerably more involved in this effort.  At the time of, say, the 2.5
setuptools question, there wasn't anybody around but me who was able to
argue the why eggs are good and useful side of the discussion, for
example.

(If you look back to the early days of setuptools, I often asked on
distutils-sig for people who could help assemble specs for various
things...  which I ended up just deciding for myself, because nobody was
there to comment on them.  It took *years* of setuptools actually being in
the field and used before enough people knew enough to *want* to take part
in the design discussions.  The versioning and metadata PEPs were things I
asked about many years prior, but nobody knew what they wanted yet, or even
knew yet why they should care.)

Similarly, in the years since then, MvL -- who originally argued against
all things setuptools at 2.5 time -- actually proposed the original
namespace package PEP.

So I don't think it's unfair to say that, seven years ago, the ideas in
setuptools were still a few years ahead of their time.  Today, console
script generation, virtual environments, namespace packages, entry point
discovery, setup.py-driven testing tools, static file inclusion, etc. are
closer to of course we should have that/everybody uses that features,
rather than esoteric oddities.

That being said, setuptools *itself* is not such a good thing.  It was
originally a *private* add-on to distutils (like numpy's distutils
extensions) and a prototyping sandbox for additions to the distutils.
(E.g. setuptools features were added to distutils in 2.4 and 2.5.)  I
honestly didn't think at the time that I was writing those features (or
even the egg stuff), that the *long term* goal would be for those things to
be maintained in a separate package.

Instead, I (rather optimistically) assumed that the value of the approaches
would be self-evident, and copied the way the other setuptools features
were.  (To this day, there are an odd variety of other little experimental
future distutils enhancements still living in the setuptools code base,
like support 

Re: [Python-Dev] Status of packaging in 3.3

[PJ Eby]

On Jun 21, 2012 11:02 AM, Zooko Wilcox-Oapos;Hearn zo...@zooko.com
wrote:

 Philip J. Eby provisionally approved of one of the patches, except for
 some specific requirement that I didn't really understand how to fix
 and that now I don't exactly remember:

 http://mail.python.org/pipermail/distutils-sig/2009-January/010880.html


I don't remember either; I just reviewed the patch and discussion, and I'm
not finding what the holdup was, exactly.  Looking at it now, it looks to
me like a good idea...  oh wait, *now* I remember the problem, or at least,
what needs reviewing.

Basically, the challenge is that it doesn't allow an .egg in a PYTHONPATH
directory to take precedence over that *specific* PYTHONPATH directory.

With the perspective of hindsight, this was purely a transitional concern,
since it only *really* mattered for site-packages; anyplace else you could
just delete the legacy package if it was a problem.  (And your patch works
fine for that case.)

However, for setuptools as it was when you proposed this, it was a
potential backwards-compatibility problem.  My best guess is that I was
considering the approach for 0.7...  which never got any serious
development time.

(It may be too late to fix the issue, in more than one sense.  Even if the
problem ceased to be a problem today, nobody's going to re-evaluate their
position on setuptools, especially if their position wasn't even based on a
personal experience with the issue.)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of packaging in 3.3

[PJ Eby]

On Jun 21, 2012 10:12 AM, Chris McDonough chr...@plope.com wrote:
 - Install package resources, which are non-Python source files that
  happen to live in package directories.

I love this phrasing, by the way (non-Python source files).

A pet peeve of mine is the insistence by some people that such files are
data and don't belong in package directories, despite the fact that if
you gave them a .py extension and added data=... around them, they'd
be considered part of the code.  A file's name and internal format aren't
what distinguishes code from data; it's the way it's *used* that matters.

I think packaging has swung the wrong way on this particular point, and
that resources and data files should be distinguished in setup.cfg, with
sysadmins *not* being given the option to muck about with resources --
especially not to install them in locations where they might be mistaken
for something editable.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of packaging in 3.3

[PJ Eby]

On Thu, Jun 21, 2012 at 11:50 AM, Chris McDonough chr...@plope.com wrote:

 On 06/21/2012 11:37 AM, PJ Eby wrote:


 On Jun 21, 2012 11:02 AM, Zooko Wilcox-Oapos;Hearn zo...@zooko.com
 mailto:zo...@zooko.com wrote:
  
   Philip J. Eby provisionally approved of one of the patches, except for
   some specific requirement that I didn't really understand how to fix
   and that now I don't exactly remember:
  
   http://mail.python.org/**pipermail/distutils-sig/2009-**
 January/010880.htmlhttp://mail.python.org/pipermail/distutils-sig/2009-January/010880.html
  

 I don't remember either; I just reviewed the patch and discussion, and
 I'm not finding what the holdup was, exactly.  Looking at it now, it
 looks to me like a good idea...  oh wait, *now* I remember the problem,
 or at least, what needs reviewing.

 Basically, the challenge is that it doesn't allow an .egg in a
 PYTHONPATH directory to take precedence over that *specific* PYTHONPATH
 directory.

 With the perspective of hindsight, this was purely a transitional
 concern, since it only *really* mattered for site-packages; anyplace
 else you could just delete the legacy package if it was a problem.  (And
 your patch works fine for that case.)

 However, for setuptools as it was when you proposed this, it was a
 potential backwards-compatibility problem.  My best guess is that I was
 considering the approach for 0.7...  which never got any serious
 development time.

 (It may be too late to fix the issue, in more than one sense.  Even if
 the problem ceased to be a problem today, nobody's going to re-evaluate
 their position on setuptools, especially if their position wasn't even
 based on a personal experience with the issue.)


 A minor backwards incompat here to fix that issue would be appropriate, if
 only to be able to say hey, that issue no longer exists to folks who
 condemn the entire ecosystem based on that bug.  At least, that is, if
 there will be another release of setuptools.  Is that likely?


Yes. At the very least, there will be updated development snapshots (which
are what buildout uses anyway).

(Official releases are in a bit of a weird holding pattern.  distribute's
versioning scheme leads to potential confusion: if I release e.g. 0.6.1,
then it sounds like it's a lesser version than whatever distribute is up to
now.  OTOH, releasing a later version number than distribute implies that
I'm supporting their feature enhancements, and I really don't want to add
new features to 0.6...  but don't have time right now to clean up all the
stuff I started in the 0.7 line either, since I've been *hoping* that the
work on packaging would make 0.7 unnecessary.  And let's not even get
started on the part where system-installed copies of distribute can prevent
people from downloading or installing setuptools in the first place.)

Anyway, changing this in a snapshot release shouldn't be a big concern; the
main user of snapshots is buildout, and buildout doesn't use .pth files
anyway, it just writes scripts that do sys.path manipulation.  (A better
approach, for everything except having stuff importable from the standard
interpreter.)

Of course, the flip side is that it means there won't be many people
testing the fix.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of packaging in 3.3

[PJ Eby]

On Thu, Jun 21, 2012 at 1:20 PM, Tarek Ziadé ta...@ziade.org wrote:

 telling us no one that is willing to maintain setuptools is able to do so.
 (according to him)


Perhaps there is some confusion or language barrier here: what I said at
that time was that the only people who I already *knew* to be capable of
taking on full responsibility for *continued development* of setuptools,
were not available/interested in the job, to my knowledge.

Specifically, the main people I had in mind were Ian Bicking and/or Jim
Fulton, both of whom had developed extensions to or significant chunks of
setuptools' functionality themselves, during which they demonstrated
exemplary levels of understanding both of the code base and the wide
variety of scenarios in which that code base had to operate.  They also
both demonstrated conservative, user-oriented design choices, that made me
feel comfortable that they would not do anything to disrupt the existing
user base, and that if they made any compatibility-breaking changes, they
would do so in a way that avoided disruption.  (I believe I also gave
Philip Jenvey as an example of someone who, while not yet proven at that
level, was someone I considered a good potential candidate as well.)

This was not a commentary on anyone *else's* ability, only on my
then-present *knowledge* of clearly-suitable persons and their
availability, or lack thereof.

I would guess that the pool of qualified persons is even larger now, but
the point is moot: my issue was never about who would maintain
setuptools, but who would *develop* it.

And I expect that we would at this point agree that future *development* of
setuptools is not something either of us are seeking. Rather, we should be
seeking to develop tools that can properly supersede it.

This is why I participated in Distutils-SIG discussion of the various
packaging PEPs, and hope to see more of them there.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] import too slow on NFS based systems

[PJ Eby]

On Thu, Jun 21, 2012 at 10:08 AM, Daniel Braniss da...@cs.huji.ac.ilwrote:

  On Thu, 21 Jun 2012 13:17:01 +0300
  Daniel Braniss da...@cs.huji.ac.il wrote:
   Hi,
   when lib/python/site-packages/ is accessed via NFS, open/stat/access
 is very
   expensive/slow.
  
   A simple solution is to use an in memory directory search/hash, so I
 was
   wondering if this has been concidered in the past, if not, and I come
   with a working solution for Unix (at least Linux/Freebsd) will it be
   concidered.
 
  There is such a thing in Python 3.3, although some stat() calls are
  still necessary to know whether the directory caches are fresh.
  Can you give it a try and provide some feedback?

 WOW!
 with a sample python program:

 in 2.7 there are:
stats   open
27369037
 in 3.3
288 57

 now I have to fix my 2.7 to work with 3.3 :-)

 any chance that this can be backported to 2.7?


As Antoine says, not in the official release.  You can, however, speed
things up substantially in 2.x by zipping the standard library and placing
it in the location given in the default sys.path, e.g.:

# python2.7
Python 2.7 (r27:82500, May  5 2011, 11:50:25)
Type help, copyright, credits or license for more information.
 import sys
 [p for p in sys.path if p.endswith('.zip')]
['/usr/lib/python27.zip']

If you include a compiled 'sitecustomize.py' in this zipfile, you would
also be able to implement a caching importer based on the default one in
pkgutil, to take up the rest of the slack.  I've previously posted sketches
of such importers; they're not that complicated to implement.  It's just
that if you don't *also* zip up the standard library, your raw interpreter
start time won't get much benefit.

(To be clear, creating the zipfile will only speed up stdlib imports,
nothing else; you'll need to implement a caching importer to get any
benefit for site-packages imports.)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of packaging in 3.3

[PJ Eby]

On Thu, Jun 21, 2012 at 4:01 PM, Paul Moore p.f.mo...@gmail.com wrote:

 End users should not need packaging tools on their machines.


Well, unless they're developers.  ;-)  Sometimes, the end user is a
developer making use of a library.


Development tools like distutils2, distribute/setuptools, bento would
 *only* be needed on developer machines, and would be purely developer
 choice. They would all interact with end users via the
 stdlib-supported standard formats. They could live outside the stdlib,
 and developers could use whichever tool suited them.


AFAIK, this was the goal behind setup.cfg in packaging, and it's a goal I
agree with.



 This is a radical idea in that it does not cater for the zipped up
 development directory as a distribution format mental model that
 current Python uses. That model could still work, but only if all the
 tools generated a stdlib-supported build definition


Again, packaging's setup.cfg is, or should be, this.  I think there are
some technical challenges with the current state of setup.cfg, but AFAIK
they aren't anything insurmountable.

(Background: the general idea is that setup.cfg contains hooks, which
name Python callables to be invoked at various stages of the process.
These hooks can dynamically add to the setup.cfg data, e.g. to list
newly-built files, binaries, etc., as well as to do any actual building.)


PS I know that setuptools includes some end-user aspects -
 multi-versioning, entry points and optional dependencies, for example.
 Maybe these are needed - personally, I have never had a need for any
 of these, so I'm not the best person to comment.


Entry points are a developer tool, and cross-project co-ordination
facility.  They allow packages to advertise classes, modules, functions,
etc. that other projects may wish to import and use in a programmatic way.
For example, a web framework may say, if you want to provide a page
template file format, register an entry point under this naming convention,
and we will automatically use it when a template has a matching file
extension.  So entry points are not really consumed by end users;
libraries and frameworks use them as ways to dynamically co-ordinate with
other installed libraries, plugins, etc.

Optional dependencies (extras), OTOH, are for end-user convenience: they
allow an author to suggest configurations that might be of interest.
Without them, people have to do things like this:

  http://pypi.python.org/pypi/celery-with-couchdb

in order to advertise what else should be installed.  If Celery were
instead to list its couchdb and SQLAlchemy requirements as extras in
setup.py, then one could easy_install celery[couchdb] or easy_install
celery[sqla] instead of needing to register separate project names on PyPI
for each of these scenarios.

As it happens, however, two of the most popular setuptools add-ons (pip and
buildout) either did not or still do not support extras, because they
were not frequently used.  Unfortunately, this meant that projects had to
do things like setup dummy projects on PyPI, because the popular tools
didn't support the scenario.

In short, nobody's likely to mourn the passing of extras to any great
degree.  They're a nice idea, but hard to bootstrap into use due to the
chicken-and-egg problem.  If you don't know what they're for, you won't use
them, and without common naming conventions (like mypackage[c_speedups] or
mypackage[test_support]), nobody will get used to asking for them.  I think
at some point we will end up reinventing them, but essentially the
challenge is that they are a generalized solution to a variety of small
problems that are not individually very motivating to anybody.  They were
only motivating to me in the aggregate because I saw lots of individual
people being bothered by their particular variation on the theme of
auxiliary dependencies or recommended options.

As for multi-versioning, it's pretty clearly a dead duck, a
proof-of-concept that was very quickly obsoleted by buildout and
virtualenv.  Buildout is a better implementation of multi-versioning for
actual scripts, and virtualenvs work fine for people who haven't yet
discovered the joys of buildout.  (I'm a recent buildout convert, in case
you can't tell.  ;-) )
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of packaging in 3.3

[PJ Eby]

On Wed, Jun 20, 2012 at 9:02 AM, Nick Coghlan ncogh...@gmail.com wrote:

 On Wed, Jun 20, 2012 at 9:46 PM, Antoine Pitrou solip...@pitrou.net
 wrote:
  Agreed, especially if the proven in the wild criterion is required
  (people won't rush to another third-party distutils replacement, IMHO).

 The existence of setuptools means that proven in the wild is never
 going to fly - a whole lot of people use setuptools and easy_install
 happily, because they just don't care about the downsides it has in
 terms of loss of control of a system configuration.


Um, this may be a smidge off topic, but what loss of control are we
talking about here?  AFAIK, there isn't anything it does that you can't
override with command line options or the config file.  (In most cases,
standard distutils options or config files.)  Do you just mean that most
people use the defaults and don't care about there being other options?
And if that's the case, which other options are you referring to?

If the long-term goal is to draw setuptools users over to packaging, then
AFAIK the packaging effort is still missing a few things, like build-time
dependencies and alternatives to setuptools' entry points and extras, as
well as the ability to integrate version control for building sdists
(without requiring the sdist's recipient to *also* have the version control
integration in order to build the package or recreate a new sdist).

These are just the missing features that I know of, from recent
distutils-sig discussions; I don't know how complete a list this is.  While
no single one of these features is directly used by every project or even a
majority of such projects, there is a correlation between size of a project
and the likelihood that they are depending on one or more of these
features.  i.e., the bigger and more widely-used the project, the more
likely it is to either use one of these features, or depend on a project
that does.

Some of these features could be built on top of packaging, in more or less
the same way setuptools is built on top of distutils.  But whether they're
done inside or outside of the packaging library, somebody's going to have
to do them, for people to be able to migrate off of setuptools.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 362: 4th edition

[PJ Eby]

On Fri, Jun 15, 2012 at 5:03 PM, R. David Murray rdmur...@bitdance.comwrote:

 On Fri, 15 Jun 2012 22:48:42 +0200, Victor Stinner 
 victor.stin...@gmail.com wrote:
   1. Should we keep 'Parameter.implemented' or not.  *Please vote*

 -1 to implemented.

  I still disagree with the deepcopy. I read somewhere that Python
  developers are consenting adult. If someone really want to modify a
  Signature, it would be nice to provide a simple method to copy it. But
  I don't see why it should be copied *by default*. I expect that
  modifying a signature is more rare than just reading a signature.

 The issue isn't consenting adults, the issue is consistency.
 Without the deepcopy, sometimes what you get back from the
 inspect function is freely modifiable and sometimes it is not.
 That inconsistency is a bad thing.


Then just copy the signature itself; as currently written, this is going to
copy the annotation objects, which could produce weird side-effects from
introspection.  Using deepcopy seems like overkill when all that's needed
is a new Signature instance with a fresh OrderedDict.

Or, better yet: make signature and parameter objects immutable (along with
the OrderedDict) and the whole problem of modification and copying goes
away altogether.  Or is there some reason not mentioned in the PEP why
mutability is necessary?  (The PEP provides no rationale at present for
making any part of a signature mutable)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] backporting stdlib 2.7.x from pypy to cpython

[PJ Eby]

On Mon, Jun 11, 2012 at 12:33 PM, Jeff Hardy jdha...@gmail.com wrote:

 On Mon, Jun 11, 2012 at 8:28 AM, Eric Snow ericsnowcurren...@gmail.com
 wrote:
  Nick's option 2 would be an improvement, but I imagine that option 3
  would have been the most effective by far.  Of course, the key thing
  is how closely the various implementors would follow the new list.
  Only they could say, though Frank Wierzbicki seemed positive about it.

 This has come up a couple of times recently (discussions on PEP 421
 and PEP 405), so I think it would be worth while. I don't have the
 time to track all of the different proposals that are in flux; it
 would be nice to know when they're done and just need a sanity check
 to make sure everything will work for other implementations.


Yes, perhaps if the list were *just* a place to cc: in or send a heads-up
to python-dev discussions, and not to have actual list discussions per se,
that would do the trick.

IOW, the idea is, If you're a contributor to a non-CPython implementation,
subscribe here to get a heads-up on Python-Dev discussions you should be
following.  Not, here's a list to discuss Python implementations in
general, and definitely not a place to *actually conduct discussions* at
all: the only things ever posted there should be cc:'d from or to
Python-Dev, or be pointers to Python-Dev threads.

That way, we'd have a solution for the periodic, hmm, we should get other
implementations to weigh in on this thread problem, that wouldn't actually
divide the discussion.  Instead, we'd have a Bat Signal (Snake Signal?)
to bring the other heroes in to meet with Commissioner Guido.  ;-)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com