Re: [Python-Dev] Make extension module initialisation more like Python module initialisation

2013-08-06 Thread Nick Coghlan 
On 6 August 2013 16:03, Stefan Behnel  wrote:
> Seriously, wouldn't python-dev be just fine for this? It's not like the
> import system is going to be rewritten for each minor release from now on.

We currently use it whenever we're doing a deep dive into import
system arcana, so python-dev only needs to worry about the question
once it's a clearly viable proposal. I think the other thread will be
quite relevant to the topic you're interested in, since we hadn't even
considered extension modules yet.

Cheers,
Nick.

-- 
Nick Coghlan   |   [email protected]   |   Brisbane, Australia
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] The Return Of Argument Clinic

2013-08-06 Thread R. David Murray 
On Mon, 05 Aug 2013 16:53:39 -0700, Larry Hastings  wrote:
> Let me put it this way: Which is more surprising to the person 
> unfamiliar with the code?  That this __init__ doesn't get all the 
> parameters, and the base class __init__ is getting called 
> automatically?  Or that this funny function "custom_init" is what gets 
> called, and this class is not allowed to have a function called __init__?

Definitely the former is more surprising.  Especially since, as Nick
points out, the last part of your statement isn't true: there can
be a function called __init__, it just has to replicate the superclass
logic if it exists, which is the way Python normally works.

I use this "call a hook method from __init__" pattern in the email
package's new header parsing code, by the way, for whatever that is
worth :)

--David
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 442 clarification for type hierarchies

2013-08-06 Thread Antoine Pitrou 
Le Mon, 05 Aug 2013 22:30:29 +0200,
Stefan Behnel  a écrit :
> 
> Hmm, it's a bit unfortunate that tp_finalize() maps so directly to
> __del__(), but I think this can be fixed. In any case, each
> tp_finalize() function must only ever be called once, so if a subtype
> inherited the tp_finalize() slot from its parent, it mustn't be
> called again.

This is already dealt with by a custom bit in the GC header (cf.
_PyGC_IS_FINALIZED, IIRC).

> >> An obvious open question is how to deal with exceptions during
> >> finalisation. Any break in the execution chain would mean that a
> >> part of the type wouldn't be finalised.
> > 
> > Let's come back to pure Python:
> > 
> > class A:
> > def __del__(self):
> > 1/0
> > 
> > class B(A):
> > def __del__(self):
> > super().__del__()
> > self.cleanup_resources()
> 
> What makes you think it's a good idea to call the parent type's
> finaliser before doing the local finalisation, and not the other way
> round? What if the subtype needs access to parts of the super type
> for its cleanup?

I'm not saying it's a good idea. I'm just saying that to reason about
the C API, it is a good idea to reason about equivalent pure Python
code. Since exceptions aren't implicitly silenced in pure Python code,
they probably shouldn't in C code.

> In other words, which makes more sense (at the C level):
> 
> try:
> super().tp_finalize()
> finally:
> local_cleanup()
> 
> or
> 
> try:
> local_cleanup()
> finally:
> super().tp_finalize()
> 
> Should that order be part of the protocol or not? (well, not for
> __del__() I guess, but maybe for tp_finalize()?)

No, it is left to the user's preference. Since tp_finalize() is meant
to be equivalent to __del__(), I think it's better if the protocols
aren't subtly different (to the extent to which it is possible, of
course).

> Coming back to the __del__() vs. tp_finalize() story, if tp_finalize()
> first recursed into the super types, the top-most one of which then
> calls __del__() and returns, we'd get an execution order that runs
> Python-level __del__() methods before C-level tp_finalize()
> functions, but loose the subtype-before-supertype execution order for
> tp_finalize() functions.

Well... to get that, you'd have to subclass a pure Python class with a
C extension type. Does that ever happen?

> That might call for a three-step cleanup:
> 
> 1) run all Python __del__() methods recursively
> 2) run all tp_finalize() functions recursively
> 3) run tp_dealloc() recursively

I don't see any reason why tp_finalize should be distinct from __del__,
while e.g. __init__ and tp_init map to the exact same thing.

(you might wonder why tp_finalize isn't called tp_del, but that's
because there is already something named tp_del - something which is
obsoleted by PEP 442, incidently ;-)).

Regards

Antoine.


___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] The Return Of Argument Clinic

2013-08-06 Thread Brett Cannon 
On Tue, Aug 6, 2013 at 12:59 AM, Nick Coghlan  wrote:

> On 6 August 2013 09:53, Larry Hastings  wrote:
> > On 08/05/2013 02:55 AM, Nick Coghlan wrote:
> > On 5 August 2013 18:48, Larry Hastings  wrote:
> >
> > Question 0: How should we integrate Clinic into the build process?
> >
> > Isn't solving the bootstrapping problem the reason for checking in the
> > clinic-generated output? If there's no Python available, we build what
> > we have (without the clinic step), then we build it again *with* the
> > clinic step.
> >
> > It solves the bootstrapping problem, but that's not the only problem
> Clinic
> > presents to the development workflow.
> >
> > If you modify some Clinic DSL in a C file in the CPython tree, then run
> > "make", should the Makefile re-run Clinic over that file?  If you say
> "no",
> > then there's no problem.  If you say "yes", then we have the problem I
> > described.
>
> Ah, I think I see the problem you mean. What is defined in the
> makefile as the way to regenerate an object file from the C file. If
> it is run clinic and then run the compiler, then you will get a
> dependency loop. If it doesn't implicitly run clinic, then we risk
> checking in inconsistent clinic metadata.
>
> I think the simplest answer may be to have "make clinic" as an
> explicit command, along with a commit hook that checks for clinic
> metadata consistency. Then "make" doesn't have to change and there's
> no nasty bootstrapping problem.


Can't we just do what we already do for the generated AST code or what we
used to do for importlib's frozen code; we have the touch extension for hg
integration for this kind of issue.
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 442 clarification for type hierarchies

2013-08-06 Thread Stefan Behnel 
Antoine Pitrou, 06.08.2013 14:12:
> Le Mon, 05 Aug 2013 22:30:29 +0200,
> Stefan Behnel a écrit :
>>
>> Hmm, it's a bit unfortunate that tp_finalize() maps so directly to
>> __del__(), but I think this can be fixed. In any case, each
>> tp_finalize() function must only ever be called once, so if a subtype
>> inherited the tp_finalize() slot from its parent, it mustn't be
>> called again.
> 
> This is already dealt with by a custom bit in the GC header (cf.
> _PyGC_IS_FINALIZED, IIRC).

But that's only at an instance level. If a type in the hierarchy inherited
the slot function for tp_finalize() from its parent, then the child must
skip its parent in the call chain to prevent calling the same slot function
twice. No instance flag can help you here.


 An obvious open question is how to deal with exceptions during
 finalisation. Any break in the execution chain would mean that a
 part of the type wouldn't be finalised.
>>>
>>> Let's come back to pure Python:
>>>
>>> class A:
>>> def __del__(self):
>>> 1/0
>>>
>>> class B(A):
>>> def __del__(self):
>>> super().__del__()
>>> self.cleanup_resources()
>>
>> What makes you think it's a good idea to call the parent type's
>> finaliser before doing the local finalisation, and not the other way
>> round? What if the subtype needs access to parts of the super type
>> for its cleanup?
> 
> I'm not saying it's a good idea. I'm just saying that to reason about
> the C API, it is a good idea to reason about equivalent pure Python
> code. Since exceptions aren't implicitly silenced in pure Python code,
> they probably shouldn't in C code.
> 
>> In other words, which makes more sense (at the C level):
>>
>> try:
>> super().tp_finalize()
>> finally:
>> local_cleanup()
>>
>> or
>>
>> try:
>> local_cleanup()
>> finally:
>> super().tp_finalize()
>>
>> Should that order be part of the protocol or not? (well, not for
>> __del__() I guess, but maybe for tp_finalize()?)
> 
> No, it is left to the user's preference. Since tp_finalize() is meant
> to be equivalent to __del__(), I think it's better if the protocols
> aren't subtly different (to the extent to which it is possible, of
> course).

Ok, fine with me. If the calls are done recursively anyway, then the child
can decide when to calls into its parent.


>> Coming back to the __del__() vs. tp_finalize() story, if tp_finalize()
>> first recursed into the super types, the top-most one of which then
>> calls __del__() and returns, we'd get an execution order that runs
>> Python-level __del__() methods before C-level tp_finalize()
>> functions, but loose the subtype-before-supertype execution order for
>> tp_finalize() functions.
> 
> Well... to get that, you'd have to subclass a pure Python class with a
> C extension type.

Maybe I'm wrong here. It's the default implementation of tp_finalize() that
calls __del__, right? If a Python class with a __del__ inherits from an
extension type that implements tp_finalize(), then whose tp_finalize() will
be executed first? The one of the Python class or the one of the extension
type?

Stefan


___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 442 clarification for type hierarchies

2013-08-06 Thread Antoine Pitrou 
Le Tue, 06 Aug 2013 17:18:59 +0200,
Stefan Behnel  a écrit :

> Antoine Pitrou, 06.08.2013 14:12:
> > Le Mon, 05 Aug 2013 22:30:29 +0200,
> > Stefan Behnel a écrit :
> >>
> >> Hmm, it's a bit unfortunate that tp_finalize() maps so directly to
> >> __del__(), but I think this can be fixed. In any case, each
> >> tp_finalize() function must only ever be called once, so if a
> >> subtype inherited the tp_finalize() slot from its parent, it
> >> mustn't be called again.
> > 
> > This is already dealt with by a custom bit in the GC header (cf.
> > _PyGC_IS_FINALIZED, IIRC).
> 
> But that's only at an instance level. If a type in the hierarchy
> inherited the slot function for tp_finalize() from its parent, then
> the child must skip its parent in the call chain to prevent calling
> the same slot function twice. No instance flag can help you here.

Ah, sorry. I had misunderstood what you were talking about.
Yes, you're right, a tp_finalize implementation should avoid calling
itself recursively.
If there's some C API that can be added to ease it, I'm ok for adding
it.

> Maybe I'm wrong here. It's the default implementation of
> tp_finalize() that calls __del__, right?

Yes.

> If a Python class with a
> __del__ inherits from an extension type that implements
> tp_finalize(), then whose tp_finalize() will be executed first?

Then only the Python __del__ gets called. It should call
super().__del__() manually, to ensure the extension type's
tp_finalize gets called.

Regards

Antoine.


___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Make extension module initialisation more like Python module initialisation

2013-08-06 Thread Ryan 
Nice idea, but some of those may break 3rd party libraries like Boost. Python 
that have their own equilavent of the Python/C API. Or Even SWIG might 
experience trouble in one or two of those.

Stefan Behnel  wrote:

>Hi,
>
>let me revive and summarize this old thread.
>
>Stefan Behnel, 08.11.2012 13:47:
>> I suspect that this will be put into a proper PEP at some point, but
>I'd
>> like to bring this up for discussion first. This came out of issues
>13429
>> and 16392.
>> 
>> http://bugs.python.org/issue13429
>> 
>> http://bugs.python.org/issue16392
>> 
>> 
>> The problem
>> ===
>> 
>> Python modules and extension modules are not being set up in the same
>way.
>> For Python modules, the module is created and set up first, then the
>module
>> code is being executed. For extensions, i.e. shared libraries, the
>module
>> init function is executed straight away and does both the creation
>and
>> initialisation. This means that it knows neither the __file__ it is
>being
>> loaded from nor its package (i.e. its FQMN). This hinders relative
>imports
>> and resource loading. In Py3, it's also not being added to
>sys.modules,
>> which means that a (potentially transitive) re-import of the module
>will
>> really try to reimport it and thus run into an infinite loop when it
>> executes the module init function again. And without the FQMN, it's
>not
>> trivial to correctly add the module to sys.modules either.
>> 
>> We specifically run into this for Cython generated modules, for which
>it's
>> not uncommon that the module init code has the same level of
>complexity as
>> that of any 'regular' Python module. Also, the lack of a FQMN and
>correct
>> file path hinders the compilation of __init__.py modules, i.e.
>packages,
>> especially when relative imports are being used at module init time.
>
>The outcome of this discussion was that the extension module import
>protocol needs to change in order to provide all necessary information
>to
>the module init function.
>
>Brett Cannon proposed to move the module object creation into the
>extension
>module importer, i.e. outside of the user provided module init
>function.
>CPython would then load the extension module, create and initialise the
>module object (set __file__, __name__, etc.) and pass it into the
>module
>init function.
>
>I proposed to make the PyModuleDef struct the new entry point instead
>of
>just a generic C function, as that would give the module importer all
>necessary information about the module to create the module object. The
>only missing bit is the entry point for the new module init function.
>
>Nick Coghlan objected to the proposal of simply extending PyModuleDef
>with
>an initialiser function, as the struct is part of the stable ABI.
>
>Alternatives I see:
>
>1) Expose a struct that points to the extension module's PyModuleDef
>struct
>and the init function and expose that struct instead.
>
>2) Expose both the PyModuleDef and the init function as public symbols.
>
>3) Provide a public C function as entry point that returns both a
>PyModuleDef pointer and a module init function pointer.
>
>4) Change the m_init function pointer in PyModuleDef_base from
>func(void)
>to func(PyObject*) iff the PyModuleDef struct is exposed as a public
>symbol.
>
>5) Duplicate PyModuleDef and adapt the new one as in 4).
>
>Alternatives 1) and 2) only differ marginally by the number of public
>symbols being exposed. 3) has the advantage of supporting more advanced
>setups, e.g. heap allocation for the PyModuleDef struct. 4) is a hack
>and
>has the disadvantage that the signature of the module init function
>cannot
>be stored across reinitialisations (PyModuleDef has no "flags" or
>"state"
>field to remember it). 5) would fix that, i.e. we could add a proper
>pointer to the new module init function as well as a flags field for
>future
>extensions. A similar effect could be achieved by carefully designing
>the
>struct in 1).
>
>I think 1-3 are all reasonable ways to do this, although I don't think
>3)
>will be necessary. 5) would be a clean fix, but has the disadvantage of
>duplicating an entire struct just to change one field in it.
>
>I'm currently leaning towards 1), with a struct that points to
>PyModuleDef,
>module init function and a flags field for future extensions. I
>understand
>that this would need to become part of the stable ABI, so explicit
>extensibility is important to keep up backwards compatibility.
>
>Opinions?
>
>Stefan
>
>
>___
>Python-Dev mailing list
>[email protected]
>http://mail.python.org/mailman/listinfo/python-dev
>Unsubscribe:
>http://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com

-- 
Sent from my Android phone with K-9 Mail. Please excuse my brevity.___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail

Re: [Python-Dev] Make extension module initialisation more like Python module initialisation

2013-08-06 Thread Stefan Behnel 
Ryan, 06.08.2013 17:02:
> Nice idea, but some of those may break 3rd party libraries like Boost.
> Python that have their own equilavent of the Python/C API. Or Even SWIG
> might experience trouble in one or two of those.

Te idea is that this will be an alternative way of initialising a module
that CPython will only use if an extension module exports the corresponding
symbol. So it won't break existing code, neither source code nor binaries.

Stefan

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 442 clarification for type hierarchies

2013-08-06 Thread Stefan Behnel 
Antoine Pitrou, 06.08.2013 17:49:
> Le Tue, 06 Aug 2013 17:18:59 +0200,
> Stefan Behnel a écrit :
>> If a Python class with a
>> __del__ inherits from an extension type that implements
>> tp_finalize(), then whose tp_finalize() will be executed first?
> 
> Then only the Python __del__ gets called. It should call
> super().__del__() manually, to ensure the extension type's
> tp_finalize gets called.

Ok, but then all I have to do in order to disable C level finalisation for
a type is to inherit from it and provide an empty __del__ method.

I think that disqualifies the feature for the use in Cython. Finalisation
at the Python level is nice, but at the C level it's usually vital. I had
originally read this PEP as a way to get better guarantees than what
dealloc can provide, but your above statement makes it rather the opposite.

Stefan


___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 442 clarification for type hierarchies

2013-08-06 Thread Antoine Pitrou 
On Tue, 06 Aug 2013 18:38:51 +0200
Stefan Behnel  wrote:
> Antoine Pitrou, 06.08.2013 17:49:
> > Le Tue, 06 Aug 2013 17:18:59 +0200,
> > Stefan Behnel a écrit :
> >> If a Python class with a
> >> __del__ inherits from an extension type that implements
> >> tp_finalize(), then whose tp_finalize() will be executed first?
> > 
> > Then only the Python __del__ gets called. It should call
> > super().__del__() manually, to ensure the extension type's
> > tp_finalize gets called.
> 
> Ok, but then all I have to do in order to disable C level finalisation for
> a type is to inherit from it and provide an empty __del__ method.
> 
> I think that disqualifies the feature for the use in Cython. Finalisation
> at the Python level is nice, but at the C level it's usually vital. I had
> originally read this PEP as a way to get better guarantees than what
> dealloc can provide, but your above statement makes it rather the opposite.

Anything vital should probably be ensured by tp_dealloc.
For example, you might close an fd early in tp_finalize, but also ensure
it gets closed in tp_dealloc in the case tp_finalize wasn't called.

(that said, you can also have fd leaks in pure Python...)

Regards

Antoine.


___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] unittest.TestSuite holding references to unittest.TestCase instances too long

2013-08-06 Thread Matt McClure 
Hi Michael,

On Tue, Aug 6, 2013 at 4:25 AM, Michael Foord wrote:

> unittest itself has changed so extensively since the last release of
> unittest2 that I'm not sure whether a completely fresh start for unittest2
> might be needed. (Although I intend to do another bugfix release of this
> version as well.)
>
> Making unittest2 will involve:
>
> * Taking the Python 3 unittest and porting code plus tests to run
> on python 2
> [ ... ]
>

I took a different approach and simply applied the patch of diffs[1] from
the Python 3 issue to the unittest2 tip.

There was a small amount of renaming "unittest" to "unittest2" required,
but other than that, the patch applied pretty cleanly, and seems to pass
the unit tests and avoid the ever-increasing memory problem in my private
test suite.

Do you think it's sufficient to port just this feature? Or if not, what am
I missing that requires resyncing more of unittest2 with the changes from
Python 3?

Is Google Code[2] still the right place for unittest2 issues? I found that
via PyPI[3].

It looks like there have been a lot of commits in the unittest2 repository
since the last PyPI release (2010-07-12 -- 0.5.1). Would you plan to do
another PyPI release of unittest2 with this feature? Or would you recommend
using unittest2 from the repository to get it? Or am I missing a more
recent packaged release somewhere else?

[1]:
https://bitbucket.org/matthewlmcclure/unittest2/compare/issue11798-tip..issue11798-base#diff
[2]: https://code.google.com/p/unittest-ext/issues/detail?id=76&sort=-id
[3]: https://pypi.python.org/pypi/unittest2

-- 
Matt McClure
http://matthewlmcclure.com
http://www.mapmyfitness.com/profile/matthewlmcclure
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Our failure at handling GSoC students

2013-08-06 Thread Antoine Pitrou 

Hello,

I would like to point out that we currently fail at handling GSoC
projects and bringing them to completion.

One cruel example is the set of PEP 3121 / PEP 384 refactorings done by
Robin Schreiber:
http://bugs.python.org/issue?%40columns=id%2Cactivity%2Ctitle%2Ccreator%2Cassignee%2Cstatus%2Ctype&%40sort=-activity&%40filter=status&%40action=searchid&ignore=file%3Acontent&%40search_text=pep+3121&submit=search&status=-1%2C1%2C3

Robin has produced many patches that seem to reach the stated goal
(refactor C extension modules to take advantage of the latest PEPs
about module initialization and extension types definition).
Unfortunately, tackling both goals at the same time produces big
patches with a lot of churn; and it is also not obvious the PEP 384
refactoring is useful for the stdlib (while the PEP 3121 refactoring
definitely is).

What didn't produce an alarm during Robin's work is that GSoC work is
done in private. Therefore, other core developers than the mentor don't
get to give an advice early, as would happen with any normal proposal
done publicly (on the mailing-list or on the bug tracker). It is also
likely that the mentor gets overworked after the GSoC period is over,
is unable to finalize the patch and push it, and other core devs have a
hard time catching up on the work and don't know what the shortcomings
are.

Regards

Antoine.


___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Our failure at handling GSoC students

2013-08-06 Thread Eli Bendersky 
On Tue, Aug 6, 2013 at 12:26 PM, Antoine Pitrou  wrote:

>
> Hello,
>
> I would like to point out that we currently fail at handling GSoC
> projects and bringing them to completion.
>
> One cruel example is the set of PEP 3121 / PEP 384 refactorings done by
> Robin Schreiber:
>
> http://bugs.python.org/issue?%40columns=id%2Cactivity%2Ctitle%2Ccreator%2Cassignee%2Cstatus%2Ctype&%40sort=-activity&%40filter=status&%40action=searchid&ignore=file%3Acontent&%40search_text=pep+3121&submit=search&status=-1%2C1%2C3
>
> Robin has produced many patches that seem to reach the stated goal
> (refactor C extension modules to take advantage of the latest PEPs
> about module initialization and extension types definition).
> Unfortunately, tackling both goals at the same time produces big
> patches with a lot of churn; and it is also not obvious the PEP 384
> refactoring is useful for the stdlib (while the PEP 3121 refactoring
> definitely is).
>
> What didn't produce an alarm during Robin's work is that GSoC work is
> done in private. Therefore, other core developers than the mentor don't
> get to give an advice early, as would happen with any normal proposal
> done publicly (on the mailing-list or on the bug tracker). It is also
> likely that the mentor gets overworked after the GSoC period is over,
> is unable to finalize the patch and push it, and other core devs have a
> hard time catching up on the work and don't know what the shortcomings
> are.
>


I would like to point out something that stands out in this list of issues:
such a method of producing dozens of patches simultaneously is extremely
unwise, unless there's a crucial piece of history I'm missing. It is much
more prudent to start with one or two exemplary modules, and if those fully
pass code review, send out patches for others. The reason is obvious - code
review may turn up problems or requests for change. Going backwards to
modify 57 patches is not something anyone would want to do.

Eli
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Our failure at handling GSoC students

2013-08-06 Thread Skip Montanaro 
> It is also likely that the mentor gets overworked after the GSoC period is 
> over,
> is unable to finalize the patch and push it...

Given that Python development is done using a good DVCS now, it seems
that if each manageable chunk of changes is done on a separate branch,
the likelihood of acceptance of any one change goes way up (as it's
much easier to analyze in isolation), and the likelihood that one
small change nukes the entire collective patch goes way down.

I don't know if that will address all concerns and improve the success
rate, but I would personally find it easier to process 100 changes,
each with 37 patch chunks than one change having 3700 chunks.  Smaller
haystacks make it easier to find the needles.  In addition, there
should be less pressure for someone to analyze the entire lot.  If you
get burned out at change 12, others should be there to pick up from
change 13 without having to start over, re-analyzing changes 1 through
12.

Skip
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Our failure at handling GSoC students

2013-08-06 Thread Antoine Pitrou 
On Tue, 6 Aug 2013 12:43:40 -0700
Eli Bendersky  wrote:
> On Tue, Aug 6, 2013 at 12:26 PM, Antoine Pitrou  wrote:
> >
> > Hello,
> >
> > I would like to point out that we currently fail at handling GSoC
> > projects and bringing them to completion.
> >
> > One cruel example is the set of PEP 3121 / PEP 384 refactorings done by
> > Robin Schreiber:
> >
> > http://bugs.python.org/issue?%40columns=id%2Cactivity%2Ctitle%2Ccreator%2Cassignee%2Cstatus%2Ctype&%40sort=-activity&%40filter=status&%40action=searchid&ignore=file%3Acontent&%40search_text=pep+3121&submit=search&status=-1%2C1%2C3
> >
> > Robin has produced many patches that seem to reach the stated goal
> > (refactor C extension modules to take advantage of the latest PEPs
> > about module initialization and extension types definition).
> > Unfortunately, tackling both goals at the same time produces big
> > patches with a lot of churn; and it is also not obvious the PEP 384
> > refactoring is useful for the stdlib (while the PEP 3121 refactoring
> > definitely is).
> >
> > What didn't produce an alarm during Robin's work is that GSoC work is
> > done in private. Therefore, other core developers than the mentor don't
> > get to give an advice early, as would happen with any normal proposal
> > done publicly (on the mailing-list or on the bug tracker). It is also
> > likely that the mentor gets overworked after the GSoC period is over,
> > is unable to finalize the patch and push it, and other core devs have a
> > hard time catching up on the work and don't know what the shortcomings
> > are.
> >
> 
> I would like to point out something that stands out in this list of issues:
> such a method of producing dozens of patches simultaneously is extremely
> unwise, unless there's a crucial piece of history I'm missing. It is much
> more prudent to start with one or two exemplary modules, and if those fully
> pass code review, send out patches for others. The reason is obvious - code
> review may turn up problems or requests for change. Going backwards to
> modify 57 patches is not something anyone would want to do.

I definitely agree, but this is part of our failure too. A
beginner contributor isn't supposed to know the best way to contribute
if nobody tells him/her beforehand.

Regards

Antoine.
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Our failure at handling GSoC students

2013-08-06 Thread Fred Drake 
On Tue, Aug 6, 2013 at 3:51 PM, Antoine Pitrou  wrote:
> I definitely agree, but this is part of our failure too.

I'd say this is strictly our failure, not the students'.  This isn't
really a new problem, I don't think, though the shape of this collection
of patches makes it obvious.

I haven't been active with GSoC the last couple of years, but if we don't
have any sort of guide for mentors, we probably should, and this is an
issue that should be mentioned as one that requires discussion with the
students.  That's our role as a community and as mentors when it comes to
GSoC.


  -Fred

-- 
Fred L. Drake, Jr.
"A storm broke loose in my mind."  --Albert Einstein
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] a Constant addition to enum

2013-08-06 Thread Ethan Furman 
A question came up on stackoverflow asking about the Planet example and the need to have the constant G defined in the 
method instead of at the class level:


http://stackoverflow.com/q/17911188/208880

Since methods and descriptors are immune to enumeration my proposed solution created a Constant descriptor that could be 
used to keep class level constants at the class level.  It's not complex, only about 7 lines.  Should we have something 
like that included in the enum module?


If we do include something like that, should it be constant, or should it be more like property?  (The important 
differences from property being that class access still returns the value, not the property itself, and setting the 
class-level value changes the value but doesn't replace the property.)


--
~Ethan~
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] The Return Of Argument Clinic

2013-08-06 Thread Antoine Pitrou 
On Mon, 05 Aug 2013 16:53:39 -0700
Larry Hastings  wrote:
> 
> On 08/05/2013 02:55 AM, Nick Coghlan wrote:
> > On 5 August 2013 18:48, Larry Hastings  wrote:
> >> Question 0: How should we integrate Clinic into the build process?
> > Isn't solving the bootstrapping problem the reason for checking in the
> > clinic-generated output? If there's no Python available, we build what
> > we have (without the clinic step), then we build it again *with* the
> > clinic step.
> 
> It solves the bootstrapping problem, but that's not the only problem 
> Clinic presents to the development workflow.
> 
> If you modify some Clinic DSL in a C file in the CPython tree, then run 
> "make", should the Makefile re-run Clinic over that file?  If you say 
> "no", then there's no problem.  If you say "yes", then we have the 
> problem I described.

I say "yes" and I think best-effort is the solution. Usually, the
current clinic should be good enough to compile future C changes. If it
isn't, just revert your working copy and start again (save your
changes and re-apply them if desired).

importlib has the same theoretical problem but it works well enough in
practice, even though it could be maddening at times when the code
wasn't quite stabilized.

Regards

Antoine.


___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] a Constant addition to enum

2013-08-06 Thread Eli Bendersky 
On Tue, Aug 6, 2013 at 1:42 PM, Ethan Furman  wrote:

> A question came up on stackoverflow asking about the Planet example and
> the need to have the constant G defined in the method instead of at the
> class level:
>
> http://stackoverflow.com/q/**17911188/208880
>
> Since methods and descriptors are immune to enumeration my proposed
> solution created a Constant descriptor that could be used to keep class
> level constants at the class level.  It's not complex, only about 7 lines.
>  Should we have something like that included in the enum module?
>
> If we do include something like that, should it be constant, or should it
> be more like property?  (The important differences from property being that
> class access still returns the value, not the property itself, and setting
> the class-level value changes the value but doesn't replace the property.)
>

Personally, I dislike all non-simple uses of Enums. One such use is adding
behavior to them. This can always be split to separate behavior from the
Enum itself, and I would prefer that. We went to great lengths to ensure
that things work in expected ways, but heaping additional features (even as
separate decorators) is just aggravating thiings. So -1 from me.

Finally, I suggest we exercise restraint in adding more capabilities to
enums in 3.4; enums are a new creature for Python and it will be extremely
useful to see them used in the wild for a while first. We can enhance them
in 3.5, but premature enhancement is IMHO much more likely to do harm than
good.

Eli
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] a Constant addition to enum

2013-08-06 Thread Antoine Pitrou 
On Tue, 6 Aug 2013 14:36:27 -0700
Eli Bendersky  wrote:
> On Tue, Aug 6, 2013 at 1:42 PM, Ethan Furman  wrote:
> 
> > A question came up on stackoverflow asking about the Planet example and
> > the need to have the constant G defined in the method instead of at the
> > class level:
> >
> > http://stackoverflow.com/q/**17911188/208880
> >
> > Since methods and descriptors are immune to enumeration my proposed
> > solution created a Constant descriptor that could be used to keep class
> > level constants at the class level.  It's not complex, only about 7 lines.
> >  Should we have something like that included in the enum module?
> >
> > If we do include something like that, should it be constant, or should it
> > be more like property?  (The important differences from property being that
> > class access still returns the value, not the property itself, and setting
> > the class-level value changes the value but doesn't replace the property.)
> >
> 
> Personally, I dislike all non-simple uses of Enums. One such use is adding
> behavior to them. This can always be split to separate behavior from the
> Enum itself, and I would prefer that. We went to great lengths to ensure
> that things work in expected ways, but heaping additional features (even as
> separate decorators) is just aggravating thiings. So -1 from me.

Agreed. Also, putting constants inside Enum declarations is just
confusing.
(it doesn't help that the presented "use case" is the typical
schoolbook example that doesn't correspond to any real-world
situation, just like parsing Roman numbers and solving Hanoi tower
puzzles)

Regards

Antoine.


___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Our failure at handling GSoC students

2013-08-06 Thread Terry Reedy 

On 8/6/2013 3:26 PM, Antoine Pitrou wrote:


I would like to point out that we currently fail at handling GSoC
projects and bringing them to completion.

One cruel example is the set of PEP 3121 / PEP 384 refactorings done by
Robin Schreiber:
http://bugs.python.org/issue?%40columns=id%2Cactivity%2Ctitle%2Ccreator%2Cassignee%2Cstatus%2Ctype&%40sort=-activity&%40filter=status&%40action=searchid&ignore=file%3Acontent&%40search_text=pep+3121&submit=search&status=-1%2C1%2C3

Robin has produced many patches that seem to reach the stated goal
(refactor C extension modules to take advantage of the latest PEPs
about module initialization and extension types definition).
Unfortunately, tackling both goals at the same time produces big
patches with a lot of churn; and it is also not obvious the PEP 384
refactoring is useful for the stdlib (while the PEP 3121 refactoring
definitely is).

What didn't produce an alarm during Robin's work is that GSoC work is
done in private. Therefore, other core developers than the mentor don't
get to give an advice early, as would happen with any normal proposal
done publicly (on the mailing-list or on the bug tracker). It is also
likely that the mentor gets overworked after the GSoC period is over,
is unable to finalize the patch and push it, and other core devs have a
hard time catching up on the work and don't know what the shortcomings
are.


There are 2 GSOC students working on Idle tests (mentored by Todd 
Rovito). Each file tested is a separate issue and separate patch. I have 
fallen behind reviewing them because of unexpected issues first with 
Idle and then with buildbots, but have been able to make some comments 
and some commits. I plan to do more before they disappear, and to get to 
everything eventually.


--
Terry Jan Reedy

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Our failure at handling GSoC students

2013-08-06 Thread Todd V Rovito 
On Aug 6, 2013, at 3:26 PM, Antoine Pitrou  wrote:
> I would like to point out that we currently fail at handling GSoC
> projects and bringing them to completion.
In the past I have noticed the same thing with IDLE.  Students and mentors act 
outside of the standard Python development process then the final student 
products never get committed. 

> One cruel example is the set of PEP 3121 / PEP 384 refactorings done by
> Robin Schreiber:
> http://bugs.python.org/issue?%40columns=id%2Cactivity%2Ctitle%2Ccreator%2Cassignee%2Cstatus%2Ctype&%40sort=-activity&%40filter=status&%40action=searchid&ignore=file%3Acontent&%40search_text=pep+3121&submit=search&status=-1%2C1%2C3
I agree this is a sad example.

> What didn't produce an alarm during Robin's work is that GSoC work is
> done in private. Therefore, other core developers than the mentor don't
> get to give an advice early, as would happen with any normal proposal
> done publicly (on the mailing-list or on the bug tracker). It is also
> likely that the mentor gets overworked after the GSoC period is over,
> is unable to finalize the patch and push it, and other core devs have a
> hard time catching up on the work and don't know

So for this year I designed an IDLE project that specifically forced the 
students to be like normal contributors and use the standard Python development 
model.  See this link for the project description:
http://wiki.python.org/moin/SummerOfCode/2013/python-core

From the project description:
"Successful student proposals should not under estimate how long it takes to 
get code committed to CPython. A student must be able to concisely communicate 
and document the unit test framework's design to the Python community in order 
to get the framework committed to the CPython source tree. Do not underestimate 
how much time this communication and documentation will actually take in your 
proposal!!! Often times it will take several passes and several code reviews 
for a patch to get committed into CPython. This project is approximately 40% 
coding and 60% communication. This project requires average Python coding 
skills with excellent communication skills and a unrelenting persistence to get 
this job done to the satisfaction of at least one Python Core Developer so the 
work will be committed into the CPython source tree."

To date the students have gotten three commits completed and seven total issues 
opened.  Here is a google spreadsheet with the details:
https://docs.google.com/spreadsheet/lv?key=0AqHo248BJw3RdFRnREo5TGtrQmxvQi1oem1HUS1PNGc

It is too early to tell how effective the students have been.  I do wish more 
unit tests were created but it all takes time to convince a core Python 
developer to make the commit (and with good reason).  In this case Terry Reedy 
has been a huge help!  I think the students are having fun and hopefully will 
stay involved for years to come.   


___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] (New) PEP 446: Make newly created file descriptors non-inheritable

2013-08-06 Thread Victor Stinner 
2013/8/6 Victor Stinner :
> Oh, the summary table is wrong for the "subprocess, default" line: all
> inheritable handles are inherited if at least one standard stream is
> replaced.

I updated the PEP:

- add a new section "Performances of Closing All File Descriptors"
- mention a previous attempt in 2007 to add open_noinherit
- complete the summary table of the status of python 3.3 to mention
the "subprocess, replace stdout" case
- Windows creates non-inheritable *handles* (not fds) by default

See the history: http://hg.python.org/peps/log/tip/pep-0446.txt

Victor
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Our failure at handling GSoC students

2013-08-06 Thread Nick Coghlan 
On 7 August 2013 05:26, Antoine Pitrou  wrote:
>
> Hello,
>
> I would like to point out that we currently fail at handling GSoC
> projects and bringing them to completion.

Agreed.

> What didn't produce an alarm during Robin's work is that GSoC work is
> done in private. Therefore, other core developers than the mentor don't
> get to give an advice early, as would happen with any normal proposal
> done publicly (on the mailing-list or on the bug tracker).

This isn't the way GSoC is supposed to work. Mentors are supposed to
nudge students towards the regular channels for the project. This may
mean a sig (e.g. the import engine work a few years ago was discussed
on import-sig. That didn't end up being committed, since Greg's work
revealed some fundamental problems with the proposed architecture, but
the knowledge wasn't restricted to just myself and Greg), or else a
more general channel like core-mentorship or python-ideas.

Ideally (and this isn't going to be possible for every GSoC project),
mentors will be able to help break the project down into reviewable
chunks proposed as incremental issues, rather than producing one big
patch at the end of the summer.

> It is also
> likely that the mentor gets overworked after the GSoC period is over,
> is unable to finalize the patch and push it, and other core devs have a
> hard time catching up on the work and don't know what the shortcomings
> are.

Indeed. I added some preliminary guidelines for mentors to the GSoC
"Expectations" page:

http://wiki.python.org/moin/SummerOfCode/Expectations#guidelines-for-mentors

I also added a link to the expectations page from
http://wiki.python.org/moin/SummerOfCode/2013#prospective-mentors

Cheers,
Nick.

-- 
Nick Coghlan   |   [email protected]   |   Brisbane, Australia
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] a Constant addition to enum

2013-08-06 Thread Nick Coghlan 
On 7 August 2013 07:36, Eli Bendersky  wrote:
> On Tue, Aug 6, 2013 at 1:42 PM, Ethan Furman  wrote:
>>
>> A question came up on stackoverflow asking about the Planet example and
>> the need to have the constant G defined in the method instead of at the
>> class level:
>>
>> http://stackoverflow.com/q/17911188/208880
>>
>> Since methods and descriptors are immune to enumeration my proposed
>> solution created a Constant descriptor that could be used to keep class
>> level constants at the class level.  It's not complex, only about 7 lines.
>> Should we have something like that included in the enum module?
>>
>> If we do include something like that, should it be constant, or should it
>> be more like property?  (The important differences from property being that
>> class access still returns the value, not the property itself, and setting
>> the class-level value changes the value but doesn't replace the property.)
>
>
> Personally, I dislike all non-simple uses of Enums. One such use is adding
> behavior to them. This can always be split to separate behavior from the
> Enum itself, and I would prefer that. We went to great lengths to ensure
> that things work in expected ways, but heaping additional features (even as
> separate decorators) is just aggravating thiings. So -1 from me.
>
> Finally, I suggest we exercise restraint in adding more capabilities to
> enums in 3.4; enums are a new creature for Python and it will be extremely
> useful to see them used in the wild for a while first. We can enhance them
> in 3.5, but premature enhancement is IMHO much more likely to do harm than
> good.

Agreed. I wouldn't be averse to taking those advanced examples out of
the docs, too.

Like metaclasses, you can do crazy things with enums. "Can" doesn't
mean "should", however. We've had a lot of success with metaclasses by
soft-pedalling them in the standard library, so people only explore
them when they *really* need them. I think we'd be well advised to
pursue a similar path with advanced Enum tricks.

Cheers,
Nick.

-- 
Nick Coghlan   |   [email protected]   |   Brisbane, Australia
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Our failure at handling GSoC students

2013-08-06 Thread Stephen J. Turnbull 
Nick Coghlan writes:
 > On 7 August 2013 05:26, Antoine Pitrou  wrote:

 > > I would like to point out that we currently fail at handling GSoC
 > > projects and bringing them to completion.
 > 
 > Agreed.

I have no opinion on that statement, having not looked at the
projects.

 > > What didn't produce an alarm during Robin's work is that GSoC work is
 > > done in private.
 > 
 > This isn't the way GSoC is supposed to work.

Indeed.  I've seen this in one of the orgs I mentor for, and we may
ask that mentor to go elsewhere if we don't get a credible promise to
shape up.  (That's an indication of how seriously my orgs take "work
in public", not a suggestion for Python action in any case.)

 > or else a more general channel like core-mentorship or
 > python-ideas.

+1 for python-ideas if there is no better fit in a specialist list,
moving to python-dev as usual if the mentor judges the student
sufficiently mature.[1] If the student's posting becomes annoying on
python-ideas, the mentor should provide netiquette guidance.  IMO the
project-specific mentoring will become an annoyance on core-mentorship
since it continues for the whole summer, and changing "preliminary"
venues midstream doesn't seem like a great idea to me.

My orgs require (in only one case successfully :-) weekly progress
reports to the developers' list (per above, that would be
python-ideas, not python-dev).  In that successful case, GSoC is
essentially the only content posted to that dev list, though.  I'm not
sure if that matters.

 > Ideally (and this isn't going to be possible for every GSoC project),
 > mentors will be able to help break the project down into reviewable
 > chunks proposed as incremental issues, rather than producing one big
 > patch at the end of the summer.

GSoC suggests (at about the level of an RFC SHOULD) that students be
committing early and often to a publicly accessible branch.  I don't
see a good reason why that wouldn't work even for complex projects
that can't be merged until end of summer.  (I wonder whether such
projects should be used as GSoC tasks, as well, but as I haven't
actually looked at Python GSoC tasks, I'll leave that in parens.)


Footnotes: 
[1]  By that I mean that I often observe students mixing blue-sky
design with hard-core implementation details and necessary design
revision late in the summer.  If the project and student are "mature,"
that won't happen.

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Our failure at handling GSoC students

2013-08-06 Thread Lennart Regebro 
On Tue, Aug 6, 2013 at 9:26 PM, Antoine Pitrou  wrote:
> What didn't produce an alarm during Robin's work is that GSoC work is
> done in private.

Why is it done in private?

//Lennart
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com