Re: [Python-Dev] 2.7 Windows installers (Was: death to 2.7; long live 2.7)

[Jeff Hardy]

On Sun, Apr 13, 2014 at 9:51 PM, "Martin v. Löwis"  wrote:
> As I just said: to get started, run the current build process. Without
> knowing WiX in detail, I'd still claim that msi.py is superiour in
> terms of expressiveness (i.e. it can better compute what files go into
> the MSI). I'm almost certain that you need a full programming language
> to actually perform the entire build; one alternative to msi.py would
> be to have a Python script generating the WiX configuration files.

FWIW, that's exactly what IronPython does - most of the files are
fixed, but there are scripts to generate inclusions for the stdlib.
WiX and MSBuild are pretty powerful combo, even if the excess of
stabby brackets makes *me* stabby after a while.

- Jeff
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Appeal for reviews

[Nick Coghlan]

On 14 Apr 2014 01:56, "Stephen J. Turnbull"  wrote:
>
> mar...@v.loewis.de writes:
>
>  > For gaining commit access, it's really more important that the patch
>  > is factually finished, than that it's author believes it to. If people
>  > get it right the first time often enough, they get commit access.
>
> Yes, that's what I had in mind, but I guess I explained it poorly.

We should capture this discussion clearly in the dev guide. Even if we
switch to a core reviewer model at some point (as I propose in PEP 462),
the criteria for core reviewer status will match those for core commiter
status.

There are actually a few things I'm personally looking for:

* good judgement on when a patch is "finished enough" to merge
* good judgement on whether a change is a new feature or a bug fix
* good judgement whether a new feature is worth the additional cognitive
burden
* good ability to assess backwards compatibility risks
* sufficient humility to answer "I don't know" to the above questions when
appropriate and ask the relevant domain experts, their sponsoring mentor,
the core-mentorship list or python-dev at large for advice on what to do

It's that last one which is really most critical - even Guido asks for
additional input when he's uncertain about something, and that's a key part
of why I trust his decisions on those rare occasions when he finds it
necessary to exercise BDFL fiat (although his long history of demonstrating
excellent language design instincts certainly helps!)

Cheers,
Nick.

>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Appeal for reviews

[R. David Murray]

On Mon, 14 Apr 2014 08:18:13 -0400, Nick Coghlan  wrote:
> On 14 Apr 2014 01:56, "Stephen J. Turnbull"  wrote:
> >
> > mar...@v.loewis.de writes:
> >
> >  > For gaining commit access, it's really more important that the patch
> >  > is factually finished, than that it's author believes it to. If people
> >  > get it right the first time often enough, they get commit access.
> >
> > Yes, that's what I had in mind, but I guess I explained it poorly.
> 
> We should capture this discussion clearly in the dev guide. Even if we
> switch to a core reviewer model at some point (as I propose in PEP 462),
> the criteria for core reviewer status will match those for core commiter
> status.
> 
> There are actually a few things I'm personally looking for:
> 
> * good judgement on when a patch is "finished enough" to merge
> * good judgement on whether a change is a new feature or a bug fix
> * good judgement whether a new feature is worth the additional cognitive
> burden
> * good ability to assess backwards compatibility risks
> * sufficient humility to answer "I don't know" to the above questions when
> appropriate and ask the relevant domain experts, their sponsoring mentor,
> the core-mentorship list or python-dev at large for advice on what to do

When considering who we give commit access to, I think we would be
well served to start giving more weight to the quality of the code
reviews that someone does.  Producing good patches is important,
but even without moving the infrastructure to Nick's "core reviewer"
model, doing those reviews is an important part of what committers
do, and it is a different (although related) skill to that of
writing good patches.

Or to put it another way, I'd like to encourage contributors who
want to get commit access to focus just as much on doing good reviews as
they do on writing new patches.  Currently the focus is all on
getting patches accepted.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] [numpy wishlist] PyMem_*Calloc

[Nathaniel Smith]

Hi all,

The new tracemalloc infrastructure in python 3.4 is super-interesting
to numerical folks, because we really like memory profiling. Numerical
programs allocate a lot of memory, and sometimes it's not clear which
operations allocate memory (some numpy operations return views of the
original array without allocating anything; others return copies). So
people actually use memory tracking tools[1], even though
traditionally these have been pretty hacky (i.e., just checking RSS
before and after each line is executed), and numpy has even grown its
own little tracemalloc-like infrastructure [2], but it only works for
numpy data.

BUT, we also really like calloc(). One of the basic array creation
routines in numpy is numpy.zeros(), which returns an array full of --
you guessed it -- zeros. For pretty much all the data types numpy
supports, the value zero is represented by the bytestring consisting
of all zeros. So numpy.zeros() usually uses calloc() to allocate its
memory.

calloc() is more awesome than malloc()+memset() for two reasons.
First, calloc() for larger allocations is usually implemented using
clever VM tricks, so that it doesn't actually allocate any memory up
front, it just creates a COW mapping of the system zero page and then
does the actual allocation one page at a time as different entries are
written to. This means that in the somewhat common case where you
allocate a large array full of zeros, and then only set a few
scattered entries to non-zero values, you can end up using much much
less memory than otherwise. It's entirely possible for this to make
the difference between being able to run an analysis versus not.
memset() forces the whole amount of RAM to be committed immediately.

Secondly, even if you *are* going to touch all the memory, then
calloc() is still faster than malloc()+memset(). The reason is that
for large allocations, malloc() usually does a calloc() no matter what
-- when you get a new page from the kernel, the kernel has to make
sure you can't see random bits of other processes's memory, so it
unconditionally zeros out the page before you get to see it. calloc()
knows this, so it doesn't bother zeroing it again. malloc()+memset(),
by contrast, zeros the page twice, producing twice as much memory
traffic, which is huge.

SO, we'd like to route our allocations through PyMem_* in order to let
tracemalloc "see" them, but because there is no PyMem_*Calloc, doing
this would force us to give up on the calloc() optimizations.

The obvious solution is to add a PyMem_*Calloc to the API. Would this
be possible? Unfortunately it would require adding a new field to the
PyMemAllocator struct, which would be an ABI/API break; PyMemAllocator
is exposed directly in the C API and passed by value:
  https://docs.python.org/3.5/c-api/memory.html#c.PyMemAllocator
(Too bad we didn't notice this a few months ago before 3.4 was
released :-(.) I guess we could just rename the struct in 3.5, to
force people to update their code. (I guess there aren't too many
people who would have to update their code.)

Thoughts?

-n

[1] 
http://scikit-learn.org/stable/developers/performance.html#memory-usage-profiling
[2] https://github.com/numpy/numpy/pull/309

-- 
Nathaniel J. Smith
Postdoctoral researcher - Informatics - University of Edinburgh
http://vorpus.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Python "2migr8"

[Steve Dower]

Just in case there's anyone out there who isn't yet sick of discussing how to 
proceed with Python 2.7, I have some more inputs to contribute.

To put it up front, I'm totally against "CPython 2.8" ever becoming a real 
thing. Anything that comes out should be seen as a migration path, not an 
upgrade path. I'll also admit I'm not heavily invested in working on it myself, 
but I had a number of conversations during PyCon (as well as being at the 
language summit) that puts me in a position to share the ideas and concerns 
that have been raised.

The main trigger was a conversation I had with two employees of a very large 
bank that has about 3000 Python users (not developers - mostly financial 
analysts) and 16 million lines of code running on 2.7. They are keen to migrate 
to 3.x but cannot afford to stop work entirely while their code is updated. 
(There was much more to the conversation than I'm relating here - I'm keeping 
to the directly relevant bits.)

In describing the approach they'd like to take, they made me realise that there 
is definitely a place for a Python that is different but mostly compatible with 
2.7, in a way that 2.7.x could not be. For the sake of having a name, I'll 
refer to this as "Python 2migr8" (pronounced "to migrate" :) ).

The two important components of Python 2migr8 would be the ability to disable 
2.7-only features, and to do so on a module-by-module basis.

My best idea so far would be to have a magic comment (to ensure 2.7 
compatibility better than a "from __future__ ...") near the top of the file 
that marks that file as "must straddle 2.7 and 3.3". Adding this comment causes 
(for example) the parser to treat "except x, y" as a syntax error in this file, 
forces "from __future__ import ...", hides "dict.iterkeys", undefines 
"basestring", etc., but only for this file. (I haven't thought through all the 
possibilities or implications - Eric Snow said he was going to sprint on this 
today/tomorrow, so he'll soon have a better idea just what can be done.)

In effect, 2migr8 would be the version that *only* supports "single-source" 
files. This allows large code bases to progressively migrate modules from 2.x 
to single-source while continuing to run against Python 2.7. As files are 
updated, they are marked as being single-source. When all files have this 
marker, it should be possible to flip the switch and run with Python 3.3 or 
later.

You could also think of this as enabling "-3 --warnings-as-errors" for 
individual modules, though since the user has already opted in to 2migr8, it 
isn't unreasonable to make more significant changes, like having dict.keys 
returning a list that warns if it is mutated. This sort of warning can only 
really be done by changing the interpreter - static analysis just can't catch 
everything - and only when users accept a potential performance hit and low 
probability of breakage when they move to 2migr8 (followed by a 
not-quite-as-low probability of breaking when they eventually move from 2migr8 
to 3.x, but it's still better than guaranteed breakage).

As a fork, it would also be possible to bundle the modules that have been 
backported, and possibly also to disallow importing deprecated stdlib modules 
when 2.7 functionality is disabled. As I said, I haven't thought through all 
the possibilities, but the general idea is to take 2.7 and *remove* features so 
it becomes easier to migrate.

Where does python-dev come in? Obviously this is where a fork like this would 
have to start - there has been such strong and public opposition to any 
significant changes like this that you'd be hard pressed to find someone 
willing to start and promote it from outside. There is also a good opportunity 
to make a start and directly invite those using it to contribute the rules or 
warnings that they need - the 3000 Python "users" I mentioned earlier are 
backed by a team of true developers who are more than capable of contributing, 
and this would be a great opportunity to directly invite them. However unfair 
and incorrect it may be, there is a perception in some businesses that 
open-source projects do not want contributions from them. I invited more than 
one business to have someone join python-dev and get involved during PyCon, and 
I heard that others did the same - it may not be at the level of employing a 
core developer full time, but it's the starting point that some companies will 
ne
 ed to be able to become comfortable with employing a core dev.

I'm not pretending to have a full plan on how this will work. I was privileged 
to have some private conversations during PyCon that are directly relevant, so 
I'm bringing it here to promote the discussion. Thanks to everyone I had a 
chance to chat to, and to everyone generally for a great PyCon.

Cheers,
Steve

 
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.pyt

Re: [Python-Dev] [numpy wishlist] PyMem_*Calloc

[Benjamin Peterson]

On Sun, Apr 13, 2014, at 22:39, Nathaniel Smith wrote:
> Hi all,
> 
> The new tracemalloc infrastructure in python 3.4 is super-interesting
> to numerical folks, because we really like memory profiling. Numerical
> programs allocate a lot of memory, and sometimes it's not clear which
> operations allocate memory (some numpy operations return views of the
> original array without allocating anything; others return copies). So
> people actually use memory tracking tools[1], even though
> traditionally these have been pretty hacky (i.e., just checking RSS
> before and after each line is executed), and numpy has even grown its
> own little tracemalloc-like infrastructure [2], but it only works for
> numpy data.
> 
> BUT, we also really like calloc(). One of the basic array creation
> routines in numpy is numpy.zeros(), which returns an array full of --
> you guessed it -- zeros. For pretty much all the data types numpy
> supports, the value zero is represented by the bytestring consisting
> of all zeros. So numpy.zeros() usually uses calloc() to allocate its
> memory.
> 
> calloc() is more awesome than malloc()+memset() for two reasons.
> First, calloc() for larger allocations is usually implemented using
> clever VM tricks, so that it doesn't actually allocate any memory up
> front, it just creates a COW mapping of the system zero page and then
> does the actual allocation one page at a time as different entries are
> written to. This means that in the somewhat common case where you
> allocate a large array full of zeros, and then only set a few
> scattered entries to non-zero values, you can end up using much much
> less memory than otherwise. It's entirely possible for this to make
> the difference between being able to run an analysis versus not.
> memset() forces the whole amount of RAM to be committed immediately.
> 
> Secondly, even if you *are* going to touch all the memory, then
> calloc() is still faster than malloc()+memset(). The reason is that
> for large allocations, malloc() usually does a calloc() no matter what
> -- when you get a new page from the kernel, the kernel has to make
> sure you can't see random bits of other processes's memory, so it
> unconditionally zeros out the page before you get to see it. calloc()
> knows this, so it doesn't bother zeroing it again. malloc()+memset(),
> by contrast, zeros the page twice, producing twice as much memory
> traffic, which is huge.
> 
> SO, we'd like to route our allocations through PyMem_* in order to let
> tracemalloc "see" them, but because there is no PyMem_*Calloc, doing
> this would force us to give up on the calloc() optimizations.
> 
> The obvious solution is to add a PyMem_*Calloc to the API. Would this
> be possible? Unfortunately it would require adding a new field to the
> PyMemAllocator struct, which would be an ABI/API break; PyMemAllocator
> is exposed directly in the C API and passed by value:
>   https://docs.python.org/3.5/c-api/memory.html#c.PyMemAllocator
> (Too bad we didn't notice this a few months ago before 3.4 was
> released :-(.) I guess we could just rename the struct in 3.5, to
> force people to update their code. (I guess there aren't too many
> people who would have to update their code.)

Well, the allocator API is not part of the stable ABI, so we can change
it if we want.

> 
> Thoughts?

I think the request is completely reasonable.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python "2migr8"

[Chris Angelico]

On Tue, Apr 15, 2014 at 1:32 AM, Steve Dower  wrote:
> My best idea so far would be to have a magic comment (to ensure 2.7 
> compatibility better than a "from __future__ ...") near the top of the file 
> that marks that file as "must straddle 2.7 and 3.3". Adding this comment 
> causes (for example) the parser to treat "except x, y" as a syntax error in 
> this file, forces "from __future__ import ...", hides "dict.iterkeys", 
> undefines "basestring", etc., but only for this file.
>

So if I understand you, this isn't really a "new version of Python" so
much as "Python 2.7 with checks for anything that would break on 3.3".
(BTW, any particular reason for targeting 3.3 specifically? PEP 461,
currently slated for 3.5, may make your job rather easier.) And the
file should run perfectly on both 2.7 and 3.3, and have the exact same
semantics. Is that correct?

If that's the case, how critical are false positives and false
negatives? You could make something that heuristically tries to figure
out when something could be a problem, and in dubious cases issues a
warning. That could afford to be a bit more aggressive in its warnings
than something that actually throws stuff out. You could even do a
whole lot of messy monkey-patching that checks stuff at run-time; if
performance becomes a problem, just switch back to 2.7 and it'll still
all run fine.

Ultimately, the best test of whether or not a piece of code runs in
2.7 and 3.3 is to simply run it in 2.7 and 3.3, but obviously that's
hard to do on a partial codebase. The trouble with a directive like
this is that it's hard to pin-point the source of a problem. For
instance, you say:

> ... it isn't unreasonable to make more significant changes, like having 
> dict.keys returning a list that warns if it is mutated.
>

Suppose function foo calls function bar, passing it the .keys() of a
dict; function bar treats its argument as a list. Which one is at
fault? If they're in different modules and only one of them is marked
for migration mode, do you get a warning/error or not? Going for
warnings is the safe option here. You could warn for both half-cases
and error for the double. That is, dict.keys() returns a subclass of
list that, on mutation, will warn if the calling module is marked
migration-safe xor the module that originally created it is so marked,
and throws an error if both were marked. But for that to work, you
have to accept that there'll be both false positives (warnings issued
when the problem isn't in the migration-safe code) and false negatives
(warnings, rather than errors, when your code has a problem). Does
that kill the proposal's usefulness, or is that an acceptable cost?

ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] cpython (3.1): disallow a negative idx parameter

[Zachary Ware]

On Mon, Apr 14, 2014 at 10:52 AM, benjamin.peterson
 wrote:
> http://hg.python.org/cpython/rev/4bd1fb0f4f44
> changeset:   90256:4bd1fb0f4f44
> branch:  3.1
> parent:  90235:a8facac493ef
> user:Benjamin Peterson 
> date:Mon Apr 14 11:45:21 2014 -0400
> summary:
>   disallow a negative idx parameter
>
> files:
>   Modules/_json.c |  9 +
>   1 files changed, 5 insertions(+), 4 deletions(-)
>
>
> diff --git a/Modules/_json.c b/Modules/_json.c
> --- a/Modules/_json.c
> +++ b/Modules/_json.c
> @@ -902,10 +902,11 @@
>  PyObject *res;
>  Py_UNICODE *str = PyUnicode_AS_UNICODE(pystr);
>  Py_ssize_t length = PyUnicode_GET_SIZE(pystr);
> -if (idx < 0)
> -/* Compatibility with Python version. */
> -idx += length;
> -if (idx < 0 || idx >= length) {
> +if (idx < 0) {
> +PyErr_SetString(PyExc_ValueError, "idx canont be negative");

s/canont/cannot/

Also in the comparable 3.2 commit, but not the 3.3+ merge.

Regards,
-- 
Zach
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] cpython (3.1): disallow a negative idx parameter

[Benjamin Peterson]

On Mon, Apr 14, 2014, at 9:14, Zachary Ware wrote:
> On Mon, Apr 14, 2014 at 10:52 AM, benjamin.peterson
>  wrote:
> > http://hg.python.org/cpython/rev/4bd1fb0f4f44
> > changeset:   90256:4bd1fb0f4f44
> > branch:  3.1
> > parent:  90235:a8facac493ef
> > user:Benjamin Peterson 
> > date:Mon Apr 14 11:45:21 2014 -0400
> > summary:
> >   disallow a negative idx parameter
> >
> > files:
> >   Modules/_json.c |  9 +
> >   1 files changed, 5 insertions(+), 4 deletions(-)
> >
> >
> > diff --git a/Modules/_json.c b/Modules/_json.c
> > --- a/Modules/_json.c
> > +++ b/Modules/_json.c
> > @@ -902,10 +902,11 @@
> >  PyObject *res;
> >  Py_UNICODE *str = PyUnicode_AS_UNICODE(pystr);
> >  Py_ssize_t length = PyUnicode_GET_SIZE(pystr);
> > -if (idx < 0)
> > -/* Compatibility with Python version. */
> > -idx += length;
> > -if (idx < 0 || idx >= length) {
> > +if (idx < 0) {
> > +PyErr_SetString(PyExc_ValueError, "idx canont be negative");
> 
> s/canont/cannot/
> 
> Also in the comparable 3.2 commit, but not the 3.3+ merge.

Thanks
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python "2migr8"

[Guido van Rossum]

Some quick thoughts:

- I'd prefer a name that plays on 2 and 3, not 2 and 8. :-)

- Are you sure this isn't better directed to python-ideas first? Most ideas
have to prove their worth in that list before python-dev will give them the
light of day.

- When it comes to purely syntactic issues (e.g. "except x, y:") a linter
or some other separate tool can handle this well (heck, you can build it
into an import hook probably :-).

- When it's about backported modules, a sumo distribution is probably the
way to go; when it's about renamed stdlib modules, six (perhaps an extended
version) should cover you.

- Regarding warning about the changed dict API, I wonder how you plan to
implement that if you allow passing dict object back and forth between code
that has opted in to single-source and code that hasn't yet. Please think
through some specific examples before responding.

- But the biggest issue is of course bytes vs. text. You would have to
first do a careful analysis of the *whole* problem before you can even
think about proposing a solution. Too many people think their is an easy
solution for this; but almost everybody is focused on only part of the
problem (the part that causes them immediate pain) without realizing that
other people's pain may be different.

- As far as your assertion that it would have to come from python-dev
because nobody outside is going to tackle it, I think it's the opposite:
the core developers would prefer not to have to deal with this, while some
folks outside the inner circles will not be discouraged by our opinions
(e.g. Stackless is working on "Stackless 2.8").

- Regarding open source projects having a reputation of "not taking
contributions", I would guess that this is usually about those
"contributions" violating the most basic rules of the project (and I don't
mean the coding style). I do want to discourage discussions with users like
the company you referred to, but I think it would be much more useful if
they laid out their problems for us instead of expecting they can buy
acceptance for a "solution" they develop in-house. We could then hopefully
have a productive dialog over many months where we iterate over possible
approaches that could be acceptable both to Python and to the customer. But
it will take a significant investment of time on both sides -- there is no
shortcut. And it's not a particularly interesting problem (for most people)
to work on -- things like designing a notation for optional type
declarations are always much more fun. :-)




On Mon, Apr 14, 2014 at 11:32 AM, Steve Dower wrote:

>
> Just in case there's anyone out there who isn't yet sick of discussing how
> to proceed with Python 2.7, I have some more inputs to contribute.
>
> To put it up front, I'm totally against "CPython 2.8" ever becoming a real
> thing. Anything that comes out should be seen as a migration path, not an
> upgrade path. I'll also admit I'm not heavily invested in working on it
> myself, but I had a number of conversations during PyCon (as well as being
> at the language summit) that puts me in a position to share the ideas and
> concerns that have been raised.
>
> The main trigger was a conversation I had with two employees of a very
> large bank that has about 3000 Python users (not developers - mostly
> financial analysts) and 16 million lines of code running on 2.7. They are
> keen to migrate to 3.x but cannot afford to stop work entirely while their
> code is updated. (There was much more to the conversation than I'm relating
> here - I'm keeping to the directly relevant bits.)
>
> In describing the approach they'd like to take, they made me realise that
> there is definitely a place for a Python that is different but mostly
> compatible with 2.7, in a way that 2.7.x could not be. For the sake of
> having a name, I'll refer to this as "Python 2migr8" (pronounced "to
> migrate" :) ).
>
> The two important components of Python 2migr8 would be the ability to
> disable 2.7-only features, and to do so on a module-by-module basis.
>
> My best idea so far would be to have a magic comment (to ensure 2.7
> compatibility better than a "from __future__ ...") near the top of the file
> that marks that file as "must straddle 2.7 and 3.3". Adding this comment
> causes (for example) the parser to treat "except x, y" as a syntax error in
> this file, forces "from __future__ import ...", hides "dict.iterkeys",
> undefines "basestring", etc., but only for this file. (I haven't thought
> through all the possibilities or implications - Eric Snow said he was going
> to sprint on this today/tomorrow, so he'll soon have a better idea just
> what can be done.)
>
> In effect, 2migr8 would be the version that *only* supports
> "single-source" files. This allows large code bases to progressively
> migrate modules from 2.x to single-source while continuing to run against
> Python 2.7. As files are updated, they are marked as being single-source.
> When all files have this marker, it should be po

Re: [Python-Dev] Python "2migr8"

[Terry Reedy]

On 4/14/2014 1:19 PM, Guido van Rossum wrote:

Some quick thoughts:

- I'd prefer a name that plays on 2 and 3, not 2 and 8. :-)

- Are you sure this isn't better directed to python-ideas first? Most
ideas have to prove their worth in that list before python-dev will give
them the light of day.

- When it comes to purely syntactic issues (e.g. "except x, y:") a
linter or some other separate tool can handle this well (heck, you can
build it into an import hook probably :-).

- When it's about backported modules, a sumo distribution is probably
the way to go; when it's about renamed stdlib modules, six (perhaps an
extended version) should cover you.

- Regarding warning about the changed dict API, I wonder how you plan to
implement that if you allow passing dict object back and forth between
code that has opted in to single-source and code that hasn't yet. Please
think through some specific examples before responding.

- But the biggest issue is of course bytes vs. text. You would have to
first do a careful analysis of the *whole* problem before you can even
think about proposing a solution. Too many people think their is an easy
solution for this; but almost everybody is focused on only part of the
problem (the part that causes them immediate pain) without realizing
that other people's pain may be different.

- As far as your assertion that it would have to come from python-dev
because nobody outside is going to tackle it, I think it's the opposite:
the core developers would prefer not to have to deal with this, while
some folks outside the inner circles will not be discouraged by our
opinions (e.g. Stackless is working on "Stackless 2.8").

- Regarding open source projects having a reputation of "not taking
contributions", I would guess that this is usually about those
"contributions" violating the most basic rules of the project (and I
don't mean the coding style). I do want to discourage discussions with


Did you mean 'don't want to discourage'?


users like the company you referred to, but I think it would be much
more useful if they laid out their problems for us instead of expecting
they can buy acceptance for a "solution" they develop in-house. We could
then hopefully have a productive dialog over many months where we
iterate over possible approaches that could be acceptable both to Python
and to the customer. But it will take a significant investment of time
on both sides -- there is no shortcut. And it's not a particularly
interesting problem (for most people) to work on -- things like
designing a notation for optional type declarations are always much more
fun. :-)




On Mon, Apr 14, 2014 at 11:32 AM, Steve Dower mailto:steve.do...@microsoft.com>> wrote:


Just in case there's anyone out there who isn't yet sick of
discussing how to proceed with Python 2.7, I have some more inputs
to contribute.

To put it up front, I'm totally against "CPython 2.8" ever becoming
a real thing. Anything that comes out should be seen as a migration
path, not an upgrade path. I'll also admit I'm not heavily invested
in working on it myself, but I had a number of conversations during
PyCon (as well as being at the language summit) that puts me in a
position to share the ideas and concerns that have been raised.

The main trigger was a conversation I had with two employees of a
very large bank that has about 3000 Python users (not developers -
mostly financial analysts) and 16 million lines of code running on
2.7. They are keen to migrate to 3.x but cannot afford to stop work
entirely while their code is updated. (There was much more to the
conversation than I'm relating here - I'm keeping to the directly
relevant bits.)

In describing the approach they'd like to take, they made me realise
that there is definitely a place for a Python that is different but
mostly compatible with 2.7, in a way that 2.7.x could not be. For
the sake of having a name, I'll refer to this as "Python 2migr8"
(pronounced "to migrate" :) ).

The two important components of Python 2migr8 would be the ability
to disable 2.7-only features, and to do so on a module-by-module basis.

My best idea so far would be to have a magic comment (to ensure 2.7
compatibility better than a "from __future__ ...") near the top of
the file that marks that file as "must straddle 2.7 and 3.3". Adding
this comment causes (for example) the parser to treat "except x, y"
as a syntax error in this file, forces "from __future__ import ...",
hides "dict.iterkeys", undefines "basestring", etc., but only for
this file. (I haven't thought through all the possibilities or
implications - Eric Snow said he was going to sprint on this
today/tomorrow, so he'll soon have a better idea just what can be done.)

In effect, 2migr8 would be the version that *only* supports
"single-source" files. This allows large code bases to progressively
migra

Re: [Python-Dev] Python "2migr8"

[Terry Reedy]

On 4/14/2014 11:32 AM, Steve Dower wrote:


To put it up front, I'm totally against "CPython 2.8" ever becoming a
real thing. Anything that comes out should be seen as a migration
path, not an upgrade path. I'll also admit I'm not heavily invested
in working on it myself, but I had a number of conversations during
PyCon (as well as being at the language summit) that puts me in a
position to share the ideas and concerns that have been raised.


I think it great that you 'volunteered' to be a neutral, hopefully 
trusted go-between.



The main trigger was a conversation I had with two employees of a
very large bank that has about 3000 Python users (not developers -
mostly financial analysts) and 16 million lines of code running on
2.7.


Sounds like a billion-dollar company. Are they a PSF sponsor, and a 
top-tier one at that? If the company is profitable, it could afford to 
fund a half- to full-time developer.


> They are keen to migrate to 3.x but cannot afford to stop work

entirely while their code is updated.


Sounds like they are looking ahead several years and anxious to avoid 
the 'comforable with XP' trap.



In describing the approach they'd like to take, they made me realise
that there is definitely a place for a Python that is different but
mostly compatible with 2.7, in a way that 2.7.x could not be. For the
sake of having a name, I'll refer to this as "Python 2migr8"
(pronounced "to migrate" :) ).


This should be a separate project from pydev, even if under the PSF 
umbrella.



The two important components of Python 2migr8 would be the ability to
disable 2.7-only features, and to do so on a module-by-module basis.


A reasonable request of pydev would be for python-coded stdlib modules 
to be updated as much as possible, if that has not already been done. No 
'apply', no 'except SomeException, e'.



However unfair
and incorrect it may be, there is a perception in some businesses
that open-source projects do not want contributions from them.


For PSF/CPython, this is so untrue that it looks to me like an excuse to 
take without giving back. This might be 'unfair and incorrect', but it 
is my perception.



I invited more than one business to have someone join python -dev and
get involved during PyCon, and I heard that others did the same - it
may not be at the level of employing a core developer full time, but
it's the starting point that some companies will need to be able to
become comfortable with employing a core dev.


Let's hope some act on your invitation.

--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python "2migr8"

[Donald Stufft]

On Apr 14, 2014, at 3:53 PM, Terry Reedy  wrote:

> On 4/14/2014 11:32 AM, Steve Dower wrote:
> 
>> To put it up front, I'm totally against "CPython 2.8" ever becoming a
>> real thing. Anything that comes out should be seen as a migration
>> path, not an upgrade path. I'll also admit I'm not heavily invested
>> in working on it myself, but I had a number of conversations during
>> PyCon (as well as being at the language summit) that puts me in a
>> position to share the ideas and concerns that have been raised.
> 
> I think it great that you 'volunteered' to be a neutral, hopefully trusted 
> go-between.
> 
>> The main trigger was a conversation I had with two employees of a
>> very large bank that has about 3000 Python users (not developers -
>> mostly financial analysts) and 16 million lines of code running on
>> 2.7.
> 
> Sounds like a billion-dollar company. Are they a PSF sponsor, and a top-tier 
> one at that? If the company is profitable, it could afford to fund a half- to 
> full-time developer.
> 
> > They are keen to migrate to 3.x but cannot afford to stop work
>> entirely while their code is updated.
> 
> Sounds like they are looking ahead several years and anxious to avoid the 
> 'comforable with XP' trap.
> 
>> In describing the approach they'd like to take, they made me realise
>> that there is definitely a place for a Python that is different but
>> mostly compatible with 2.7, in a way that 2.7.x could not be. For the
>> sake of having a name, I'll refer to this as "Python 2migr8"
>> (pronounced "to migrate" :) ).
> 
> This should be a separate project from pydev, even if under the PSF umbrella.
> 
>> The two important components of Python 2migr8 would be the ability to
>> disable 2.7-only features, and to do so on a module-by-module basis.
> 
> A reasonable request of pydev would be for python-coded stdlib modules to be 
> updated as much as possible, if that has not already been done. No 'apply', 
> no 'except SomeException, e'.
> 
>> However unfair
>> and incorrect it may be, there is a perception in some businesses
>> that open-source projects do not want contributions from them.
> 
> For PSF/CPython, this is so untrue that it looks to me like an excuse to take 
> without giving back. This might be 'unfair and incorrect', but it is my 
> perception.

As someone who *has* given back, I can certainly understand why someone would 
feel that way. It often times *does* feel like CPython doesn’t want 
contributions.

> 
>> I invited more than one business to have someone join python -dev and
>> get involved during PyCon, and I heard that others did the same - it
>> may not be at the level of employing a core developer full time, but
>> it's the starting point that some companies will need to be able to
>> become comfortable with employing a core dev.
> 
> Let's hope some act on your invitation.
> 
> -- 
> Terry Jan Reedy
> 
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/donald%40stufft.io


-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python "2migr8"

[Guido van Rossum]

On Apr 14, 2014 2:42 PM, "Terry Reedy"  wrote:
>
> On 4/14/2014 1:19 PM, Guido van Rossum wrote:
>>
>> Some quick thoughts:
>>
>> - I'd prefer a name that plays on 2 and 3, not 2 and 8. :-)
>>
>> - Are you sure this isn't better directed to python-ideas first? Most
>> ideas have to prove their worth in that list before python-dev will give
>> them the light of day.
>>
>> - When it comes to purely syntactic issues (e.g. "except x, y:") a
>> linter or some other separate tool can handle this well (heck, you can
>> build it into an import hook probably :-).
>>
>> - When it's about backported modules, a sumo distribution is probably
>> the way to go; when it's about renamed stdlib modules, six (perhaps an
>> extended version) should cover you.
>>
>> - Regarding warning about the changed dict API, I wonder how you plan to
>> implement that if you allow passing dict object back and forth between
>> code that has opted in to single-source and code that hasn't yet. Please
>> think through some specific examples before responding.
>>
>> - But the biggest issue is of course bytes vs. text. You would have to
>> first do a careful analysis of the *whole* problem before you can even
>> think about proposing a solution. Too many people think their is an easy
>> solution for this; but almost everybody is focused on only part of the
>> problem (the part that causes them immediate pain) without realizing
>> that other people's pain may be different.
>>
>> - As far as your assertion that it would have to come from python-dev
>> because nobody outside is going to tackle it, I think it's the opposite:
>> the core developers would prefer not to have to deal with this, while
>> some folks outside the inner circles will not be discouraged by our
>> opinions (e.g. Stackless is working on "Stackless 2.8").
>>
>> - Regarding open source projects having a reputation of "not taking
>> contributions", I would guess that this is usually about those
>> "contributions" violating the most basic rules of the project (and I
>> don't mean the coding style). I do want to discourage discussions with
>
>
> Did you mean 'don't want to discourage'?

Yes. Sorry. Pylon brain fry...

>> users like the company you referred to, but I think it would be much
>> more useful if they laid out their problems for us instead of expecting
>> they can buy acceptance for a "solution" they develop in-house. We could
>> then hopefully have a productive dialog over many months where we
>> iterate over possible approaches that could be acceptable both to Python
>> and to the customer. But it will take a significant investment of time
>> on both sides -- there is no shortcut. And it's not a particularly
>> interesting problem (for most people) to work on -- things like
>> designing a notation for optional type declarations are always much more
>> fun. :-)
>>
>>
>>
>>
>> On Mon, Apr 14, 2014 at 11:32 AM, Steve Dower > > wrote:
>>
>>
>> Just in case there's anyone out there who isn't yet sick of
>> discussing how to proceed with Python 2.7, I have some more inputs
>> to contribute.
>>
>> To put it up front, I'm totally against "CPython 2.8" ever becoming
>> a real thing. Anything that comes out should be seen as a migration
>> path, not an upgrade path. I'll also admit I'm not heavily invested
>> in working on it myself, but I had a number of conversations during
>> PyCon (as well as being at the language summit) that puts me in a
>> position to share the ideas and concerns that have been raised.
>>
>> The main trigger was a conversation I had with two employees of a
>> very large bank that has about 3000 Python users (not developers -
>> mostly financial analysts) and 16 million lines of code running on
>> 2.7. They are keen to migrate to 3.x but cannot afford to stop work
>> entirely while their code is updated. (There was much more to the
>> conversation than I'm relating here - I'm keeping to the directly
>> relevant bits.)
>>
>> In describing the approach they'd like to take, they made me realise
>> that there is definitely a place for a Python that is different but
>> mostly compatible with 2.7, in a way that 2.7.x could not be. For
>> the sake of having a name, I'll refer to this as "Python 2migr8"
>> (pronounced "to migrate" :) ).
>>
>> The two important components of Python 2migr8 would be the ability
>> to disable 2.7-only features, and to do so on a module-by-module
basis.
>>
>> My best idea so far would be to have a magic comment (to ensure 2.7
>> compatibility better than a "from __future__ ...") near the top of
>> the file that marks that file as "must straddle 2.7 and 3.3". Adding
>> this comment causes (for example) the parser to treat "except x, y"
>> as a syntax error in this file, forces "from __future__ import ...",
>> hides "dict.iterkeys", undefines "basestring", etc., but only for
>> this file. 

Re: [Python-Dev] [numpy wishlist] PyMem_*Calloc

[Ethan Furman]

On 04/14/2014 08:36 AM, Benjamin Peterson wrote:

On Sun, Apr 13, 2014, at 22:39, Nathaniel Smith wrote:


SO, we'd like to route our allocations through PyMem_* in order to let
tracemalloc "see" them, but because there is no PyMem_*Calloc, doing
this would force us to give up on the calloc() optimizations.


Well, the allocator API is not part of the stable ABI, so we can change
it if we want.




Thoughts?


I think the request is completely reasonable.


+1

--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python "2migr8"

[Guido van Rossum]

On Mon, Apr 14, 2014 at 4:02 PM, Donald Stufft  wrote:

>
> On Apr 14, 2014, at 3:53 PM, Terry Reedy  wrote:
>
> > On 4/14/2014 11:32 AM, Steve Dower wrote:
> [...]
> >> However unfair
> >> and incorrect it may be, there is a perception in some businesses
> >> that open-source projects do not want contributions from them.
>
> > For PSF/CPython, this is so untrue that it looks to me like an excuse to
> take without giving back. This might be 'unfair and incorrect', but it is
> my perception.
>
> As someone who *has* given back, I can certainly understand why someone
> would feel that way. It often times *does* feel like CPython doesn’t want
> contributions.
>

Donald, your remark in itself sounds unnecessarily (and unproductively!)
passive-aggressive. What have we done wrong to you, and what can we do to
avoid making the same mistake in the future (to you, and to others)?

[PS. When I appeared to write "Pylon brain fry" earlier in this thread,
that was an unfortunate auto-correct for "PyCon brain fry". We need to get
"PyCon" into the dictionary...]

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python "2migr8"

[Guido van Rossum]

On Mon, Apr 14, 2014 at 3:53 PM, Terry Reedy  wrote:

> On 4/14/2014 11:32 AM, Steve Dower wrote:
>
[...]
>
 The main trigger was a conversation I had with two employees of a
>> very large bank that has about 3000 Python users (not developers -
>> mostly financial analysts) and 16 million lines of code running on
>> 2.7.
>>
>
> Sounds like a billion-dollar company. Are they a PSF sponsor, and a
> top-tier one at that? If the company is profitable, it could afford to fund
> a half- to full-time developer.


A few people have made similar suggestions to me at the conference, but I
personally believe that there is a better way.

I don't think we ought to make companies feel bad about not donating to the
PSF. The PSF is doing fine, but IMO it shouldn't be in the business of
employing core developers. Being an employer is fraught with difficulties,
and there are serious risks both for the PSF (due to the rigidity of
employment laws, for example) and for the employee (e.g. benefits, worry
about continuity, oversight and direction).

IMO a much better approach would be to convince companies to free up some
of their current employees or new hires to invest e.g. 50% of their time
into core Python development. Such "liaison" folks can become a valuable
resource both for the other Python users in the company and for the core
Python developers (by telling us about pain points in their company). In
the ideal situation such people will help search for solutions that benefit
their employers while also conforming to (and sometimes influencing! :-)
the general direction of Python core development. The (roughly) 50/50
division ensures that such employees are rooted in the practices of their
company and feel the pain of other Python users directly. It also makes
sure that they won't be considered as having too much of a "special status"
by their colleagues and managers, can share freely in the companies
benefits and incentives, and so on.

(I should say that this is my own situation at Dropbox and previously at
Google, and I personally wouldn't want it any other way. I know not
everybody wants to be a "regular employee", and some folks think of
themselves as "free spirits" who don't want to be "working for the man".
There's plenty of space for such folks in the Python core dev community,
but I think that the kind of company that Steve Dower is speaking of would
not be helped as much by hiring such a "consultant".)


> > They are keen to migrate to 3.x but cannot afford to stop work
>
>> entirely while their code is updated.
>>
>
> Sounds like they are looking ahead several years and anxious to avoid the
> 'comforable with XP' trap.


Ohhh, nice analogy!


>  In describing the approach they'd like to take, they made me realise
>> that there is definitely a place for a Python that is different but
>> mostly compatible with 2.7, in a way that 2.7.x could not be. For the
>> sake of having a name, I'll refer to this as "Python 2migr8"
>> (pronounced "to migrate" :) ).
>>
>
> This should be a separate project from pydev, even if under the PSF
> umbrella.
>
>
>  The two important components of Python 2migr8 would be the ability to
>> disable 2.7-only features, and to do so on a module-by-module basis.
>>
>
> A reasonable request of pydev would be for python-coded stdlib modules to
> be updated as much as possible, if that has not already been done. No
> 'apply', no 'except SomeException, e'.


I'm not sure what you're proposing here, but I don't think it would be wise
to go on a code modernizing spree of the 2.7 stdlib. Contrary to what some
believe, the stdlib often isn't a great example, because it lives under the
pressure of quite unusual constraints.


>
>
>  However unfair
>> and incorrect it may be, there is a perception in some businesses
>> that open-source projects do not want contributions from them.
>>
>
> For PSF/CPython, this is so untrue that it looks to me like an excuse to
> take without giving back. This might be 'unfair and incorrect', but it is
> my perception.
>
>  I invited more than one business to have someone join python -dev and
>>
>> get involved during PyCon, and I heard that others did the same - it
>> may not be at the level of employing a core developer full time, but
>> it's the starting point that some companies will need to be able to
>>
>> become comfortable with employing a core dev.
>>
>
> Let's hope some act on your invitation.


I really hope to have a direct conversation with some companies in this
situation, but unfortunately they didn't approach me at PyCon -- they only
approached Steve (perhaps because he works for a brand they recognize and
trust :-).

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python "2migr8"

[Donald Stufft]

On Apr 14, 2014, at 4:39 PM, Guido van Rossum  wrote:

> On Mon, Apr 14, 2014 at 4:02 PM, Donald Stufft  wrote:
> 
> On Apr 14, 2014, at 3:53 PM, Terry Reedy  wrote:
> 
> > On 4/14/2014 11:32 AM, Steve Dower wrote:
> [...]
> >> However unfair
> >> and incorrect it may be, there is a perception in some businesses
> >> that open-source projects do not want contributions from them.
> 
> > For PSF/CPython, this is so untrue that it looks to me like an excuse to 
> > take without giving back. This might be 'unfair and incorrect', but it is 
> > my perception.
> 
> As someone who *has* given back, I can certainly understand why someone would 
> feel that way. It often times *does* feel like CPython doesn’t want 
> contributions.
>  
> Donald, your remark in itself sounds unnecessarily (and unproductively!) 
> passive-aggressive. What have we done wrong to you, and what can we do to 
> avoid making the same mistake in the future (to you, and to others)?
> 
> [PS. When I appeared to write "Pylon brain fry" earlier in this thread, that 
> was an unfortunate auto-correct for "PyCon brain fry". We need to get "PyCon" 
> into the dictionary...]
> 
> -- 
> --Guido van Rossum (python.org/~guido)


Hmm, I’m sorry if I came across that way. I didn’t mean to. I do think 
contributing directly to CPython is often times off-putting to people (and 
given I know others who have privately expressed the same sentiment to me I 
don’t think I’m alone in that [1]). I only brought it up because I don’t think 
ignoring a problem (which maybe some disagree that there is a problem! but in 
my view there is so thus my comment) is a useful thing to do. Generally when I 
say something it’s because I’m trying to be helpful :) I’m sorry that my 
wording didn’t convey that appropriately.

Now I will admit I personally have probably had a harder time than some others 
because of the nature of the things I was trying to work on, and lately it’s 
gotten better (although I think that’s partially because I’m more known now, 
and I think in general the experience of contributing to CPython changes 
depending on who you are, so the more integrated into the culture you are, the 
less likely you are to see the issues and unfortunately those people are also 
the ones with the most power to change it). I do however think that just in 
general it might be getting better too? 

Specific details are hard because it’s nothing major and obvious like having 
Linus go off on rants and tearing things apart, it’s death by a thousand cuts 
so it’s hard to point a finger at one behavior (or a few behaviors) and look at 
them in isolation and “see” it. That being said I’m more than happy to *try* 
and explain it, but right this moment I don’t have a lot of time as I’m getting 
ready to step out the door, but I didn’t want to leave this email hanging 
without a reply.

[1] See Also 
http://www.curiousefficiency.org/posts/2011/04/musings-on-culture-of-python-dev.html

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python "2migr8"

[Chris Barker]

On Mon, Apr 14, 2014 at 1:26 PM, Guido van Rossum  wrote:


> >> - I'd prefer a name that plays on 2 and 3, not 2 and 8. :-)
>
How about mirg2**3 (pronounced "migrate") ?

;-)

-Chris



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python "2migr8"

[Mark Lawrence]

On 14/04/2014 22:17, Chris Barker wrote:

On Mon, Apr 14, 2014 at 1:26 PM, Guido van Rossum mailto:gu...@python.org>> wrote:

 >> - I'd prefer a name that plays on 2 and 3, not 2 and 8. :-)

How about mirg2**3 (pronounced "migrate") ?

;-)

-Chris




I agree with the grate part.  Failing that, how about a play on 2 + 8 = 
ache?


--
My fellow Pythonistas, ask not what our language can do for you, ask 
what you can do for our language.


Mark Lawrence

---
This email is free from viruses and malware because avast! Antivirus protection 
is active.
http://www.avast.com


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] this is what happens if you freeze all the modules required for startup

[Brett Cannon]

It was realized during PyCon that since we are freezing importlib we could
now consider freezing all the modules to cut out having to stat or read
them from disk. So for day 1 of the sprints I I decided to hack up a
proof-of-concept to see what kind of performance gain it would get.

Freezing everything except encodings.__init__, os, and _sysconfigdata, it
speeds up startup by 11% compared to default. Compared to 2.7 it shaves 14%
from the slowdown (27% slower vs. 41% slower). The full results are at the
end of the email.

Now the question is whether the maintenance cost of having to rebuild
Python for a select number of stdlib modules is enough to warrant putting
in the effort to make this work. My guess is the best approach would be
adding a Lib/_frozen directory where any modules that we treat like this
would be kept to act as a reminder that you need to rebuild for them (I
would probably move importlib/_boostrap.py as well to make this consistent).

Thoughts?


--

default vs the freezing:

### normal_startup ###

Min: 0.524812 -> 0.473339: 1.11x faster

Avg: 0.534403 -> 0.481245: 1.11x faster

Significant (t=61.80)

Stddev: 0.00466 -> 0.00391: 1.1909x smaller


### startup_nosite ###

Min: 0.307359 -> 0.291939: 1.05x faster

Avg: 0.317667 -> 0.300156: 1.06x faster

Significant (t=26.29)

Stddev: 0.00543 -> 0.00385: 1.4099x smaller

-

2.7 vs the freezing:

### normal_startup ###

Min: 0.367571 -> 0.465264: 1.27x slower

Avg: 0.374404 -> 0.476662: 1.27x slower

Significant (t=-90.26)

Stddev: 0.00313 -> 0.00738: 2.3603x larger


### startup_nosite ###

Min: 0.164510 -> 0.290544: 1.77x slower

Avg: 0.169833 -> 0.301109: 1.77x slower

Significant (t=-286.30)

Stddev: 0.00211 -> 0.00407: 1.9310x larger

-

As a baseline, 2.7 vs default:

### normal_startup ###

Min: 0.368916 -> 0.521758: 1.41x slower

Avg: 0.376784 -> 0.531883: 1.41x slower

Significant (t=-172.82)

Stddev: 0.00423 -> 0.00474: 1.1207x larger


### startup_nosite ###

Min: 0.165156 -> 0.309090: 1.87x slower

Avg: 0.171516 -> 0.319004: 1.86x slower

Significant (t=-283.45)

Stddev: 0.00334 -> 0.00399: 1.1948x larger
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

[Glenn Linderman]

On 4/14/2014 2:51 PM, Brett Cannon wrote:

consider freezing all the modules

...
Now the question is whether the maintenance cost of having to rebuild 
Python for a select number of stdlib modules


"all" versus "select number".

So I'm guessing the proposal is to freeze all the modules that Python 
imports just to get itself running, which would consume no additional 
memory when frozen, and saves time per your performance numbers, rather 
than the whole stdlib, which is what is sort of implied by "all".
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

[Skip Montanaro]

On Mon, Apr 14, 2014 at 4:51 PM, Brett Cannon  wrote:
> Thoughts?

Interesting idea, but YAGNI?

In my work environment (Python 2.7.2, all the heavy lifting done in
C++), startup costs are dominated by dynamic linking of all our C++
libraries and their Boost wrappers:

% time python -c 'import tradelink.snake.v11_2 ; raise SystemExit'

real 0m0.671s
user 0m0.405s
sys 0m0.044s

% time python -c 'raise SystemExit'

real 0m0.022s
user 0m0.011s
sys 0m0.009s

Skip
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

[Brett Cannon]

On Mon, Apr 14, 2014 at 6:15 PM, Skip Montanaro  wrote:

> On Mon, Apr 14, 2014 at 4:51 PM, Brett Cannon  wrote:
> > Thoughts?
>
> Interesting idea, but YAGNI?
>

Not at all. Think of every script you execute that's written in Python. One
of the things the Mercurial folks say is hindering any motivation to switch
to Python 3 is the startup performance.


>
> In my work environment (Python 2.7.2, all the heavy lifting done in
> C++), startup costs are dominated by dynamic linking of all our C++
> libraries and their Boost wrappers:
>

Sure, but not everyone uses Boost or has long running processes where
startup time is minuscule compared to the total execution time.

-Brett


>
> % time python -c 'import tradelink.snake.v11_2 ; raise SystemExit'
>
> real 0m0.671s
> user 0m0.405s
> sys 0m0.044s
>
> % time python -c 'raise SystemExit'
>
> real 0m0.022s
> user 0m0.011s
> sys 0m0.009s
>
> Skip
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python "2migr8"

[Eric Snow]

On Mon, Apr 14, 2014 at 9:32 AM, Steve Dower  wrote:
[snip]
> The two important components of Python 2migr8 would be the ability to disable 
> 2.7-only features, and to do so on a module-by-module basis.

This should be doable with an import hook.  For that matter it would
be pretty straight-forward to provide a script around the Python
binary that installs the import hook.  All this could be bundled up
and distributed through the cheeseshop.

-eric

>
> My best idea so far would be to have a magic comment (to ensure 2.7 
> compatibility better than a "from __future__ ...") near the top of the file 
> that marks that file as "must straddle 2.7 and 3.3". Adding this comment 
> causes (for example) the parser to treat "except x, y" as a syntax error in 
> this file, forces "from __future__ import ...", hides "dict.iterkeys", 
> undefines "basestring", etc., but only for this file. (I haven't thought 
> through all the possibilities or implications - Eric Snow said he was going 
> to sprint on this today/tomorrow, so he'll soon have a better idea just what 
> can be done.)
>
> In effect, 2migr8 would be the version that *only* supports "single-source" 
> files. This allows large code bases to progressively migrate modules from 2.x 
> to single-source while continuing to run against Python 2.7. As files are 
> updated, they are marked as being single-source. When all files have this 
> marker, it should be possible to flip the switch and run with Python 3.3 or 
> later.
>
> You could also think of this as enabling "-3 --warnings-as-errors" for 
> individual modules, though since the user has already opted in to 2migr8, it 
> isn't unreasonable to make more significant changes, like having dict.keys 
> returning a list that warns if it is mutated. This sort of warning can only 
> really be done by changing the interpreter - static analysis just can't catch 
> everything - and only when users accept a potential performance hit and low 
> probability of breakage when they move to 2migr8 (followed by a 
> not-quite-as-low probability of breaking when they eventually move from 
> 2migr8 to 3.x, but it's still better than guaranteed breakage).
>
> As a fork, it would also be possible to bundle the modules that have been 
> backported, and possibly also to disallow importing deprecated stdlib modules 
> when 2.7 functionality is disabled. As I said, I haven't thought through all 
> the possibilities, but the general idea is to take 2.7 and *remove* features 
> so it becomes easier to migrate.
[snip]
> I'm not pretending to have a full plan on how this will work. I was 
> privileged to have some private conversations during PyCon that are directly 
> relevant, so I'm bringing it here to promote the discussion. Thanks to 
> everyone I had a chance to chat to, and to everyone generally for a great 
> PyCon.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

[Brett Cannon]

On Mon, Apr 14, 2014 at 6:07 PM, Glenn Linderman wrote:

>  On 4/14/2014 2:51 PM, Brett Cannon wrote:
>
>  consider freezing all the modules
>
> ...
>
>  Now the question is whether the maintenance cost of having to rebuild
> Python for a select number of stdlib modules
>
>
> "all" versus "select number".
>
> So I'm guessing the proposal is to freeze all the modules that Python
> imports just to get itself running, which would consume no additional
> memory when frozen, and saves time per your performance numbers, rather
> than the whole stdlib, which is what is sort of implied by "all".
>

Yes, exactly.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

[Glenn Linderman]

On 4/14/2014 2:51 PM, Brett Cannon wrote:

Freezing everything except encodings.__init__, os, and _sysconfigdata,


I suppose these are omitted because they can vary in different environments?

But isn't Python built for a particular environment... seems like os 
could be included?


Seems like it would be helpful to have the utf8 encoding preloaded both 
to encourage people to use it rather than something else for the 
load-time performance gain (although likely minuscule for one encoding), 
and because they might as well, since they are spending the memory on it 
anyway!  :)



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python "2migr8"

[Greg Ewing]

Guido van Rossum wrote:

Some quick thoughts:

- I'd prefer a name that plays on 2 and 3, not 2 and 8. :-)


Python Twee?

Or maybe Python Tween, as in "between 2 and 3".

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python "2migr8"

[Terry Reedy]

On 4/14/2014 5:16 PM, Donald Stufft wrote:


On Apr 14, 2014, at 4:39 PM, Guido van Rossum mailto:gu...@python.org>> wrote:



On Mon, Apr 14, 2014 at 4:02 PM, Donald Stufft mailto:don...@stufft.io>> wrote:



On Apr 14, 2014, at 3:53 PM, Terry Reedy mailto:tjre...@udel.edu>> wrote:



> On 4/14/2014 11:32 AM, Steve Dower wrote:
[...]
>> However unfair
>> and incorrect it may be, there is a perception in some businesses
>> that open-source projects do not want contributions from them.


I took this to mean that they think any 'do not want' is peculiar to 
them as from a business. Perhaps I was wrong to project that on them.



> For PSF/CPython, this is so untrue that it looks to me like an
excuse to take without giving back.


What I meant would be untrue would be a claim that this project does not 
want contributions in particular from people in business organizations. 
A generic claim that pydev is unwelcoming or sometimes off-putting to 
newcomers, for instance, would be a different issue.



As someone who *has* given back, I can certainly understand why
someone would feel that way. It often times *does* feel like
CPython doesn’t want contributions.



Donald, your remark in itself sounds unnecessarily (and
unproductively!) passive-aggressive. What have we done wrong to you,
and what can we do to avoid making the same mistake in the future (to
you, and to others)?



Hmm, I’m sorry if I came across that way. I didn’t mean to. I do think
contributing directly to CPython is often times off-putting to people


Change 'often' to 'sometimes' and I would agree.

--
Terry Jan Reedy


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python "2migr8"

[Terry Reedy]

On 4/14/2014 5:00 PM, Guido van Rossum wrote:

On Mon, Apr 14, 2014 at 3:53 PM, Terry Reedy mailto:tjre...@udel.edu>> wrote:



If the company is profitable, it could afford
to fund a half- to full-time developer.


By using the vague 'fund' I meant either hire themselves or donate to 
PSF to somehow 'fund' work. I think we need both.



A few people have made similar suggestions to me at the conference, but
I personally believe that there is a better way.

I don't think we ought to make companies feel bad about not donating to
the PSF.


My only beef is with people who use Python *and* complain about unpaid 
volunteers not doing the un-fun work they want done that they could do 
or fund themselves. But that is part of life.


 The PSF is doing fine, but IMO it shouldn't be in the business

of employing core developers. Being an employer is fraught with
difficulties, and there are serious risks both for the PSF (due to the
rigidity of employment laws, for example) and for the employee (e.g.
benefits, worry about continuity, oversight and direction).


I was thinking in terms of contracting rather than employing -- perhaps 
for working on the backlog of hundreds of doc issues. But even that 
requires selection, training, and supervision. Perhaps I should write a 
grant application to pick and supervise one or more college students to 
work on neglected issues. Then the grants committee volunteers would 
just have to say yes or no and later continue or stop.



IMO a much better approach would be to convince companies to free up
some of their current employees or new hires to invest e.g. 50% of their
time into core Python development.


This would be great, but it is not something I can be very convincing 
about ;-). I hope you succeed. But I suspect that some of the things I 
think need to be done will not be done by corporate employees. Hence the 
'both' above.

...

(I should say that this is my own situation at Dropbox and previously at
Google, and I personally wouldn't want it any other way.)


I think your continued practical experience is good for Python.


Sounds like they are looking ahead several years and anxious to
avoid the 'comfortable with XP' trap.


Ohhh, nice analogy!



The two important components of Python 2migr8 would be the ability to
disable 2.7-only features, and to do so on a module-by-module basis.



A reasonable request of pydev would be for python-coded stdlib
modules to be updated as much as possible, if that has not already
been done. No 'apply', no 'except SomeException, e'.


'Reasonable request': a request plausible enough that we should discuss 
it and maybe say yes.


I looked and apply is already not hardly used except in lib2to3 and its 
test in test_builtins. There are hundreds of the old exception form. The 
re for an re.sub call would have to not match tuple commas, such as in 
'except (KeyError, IndexError):



I'm not sure what you're proposing here,


Focused, carefully considered, behavior neutral changes that help 
migration, if indeed there are such, by letting an stdlib module work 
with an altered 2.7 interpreter. The existence of decent test coverage 
for a module would be a consideration.


--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python "2migr8"

[Chris Angelico]

On Tue, Apr 15, 2014 at 7:17 AM, Chris Barker  wrote:
> On Mon, Apr 14, 2014 at 1:26 PM, Guido van Rossum  wrote:
>
>>
>> >> - I'd prefer a name that plays on 2 and 3, not 2 and 8. :-)
>
> How about mirg2**3 (pronounced "migrate") ?
>
> ;-)
>

Just read this, and laughed so loudly and suddenly that my brother's
cat jumped in fright.

Spelled 2**3 and pronounced 8 makes perfect sense, same as spelling
"Mercurial" as "hg".

ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python "2migr8"

[Ryan Gonzalez]

On Mon, Apr 14, 2014 at 9:03 PM, Chris Angelico  wrote:

> On Tue, Apr 15, 2014 at 7:17 AM, Chris Barker 
> wrote:
> > On Mon, Apr 14, 2014 at 1:26 PM, Guido van Rossum 
> wrote:
> >
> >>
> >> >> - I'd prefer a name that plays on 2 and 3, not 2 and 8. :-)
> >
> > How about mirg2**3 (pronounced "migrate") ?
> >
> > ;-)
> >
>
> Just read this, and laughed so loudly and suddenly that my brother's
> cat jumped in fright.
>
> Spelled 2**3 and pronounced 8 makes perfect sense, same as spelling
> "Mercurial" as "hg".
>

...unless the reader doesn't know math. That would be
funny...*migrtoostarstarthree*


>
> ChrisA
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com
>



-- 
Ryan
If anybody ever asks me why I prefer C++ to C, my answer will be simple:
"It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was
nul-terminated."
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python "2migr8"

[Alexander Belopolsky]

On Mon, Apr 14, 2014 at 10:03 PM, Chris Angelico  wrote:

> > How about mirg2**3 (pronounced "migrate") ?
> >
> > ;-)
> >
>
> Just read this, and laughed so loudly and suddenly that my brother's
> cat jumped in fright.
>
> Spelled 2**3 and pronounced 8 makes perfect sense, same as spelling
> "Mercurial" as "hg".


Why is six afraid of seven? :-)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python "2migr8"

[Stephen J. Turnbull]

Guido van Rossum writes:

 > On Mon, Apr 14, 2014 at 4:02 PM, Donald Stufft  wrote:

 >> As someone who *has* given back, I can certainly understand why
 >> someone would feel that way. It often times *does* feel like
 >> CPython doesn’t want contributions.

Sure, but most of the time that feeling is based on a completely
contributor-centric viewpoint.  (Not to mention that I think this is a
little ironic given the position you're taking in the PEP 440
vs. semantic versioning thread.  I agree 100%, but still: ironic.)

As a user and wannabe developer of Python, one of the things I love
about Python (compared to say, Emacsen *shudder*) is that it has been
very good about avoiding the

1. I have a problem.
2. This patch fixes my problem.
3. Now Python has a problem.

syndrome.  Contributors typically feel like they're unwanted, though,
at least at some points in the process of resolving (3) *before* the
patch gets into the mainline.  Of course mostly they're grownups and
get over it, but I suppose that feeling is what Donald is pointing to.

But there's a problem beyond "feeling" for developers paid to work *in
but not on* Python.  I suppose that often their employers *are*
sufficiently enlightened to be willing to allow their people to work
up a full patch (including tests and documentation), but *not* willing
to allow their people to spend twice as much time (or more) getting to
a "fully Pythonic" solution.

And it's not just the employer; the developer herself probably feels
that it's not a profitable use of her time.  She's solved her problem,
she's met the formal requirements for a Python contribution, it's
probably code she's proud of, and she's got 10 more tasks of the same
ilk listed on a whiteboard in her cubicle.  As a professional, of
course she wants to get to work on those.

I can see a few possible ways to address this.

1.  Commit core resources to fast-tracking (through review, concrete
suggestions for improvement, and additional code) contributions
from people who are "on deadline".  (Yeah, I know, that's
problematic, and not just because of the scarcity of core
reviewers.  But what else can we do?)

2.  Educate the employers about the benefits, not only from higher-
quality code, but access to resources like core developer
attention and (perhaps future hiring :-), and happier developers,
that comes from encouraging their people to participate fully in
the process.

The point is to create a "you're *supposed* to spend the extra
time" atmosphere *in the employment context*.

This probably would work best if Guido's boss talks to the
developer's boss at $BILLION_DOLLAR_BUSINESS. :-)

Do Google and DropBox have blogs where they explain their "tithe
to open source" policies?

3.  Related to 2.  I think there's an impression out there that "open
source tithes" are for the top 1% of programmers.  I know that at
LUG meetings and the like I've often heard comments like "nice
work if you can get it, but my employer never will offer it to me".

This impression must be debunked.  And it can be debunked.  Sure,
there are many "we created this job for a particular genius
programmer" positions out there, but many excellent companies have
a generic tithe for all their developers.  We need examples other
than Google.

4.  Get those developers to PyCon.  Get their bosses to PyCon. :-)
Or maybe this should be point #1. :-)

 > Donald, your remark in itself sounds unnecessarily (and
 > unproductively!) passive-aggressive.

FWIW, I didn't read it that way.  I took it as a genuinely "more in
sorrow than in anger" statement.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com