Re: [Python-Dev] Status of the fix for the hash collision vulnerability

[martin]

It doesn't change anything, you will still get collisions.



That depends right? If the collision is because they all have the same
hash(), yes. It might be different if it is because the secondary hashing
(or whatever it's called :-) causes collisions.


But Python deals with the latter case just fine already. The open hashing
approach relies on the dict resizing "enough" to prevent collisions after
the dictionary has grown. Unless somebody can demonstrate a counter example,
I believe this discussion is a red herring.

Plus: if an attacker could craft keys that deliberately cause collisions
because of the dictionary size, they could likely also craft keys in the same
number that collide on actual hash values, bringing us back to the original
problem.

Regards,
Martin


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Backwards incompatible sys.stdout.write() behavior in Python 3 (Was: [Python-ideas] Pythonic buffering in Py3 print())

[anatoly techtonik]

On Fri, Jan 13, 2012 at 7:19 PM, Antoine Pitrou  wrote:

> On Fri, 13 Jan 2012 17:00:57 +0100
> Xavier Morel  wrote:
> > FWIW this is not restricted to Linux (the same behavior change can
> > be observed in OSX), and the script is overly complex you can expose
> > the change with 3 lines
> >
> > import sys
> > sys.stdout.write('promt>')
> > sys.stdin.read(1)
> >
> > Python 2 displays "prompt" and terminates execution on [Return],
> > Python 3 does not display anything until [Return] is pressed.
> >
> > Interestingly, the `-u` option is not sufficient to make
> > "prompt>" appear in Python 3, the stream has to be flushed
> > explicitly unless the input is ~16k characters (I guess that's
> > an internal buffer size of some sort)
>
> "-u" forces line-buffering mode for stdout/stderr, which is already the
> default if they are wired to an interactive device (isattr() returning
> True).
>
> But this was already rehashed on python-ideas and the bug tracker, and
> apparently Anatoly thought it would be a good idea to post on a third
> medium. Sigh.
>

If you track this more closely, you'll notice there are four issues
(surprises) from the user point of view:
1. print() buffers output on Python3
2. print() also buffers output on Python2, but only on Linux
3. there is some useless '-u' command line parameter
(useless, because the last thing user wants is not only care about
Python 2/3, but also how to invoke them)
4. print() is not guilty - it is sys.stdout.write() that buffers output

1-2 discussion was about idea to make new print() function behavior more
'pythonic', i.e. 'user-friendly' or just KISS, which resulted in adding a
flush parameter
3 is a just a side FYI remark
4 doesn't relate to python-ideas anymore about fixing print() - it is about
the *cause* of the problem with print() UX, which is underlying
sys.stdout.write() behavior

I asked 4 here, because it is the more appropriate place not only to ask if
it can be/will be fixed, but also why. The target audience of the question
are developers.

Hope that helps Antoine recover from the sorrow. ;)
-- 
anatoly t.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of the fix for the hash collision vulnerability

[Stephen J. Turnbull]

mar...@v.loewis.de writes:
 > >> It doesn't change anything, you will still get collisions.
 > >
 > >
 > > That depends right? If the collision is because they all have the same
 > > hash(), yes. It might be different if it is because the secondary hashing
 > > (or whatever it's called :-) causes collisions.
 > 
 > But Python deals with the latter case just fine already. The open hashing
 > approach relies on the dict resizing "enough" to prevent collisions after
 > the dictionary has grown. Unless somebody can demonstrate a counter example,
 > I believe this discussion is a red herring.
 >
 > Plus: if an attacker could craft keys that deliberately cause collisions
 > because of the dictionary size, they could likely also craft keys in the same
 > number that collide on actual hash values, bringing us back to the original
 > problem.

I thought that the original problem was that with N insertions in the
dictionary, by repeatedly inserting different keys generating the same
hash value an attacker could arrange that the cost of finding an open
slot is O(N), and thus the cost of N insertions is O(N^2).

If so, frequent resizing could make the attacker's problem much more
difficult, as the distribution of secondary probes should change with
each resize.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of the fix for the hash collision vulnerability

[Victor Stinner]

> I thought that the original problem was that with N insertions in the
> dictionary, by repeatedly inserting different keys generating the same
> hash value an attacker could arrange that the cost of finding an open
> slot is O(N), and thus the cost of N insertions is O(N^2).
>
> If so, frequent resizing could make the attacker's problem much more
> difficult, as the distribution of secondary probes should change with
> each resize.

The attack creates 60,000 strings (or more) with exactly the same hash
value. A dictionary uses hash(str) & DICT_MASK to compute the bucket
index, where DICT_HASH is the number of buckets minus one. If all
strings have the same hash value, we always start in the same bucket
and the key has to be compared to all previous strings to find the
next empty bucket. The attack works because a LOT of strings are
compared and comparing strings is slow.

If hash(str1)&DICT_MASK == hash(str2)&DICT_MASK but
hash(str1)!=hash(str2), strings are not compared (this is a common
optimization in Python), and the so the attack would not be successful
(it would be slow, but not as slow as comparing two strings).

Victor
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Backwards incompatible sys.stdout.write() behavior in Python 3 (Was: [Python-ideas] Pythonic buffering in Py3 print())

[Ronald Oussoren]

On 17 Jan, 2012, at 11:59, anatoly techtonik wrote:
> 
> 
> If you track this more closely, you'll notice there are four issues 
> (surprises) from the user point of view:
> 1. print() buffers output on Python3
> 2. print() also buffers output on Python2, but only on Linux
> 3. there is some useless '-u' command line parameter
> (useless, because the last thing user wants is not only care about Python 
> 2/3, but also how to invoke them)
> 4. print() is not guilty - it is sys.stdout.write() that buffers output
> 
> 1-2 discussion was about idea to make new print() function behavior more 
> 'pythonic', i.e. 'user-friendly' or just KISS, which resulted in adding a 
> flush parameter
> 3 is a just a side FYI remark
> 4 doesn't relate to python-ideas anymore about fixing print() - it is about 
> the *cause* of the problem with print() UX, which is underlying 
> sys.stdout.write() behavior
> 
> I asked 4 here, because it is the more appropriate place not only to ask if 
> it can be/will be fixed, but also why. The target audience of the question 
> are developers.

All four "issues" are related to output buffering and how that is not 
user-friendly. The new issue you raise is the same as before: sys.stdout is 
line buffered when writing to a tty, which means that you have to explictly 
flush output when you want to output a partial line.  Why is this a problem for 
you? Is that something that bothers you personally or do you have data that 
suggests that this is a problem for a significant amount of (new) users?

Ronald



smime.p7s
Description: S/MIME cryptographic signature
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of the fix for the hash collision vulnerability

[Victor Stinner]

I finished my patch transforming hash(str) to a randomized hash
function, see random-8.patch attached to the issue:
http://bugs.python.org/issue13703

The remaining question is which random number generator should be used
on Windows to initialize the hash secret (CryptoGen adds an overhead
of 10%, at least when the DLL is loaded dynamically), read the issue
for the details.

I plan to commit my fix to Python 3.3 if it is accepted. Then write a
simplified version to Python 3.2 and backport it to 3.1. Then backport
the simplified fix to 2.7, and finally to 2.6.

The vulnerability is public since one month, it is maybe time to fix
it before it is widely exploited.

Victor
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of the fix for the hash collision vulnerability

[Jeremy Sanders]

Victor Stinner wrote:

> If hash(str1)&DICT_MASK == hash(str2)&DICT_MASK but
> hash(str1)!=hash(str2), strings are not compared (this is a common
> optimization in Python), and the so the attack would not be successful
> (it would be slow, but not as slow as comparing two strings).

It's a shame the hash function can't take a second salt parameter to include 
in the hash. Each dict could have its own salt, generated from a quick 
pseudo-random generator.

Jeremy



___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of the fix for the hash collision vulnerability

[Jeremy Sanders]

Jeremy Sanders wrote:

> Victor Stinner wrote:
> 
>> If hash(str1)&DICT_MASK == hash(str2)&DICT_MASK but
>> hash(str1)!=hash(str2), strings are not compared (this is a common
>> optimization in Python), and the so the attack would not be successful
>> (it would be slow, but not as slow as comparing two strings).
> 
> It's a shame the hash function can't take a second salt parameter to
> include in the hash. Each dict could have its own salt, generated from a
> quick pseudo-random generator.

Please ignore... forgot that the hashes are cached for strings!

Jeremy



___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] cpython: add str.casefold() (closes #13752)

[Éric Araujo]

Hi,


changeset:   d4669f43d05f
user:Benjamin Peterson 
date:Sat Jan 14 13:23:30 2012 -0500
summary:
  add str.casefold() (closes #13752)



diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst
--- a/Doc/library/stdtypes.rst
+++ b/Doc/library/stdtypes.rst
@@ -1002,6 +1002,14 @@
rest lowercased.


+.. method:: str.casefold()
+
+   Return a casefolded copy of the string. Casefolded strings may be 
used for
+   caseless matching. For example, ``"MASSE".casefold() == 
"maße".casefold()``.

+
+   .. versionadded:: 3.3


I think this method requires at least a link to relevant definitions
(Unicode website or Wikipedia), and at best a bit more explanation (for
example, it is not locale-dependent, even though the example above is
only meaningful for German).

Cheers
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] cpython: provide a common method to check for RETR_DATA validity, first checking the

[Éric Araujo]

Hi Giampaolo,


changeset:   53a5a5b8859d
user:Giampaolo Rodola' 
date:Mon Jan 09 17:10:10 2012 +0100
summary:
  provide a common method to check for RETR_DATA validity, first
checking the expected len and then the actual data content; this
way we get a failure on len mismatch rather than content mismatch
(which is very long and unreadable)


My trick is to convert long strings to lists (with
data.split(appropriate line ending)) and pass them to assertEqual.
Then I get more readable element-based diffs when there is a test
failure.

Another trick I use is this (for example when I don’t want to make
too much diff noise, or when I don’t want to build the list of
expected results):

  self.assertEqual(len(got), 3, got)

unittest will print the third argument on failure.

Regards
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Sphinx version for Python 2.x docs

[Sandro Tosi]

On Mon, Jan 16, 2012 at 16:42, Éric Araujo  wrote:
> Hi,
>
> Le 14/01/2012 15:31, Sandro Tosi a écrit :
>>
>> On Sat, Jan 14, 2012 at 04:24, Éric Araujo  wrote:

 Doc/glossary.rst:520: WARNING: unknown keyword: nonlocal
>>>
>>> That’s a mistake I did in cefe4f38fa0e.  This sentence should be removed.
>>
>> Do you mean revert this whole hunk:
>> [...]
>>
>> or just "The :keyword:`nonlocal` allows writing to outer scopes."?
>
>
> My proposal was to remove just that one last sentence, but the only
> other change in the diff hunk is the addition of “by default”, which is
> connected to the existence of nonlocal.  Both changes, i.e. the whole
> hunk, should be reverted (I think I’ll have time to do that today).

I've reverted it with ef1612a6a4f7

 Doc/library/stdtypes.rst:2372: WARNING: more than one target found for
 cross-reference u'next':
>>>
>>> Need to use :meth:`.next` to let Sphinx find the right target (more info
>>> on request :)
>>
>> it seems what it needed to was :meth:`next` (without the dot). The
>> current page links all 'next' in file.next() to functions.html#next,
>> and using :meth:`next` does that.
>
>
> I should have given more info, as I wanted the opposite result :)
> file.next should not link to the next function but to the file.next
> method.  Because Sphinx does not differentiate between
> meth/func/class/mod roles, :meth:`next` is not resolved to the nearest
> next method as one could expect but to the next function, so we have to
> use :meth:`~SomeClass.next` or :meth:`.next` (local ref markup) to get
> our links to methods.

I tried :meth:`.next` but got a lots of :

/home/morph/cpython/py27/Doc/library/stdtypes.rst:2372: WARNING: more
than one target found for cross-reference u'next': iterator.next,
multifile.MultiFile.next, csv.csvreader.next, dbhash.dbhash.next,
mailbox.oldmailbox.next, ttk.Treeview.next, nntplib.NNTP.next,
file.next, bsddb.bsddbobject.next, tarfile.TarFile.next,
generator.next

so I ended up with :meth:`next` but it was still wrong. I've committed
51e11b4937b7 which uses :meth:`~file.next` instead, and it works.

Cheers,
-- 
Sandro Tosi (aka morph, morpheus, matrixhasu)
My website: http://matrixhasu.altervista.org/
Me at Debian: http://wiki.debian.org/SandroTosi
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Sphinx version for Python 2.x docs

[Georg Brandl]

Am 17.01.2012 19:02, schrieb Sandro Tosi:

>> I should have given more info, as I wanted the opposite result :)
>> file.next should not link to the next function but to the file.next
>> method.  Because Sphinx does not differentiate between
>> meth/func/class/mod roles, :meth:`next` is not resolved to the nearest
>> next method as one could expect but to the next function, so we have to
>> use :meth:`~SomeClass.next` or :meth:`.next` (local ref markup) to get
>> our links to methods.
> 
> I tried :meth:`.next` but got a lots of :
> 
> /home/morph/cpython/py27/Doc/library/stdtypes.rst:2372: WARNING: more
> than one target found for cross-reference u'next': iterator.next,
> multifile.MultiFile.next, csv.csvreader.next, dbhash.dbhash.next,
> mailbox.oldmailbox.next, ttk.Treeview.next, nntplib.NNTP.next,
> file.next, bsddb.bsddbobject.next, tarfile.TarFile.next,
> generator.next
> 
> so I ended up with :meth:`next` but it was still wrong. I've committed
> 51e11b4937b7 which uses :meth:`~file.next` instead, and it works.

No need to try, just read the docs :)

`next` looks in the current (class, then module) namespaces.
`.next` looks everywhere, so the match must be unique.
So for something as common as "next", an explicit `file.next` is
required.

Georg

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Script(s) for building Python on Windows

[Martin v. Löwis]

> Are you suggesting creating vs10to9, which would be congruent to vs9to8, or
> vs9to10?

After reconsidering, I don't think I want anything like this in the tree
at this point. The code will be outdated by the time Python 3.3 is
released, as Python 3.3 will be built with a Visual Studio different
from 2008.

Regards,
Martin

P.S. Please shorten your messages. They contain too much text for me to
grasp.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of the fix for the hash collision vulnerability

[Martin v. Löwis]

> I thought that the original problem was that with N insertions in the
> dictionary, by repeatedly inserting different keys generating the same
> hash value an attacker could arrange that the cost of finding an open
> slot is O(N), and thus the cost of N insertions is O(N^2).
> 
> If so, frequent resizing could make the attacker's problem much more
> difficult, as the distribution of secondary probes should change with
> each resize.

Not sure what you mean by "distribution of secondary probes".

Let H be the initial hash value, and let MASK be the current size
of the dictionary. Then I(n), the sequence of dictionary indices
being probed, is computed as

   I(0) = H & MASK
   PERTURB(0) = H
   I(n+1) = (5*I(n) + 1 + PERTURB(n)) & MASK
   PERTURN(n+1) = PERTURB(n) >> 5

So if two objects O1 and O2 have the same hash value H, the sequence of
probed indices is the same for any MASK value. It will be a different
sequence, yes, but they will still collide on each and every slot.

This is the very nature of open addressing. If it *wouldn't* try all
indices in the probe sequence, it may not be possible to perform
the lookup for a key correctly.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 407: New release cycle and introducing long-term support versions

[Antoine Pitrou]

Hello,

We would like to propose the following PEP to change (C)Python's release
cycle. Discussion is welcome, especially from people involved in the
release process, and maintainers from third-party distributions of
Python.

Regards

Antoine.


PEP: 407
Title: New release cycle and introducing long-term support versions
Version: $Revision$
Last-Modified: $Date$
Author: Antoine Pitrou ,
Georg Brandl ,
Barry Warsaw 
Status: Draft
Type: Process
Content-Type: text/x-rst
Created: 2012-01-12
Post-History:
Resolution: TBD


Abstract


Finding a release cycle for an open-source project is a delicate
exercise in managing mutually contradicting constraints: developer
manpower, availability of release management volunteers, ease of
maintenance for users and third-party packagers, quick availability of
new features (and behavioural changes), availability of bug fixes
without pulling in new features or behavioural changes.

The current release cycle errs on the conservative side.  It is
adequate for people who value stability over reactivity.  This PEP is
an attempt to keep the stability that has become a Python trademark,
while offering a more fluid release of features, by introducing the
notion of long-term support versions.


Scope
=

This PEP doesn't try to change the maintenance period or release
scheme for the 2.7 branch.  Only 3.x versions are considered.


Proposal


Under the proposed scheme, there would be two kinds of feature
versions (sometimes dubbed "minor versions", for example 3.2 or 3.3):
normal feature versions and long-term support (LTS) versions.

Normal feature versions would get either zero or at most one bugfix
release; the latter only if needed to fix critical issues.  Security
fix handling for these branches needs to be decided.

LTS versions would get regular bugfix releases until the next LTS
version is out.  They then would go into security fixes mode, up to a
termination date at the release manager's discretion.

Periodicity
---

A new feature version would be released every X months.  We
tentatively propose X = 6 months.

LTS versions would be one out of N feature versions.  We tentatively
propose N = 4.

With these figures, a new LTS version would be out every 24 months,
and remain supported until the next LTS version 24 months later.  This
is mildly similar to today's 18 months bugfix cycle for every feature
version.

Pre-release versions


More frequent feature releases imply a smaller number of disruptive
changes per release.  Therefore, the number of pre-release builds
(alphas and betas) can be brought down considerably.  Two alpha builds
and a single beta build would probably be enough in the regular case.
The number of release candidates depends, as usual, on the number of
last-minute fixes before final release.


Effects
===

Effect on development cycle
---

More feature releases might mean more stress on the development and
release management teams.  This is quantitatively alleviated by the
smaller number of pre-release versions; and qualitatively by the
lesser amount of disruptive changes (meaning less potential for
breakage).  The shorter feature freeze period (after the first beta
build until the final release) is easier to accept.  The rush for
adding features just before feature freeze should also be much
smaller.

Effect on bugfix cycle
--

The effect on fixing bugs should be minimal with the proposed figures.
The same number of branches would be simultaneously open for regular
maintenance (two until 2.x is terminated, then one).

Effect on workflow
--

The workflow for new features would be the same: developers would only
commit them on the ``default`` branch.

The workflow for bug fixes would be slightly updated: developers would
commit bug fixes to the current LTS branch (for example ``3.3``) and
then merge them into ``default``.

If some critical fixes are needed to a non-LTS version, they can be
grafted from the current LTS branch to the non-LTS branch, just like
fixes are ported from 3.x to 2.7 today.

Effect on the community
---

People who value stability can just synchronize on the LTS releases
which, with the proposed figures, would give a similar support cycle
(both in duration and in stability).

People who value reactivity and access to new features (without taking
the risk to install alpha versions or Mercurial snapshots) would get
much more value from the new release cycle than currently.

People who want to contribute new features or improvements would be
more motivated to do so, knowing that their contributions will be more
quickly available to normal users.  Also, a smaller feature freeze
period makes it less cumbersome to interact with contributors of
features.


Discussion
==

These are open issues that should be worked out during discussion:

* Decide on X (months between feature releases) and N (fe

[Python-Dev] Switching to Visual Studio 2010

[Martin v. Löwis]

It seems a number of people are interested that the Python trunk
switches to Visual Studio 2010 *now*. I've been hesitant to agree
to such a change, as I still hope that Python can skip over VS 2010
(a.k.a.  VS 10), and go straight to VS 11.

However, I just learned that VS 11 supposed ready VS 10 project files
just fine, with no need of conversion.

So I'd be willing to agree to converting the Python trunk now. It
will surely cause all kinds of issues, as any switching of Visual Studio
releases has caused in the past.

Since a number of people have already started with such a project,
I'd like to ask for a volunteer who will lead this project. You
get the honor to commit the changes, and you will be in charge if
something breaks, hopefully finding out solutions in a timely manner
(not necessarily implementing the solutions yourself).

Any volunteers?

Regards,
Martin

P.S. Here is my personal list of requirements and non-requirements:
- must continue to live in PCbuild, and must replace the VS 9
  project files "for good"
- may or may not support automatic conversion to VS 9. If it turns
  out that conversion to old project files is not feasible, we could
  either decide to maintain old project files manually (in PC/VS9),
  or just give up on maintaining build support for old VS releases.
- must generate binaries that run on Windows XP
- must support x86 and AMD64 builds
- must support debug and no-debug builds
- must support PGO builds
- must support buildbot
- must support building all extensions that we currently build
- may break existing buildbot installations until they upgrade to
  a new VS release
- must support PCbuild/rt.bat
- should support Tools/msi. If it doesn't, I'll look into it.
- must nearly pass the test suite (i.e. number of additional failures
  due to VS 2010 should be "small")
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Switching to Visual Studio 2010

[Brian Curtin]

On Tue, Jan 17, 2012 at 14:43, "Martin v. Löwis"  wrote:
> It seems a number of people are interested that the Python trunk
> switches to Visual Studio 2010 *now*. I've been hesitant to agree
> to such a change, as I still hope that Python can skip over VS 2010
> (a.k.a.  VS 10), and go straight to VS 11.
>
> However, I just learned that VS 11 supposed ready VS 10 project files
> just fine, with no need of conversion.
>
> So I'd be willing to agree to converting the Python trunk now. It
> will surely cause all kinds of issues, as any switching of Visual Studio
> releases has caused in the past.
>
> Since a number of people have already started with such a project,
> I'd like to ask for a volunteer who will lead this project. You
> get the honor to commit the changes, and you will be in charge if
> something breaks, hopefully finding out solutions in a timely manner
> (not necessarily implementing the solutions yourself).
>
> Any volunteers?

I previously completed the port at my old company (but could not
release it), and I have a good bit of it completed for us at
http://hg.python.org/sandbox/vs2010port/. That repo is a little bit
behind 'default' but updating it shouldn't pose any problems.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of the fix for the hash collision vulnerability

[Martin v. Löwis]

> I plan to commit my fix to Python 3.3 if it is accepted. Then write a
> simplified version to Python 3.2 and backport it to 3.1.

I'm opposed to any change to the hash values of strings in maintenance
releases, so I guess I'm opposed to your patch in principle.

See my next message for an alternative proposal.

> The vulnerability is public since one month, it is maybe time to fix
> it before it is widely exploited.

I don't think there is any urgency. The vulnerability has been known for
more than five years now. From creating a release to the point where
the change actually arrives at end users, many months will pass.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Hashing proposal: change only string-only dicts

[Martin v. Löwis]

I'd like to propose a different approach to seeding the string hashes:
only do so for dictionaries involving only strings, and leave the
tp_hash slot of strings unchanged.

Each string would get two hashes: the "public" hash, which is constant
across runs and bugfix releases, and the dict-hash, which is only used
by the dictionary implementation, and only if all keys to the dict are
strings. In order to allow caching of the hash, all dicts should use
the same hash (if caching wasn't necessary, each dict could use its own
seed).

There are several variants of that approach wrt. caching of the hash
1. add an additional field to all string objects, to cache the second
   hash value.
   a) variant: in 3.3, drop the extra field, and declare that hashes
   may change across runs
2. only cache the dict-hash, recomputing the public hash each time
3. on a per-string choice, cache either the dict-hash or the public
   hash, depending on which one gets computed first, and recompute
   the other one every time it's needed.

As you can see, 1 vs. 2/3 is a classical time-space-tradeoff.

What do you think?

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Switching to Visual Studio 2010

[Martin v. Löwis]

> I previously completed the port at my old company (but could not
> release it), and I have a good bit of it completed for us at
> http://hg.python.org/sandbox/vs2010port/. That repo is a little bit
> behind 'default' but updating it shouldn't pose any problems.

So: do you agree that we switch? Do you volunteer to drive the change?

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python as a Metro-style App

[Martin v. Löwis]

> Just wondering, do Metro apps define UNDER_CE or _WIN32_WCE? The point
> is that the old ANSI functions (CreateFileA etc) have been removed from
> the embedded MS Windows CE long ago, too, and MS Windows Mobile used to
> be a custom CE variant or at least strongly related. In any case, it
> could help using the existing (incomplete) CE port as base for Metro.

I have now completed building Python as a Metro DLL; the WinRT
restrictions are fairly limited (code-wise, not so in impact).

They are quite different from the CE restrictions. For example,
CreateSemaphore is not available on WinRT, you have to use
CreateSemaphoreExW (which is new in Windows Vista). No traces of
the CE API can be seen in the restrictions, and the separation
is done in a different manner (WINAPI_FAMILY==2).

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Switching to Visual Studio 2010

[Brian Curtin]

On Tue, Jan 17, 2012 at 15:01, "Martin v. Löwis"  wrote:
>> I previously completed the port at my old company (but could not
>> release it), and I have a good bit of it completed for us at
>> http://hg.python.org/sandbox/vs2010port/. That repo is a little bit
>> behind 'default' but updating it shouldn't pose any problems.
>
> So: do you agree that we switch? Do you volunteer to drive the change?

I do, and I'll volunteer.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Hashing proposal: change only string-only dicts

[Antoine Pitrou]

On Tue, 17 Jan 2012 21:59:28 +0100
"Martin v. Löwis"  wrote:
> I'd like to propose a different approach to seeding the string hashes:
> only do so for dictionaries involving only strings, and leave the
> tp_hash slot of strings unchanged.

I think Python 3 would be better with a clean fix (all hashes
randomized).
Now for Python 2... The problem with this idea is that it only
addresses str dicts. Unicode dicts, and any other dicts, are left
vulnerable. Unicode dicts are quite likely in Web
frameworks/applications and other places which have well-thought text
semantics.

That said, here's a suggestion to squeeze those bits:

> 1. add an additional field to all string objects, to cache the second
>hash value.
>a) variant: in 3.3, drop the extra field, and declare that hashes
>may change across runs

In 2.7, a string object has the following fields:

long ob_shash;
int ob_sstate;

Only 2 bits are used in ob_sstate, meaning 30 are left. These 30 bits
could cache a "hash perturbation" computed from the string and the
random bits:

- hash() would use ob_shash
- dict_lookup() would use ((ob_shash * 103) ^ (ob_sstate & ~3))

This way, you cache almost all computations, adding only a computation
and a couple logical ops when looking up a string in a dict.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Coroutines and PEP 380

[Mark Shannon]

Hi all.

Lets start controversially: I don't like PEP 380, I think it's a kludge.

I think that CPython should have proper coroutines, rather than add more 
bits and pieces to generators in an attempt to make them more like 
coroutines.


I have mentioned this before, but this time I have done something about 
it :)


I have a working, portable, (asymmetric) coroutine implementation here:

https://bitbucket.org/markshannon/hotpy_coroutines

Its all standard C, no messing with the C stack, just using standard 
techniques to convert recursion to iteration
(in the VM not at the Python level) and a revised internal calling 
convention to make CPython stackless:


https://bitbucket.org/markshannon/hotpy_stackless

Then I've added a Coroutine class and fiddled with the implementation of 
YIELD_VALUE to support it.


I think the stackless implementation is pretty solid, but the
coroutine stuff needs some refinement.
I've not tested it well (it passes the test suite, but I've added no new 
tests).

It is (surprisingly) a bit faster than tip (on my machine).
There are limitations: all calls must be Python-to-Python calls,
which rules out most __xxx__ methods. It might be worth special casing 
__iter__, but I've not done that yet.


To try it out:

>>> import coroutine
To send a value to a coroutine:
>>> co.send(val)
where co is a Coroutine()
To yield a value:
>>> coroutine.co_yield(val)
send() is a method, co_yield is a function.

Here's a little program to demonstrate:

import coroutine

class Node:
def __init__(self, l, item, r):
self.l = l
self.item = item
self.r = r

def make_tree(n):
if n == 0:
return Node(None, n, None)
else:
return Node(make_tree(n-1), n, make_tree(n-1))

def walk_tree(t, f):
if t is not None:
walk_tree(t.l, f)
f(t)
walk_tree(t.r, f)

def yielder(t):
coroutine.co_yield(t.item)

def tree_yielder(t):
walk_tree(t, yielder)

co = coroutine.Coroutine(tree_yielder, (make_tree(2),))

while True:
print(co.send(None))

Which will output:

0
1
0
2
0
1
0
None
Traceback (most recent call last):
  File "co_demo.py", line 30, in 
print(co.send(None))
TypeError: can't send to a halted coroutine


Cheers,
Mark.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Hashing proposal: change only string-only dicts

[Victor Stinner]

2012/1/17 "Martin v. Löwis" :
> I'd like to propose a different approach to seeding the string hashes:
> only do so for dictionaries involving only strings, and leave the
> tp_hash slot of strings unchanged.

The real problem is in dict (or any structure using an hash table), so
if it is possible, it would also prefer to fix the problem directly in
dict.

> There are several variants of that approach wrt. caching of the hash
> 1. add an additional field to all string objects, to cache the second
>   hash value.
>   a) variant: in 3.3, drop the extra field, and declare that hashes
>   may change across runs
> 2. only cache the dict-hash, recomputing the public hash each time
> 3. on a per-string choice, cache either the dict-hash or the public
>   hash, depending on which one gets computed first, and recompute
>   the other one every time it's needed.

There is a simpler solution:

bucket_index = (hash(str) ^ secret) & DICT_MASK.

Remark: set must also be fixed.

Victor
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Hashing proposal: change only string-only dicts

[Victor Stinner]

> There is a simpler solution:
>
> bucket_index = (hash(str) ^ secret) & DICT_MASK.

Oops, hash^secret doesn't add any security.

Victor
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 407: New release cycle and introducing long-term support versions

[Eric Snow]

On Tue, Jan 17, 2012 at 1:34 PM, Antoine Pitrou  wrote:
> Under the proposed scheme, there would be two kinds of feature
> versions (sometimes dubbed "minor versions", for example 3.2 or 3.3):
> normal feature versions and long-term support (LTS) versions.
...
> A new feature version would be released every X months.  We
> tentatively propose X = 6 months.
>
> LTS versions would be one out of N feature versions.  We tentatively
> propose N = 4.

It sounds like every six months we would get a new feature version,
with every fourth one an LTS release.  That sounds great, but, unless
I've misunderstood, there has been a strong desire to keep that number
to one digit.  It doesn't matter to me all that much.  However, if
there is such a limit, implied or explicit, it should be mentioned and
factor into the PEP.

That aside, +1.

-eric
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Hashing proposal: change only string-only dicts

[Victor Stinner]

> Each string would get two hashes: the "public" hash, which is constant
> across runs and bugfix releases, and the dict-hash, which is only used
> by the dictionary implementation, and only if all keys to the dict are
> strings.

The distinction between secret (private, secure) and "public" hash
(deterministic) is not clear to me.

Example: collections.UserDict implements __hash__() using
hash(self.data). Should it use the public or the private hash?
collections.abc.Set computes its hash using hash(x) of each item. Same
question.

If we need to use the secret hash, it should be exposed in Python.
Which function/method would be used? I suppose that we cannot add
anything to stable releases like 2.7.

Victor
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 407: New release cycle and introducing long-term support versions

[Matt Joiner]

If minor/feature releases are introducing breaking changes perhaps it's
time to adopt accelerated major versioning schedule. For instance there are
breaking ABI changes between 3.0/3.1, and 3.2, and while acceptable for the
early adoption state of Python 3, such changes should normally be reserved
for major versions.

If every 4th or so feature release is sufficiently different to be worth of
an LTS, consider this a major release albeit with smaller beading changes
than Python 3.

Aside from this, given the radical features of 3.3, and the upcoming Ubuntu
12.04 LTS, I would recommend adopting 2.7 and 3.2 as the first LTSs, to be
reviewed 2 years hence should this go ahead.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Coroutines and PEP 380

[Matt Joiner]

Just to clarify, this differs in functionality from enhanced generators by
allowing you to yield from an arbitrary call depth rather than having to
"yield from" through a chain of calling generators? Furthermore there's no
syntactical change except to the bottommost frame doing a co_yield? Does
this capture the major differences?
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 407: New release cycle and introducing long-term support versions

[Antoine Pitrou]

Hello,

On Wed, 18 Jan 2012 10:04:19 +1100
Matt Joiner  wrote:
> If minor/feature releases are introducing breaking changes perhaps it's
> time to adopt accelerated major versioning schedule.

The PEP doesn't propose to accelerate compatibility breakage. So I don't
think a change in numbering is required.

> For instance there are
> breaking ABI changes between 3.0/3.1, and 3.2, and while acceptable for the
> early adoption state of Python 3, such changes should normally be reserved
> for major versions.

Which "breaking ABI changes" are you thinking about? Python doesn't
guarantee any A*B*I (as opposed to API), unless you use Py_LIMITED_API
which was introduced in 3.2.

Regards

Antoine.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of the fix for the hash collision vulnerability

[Victor Stinner]

>> I plan to commit my fix to Python 3.3 if it is accepted. Then write a
>> simplified version to Python 3.2 and backport it to 3.1.
>
> I'm opposed to any change to the hash values of strings in maintenance
> releases, so I guess I'm opposed to your patch in principle.

If randomized hash cannot be turned on by default, an alternative is
to switch them off by default, and add an option (command line option,
environment variable, etc.) to enable it.

>> The vulnerability is public since one month, it is maybe time to fix
>> it before it is widely exploited.
>
> I don't think there is any urgency. The vulnerability has been known for
> more than five years now. From creating a release to the point where
> the change actually arrives at end users, many months will pass.

In 2003, Python was not seen as vulnerable. Maybe because the hash
function is different than Perl hash function, or because nobody tried
to generate collisions. Today it is clear that Python is vulnerable
(64 bits version is also affected), and it's really fast to generate
collisions using the right algorithm.

Why is it so long to fix the vulnerability in Python, whereas it was
fixed quickly in Ruby? (they chose to use a randomized hash)

Victor
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 407: New release cycle and introducing long-term support versions

[Terry Reedy]

On 1/17/2012 3:34 PM, Antoine Pitrou wrote:


Hello,

We would like to propose the following PEP to change (C)Python's release
cycle. Discussion is welcome, especially from people involved in the
release process, and maintainers from third-party distributions of
Python.

Regards

Antoine.


PEP: 407
Title: New release cycle and introducing long-term support version


To me, as I understand the proposal, the title is wrong. Our current 
feather releases already are long-term support versions. They get bugfix 
releases at close to 6 month intervals for 1 1/2 -2 years and security 
fixes for 3 years. The only change here is that you propose, for 
instance, a fixed 6-month interval and 2 year period.


As I read this, you propose to introduce a new short-term (interim, 
preview) feature release along with each bugfix release. Each would have 
all the bugfixes plus a preview of the new features expected to be in 
the next long-term release. (I know, this is not exactly how you spun it.)


There has been discussion on python-ideas about whether new features are 
or can be considered experimental, or whether there should be an 
'experimental' package. An argument against is that long-term production 
releases should not have experimental features that might go away or 
have their apis changed.


If the short-term, non-production, interim feature releases were called 
preview releases, then some or all of the new features could be labelled 
experimental and subject to change. It might actually be good to have 
major new features tested in at least one preview release before being 
frozen. Maybe then more of the initial bugs would be found and repaired 
*before* their initial appearance in a long-term release. (All of this 
is not to say that experimental features should be casually changed or 
reverted without good reason.)


One problem, at least on Windows, is that short-term releases would 
almost never have compiled binaries for 3rd-party libraries. It already 
takes awhile for them to appear for the current long-term releases. On 
the other hand,  library authors might be more inclined to test new 
features, a few at a time, if part of tested preview releases, than if 
just in the repository. So the result *might* be quicker library updates 
after each long-term release.


--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Coroutines and PEP 380

[Ethan Furman]

Mark Shannon wrote:
I think that CPython should have proper coroutines, rather than add more 
bits and pieces to generators in an attempt to make them more like 
coroutines.


I have mentioned this before, but this time I have done something about 
it :)


I have a working, portable, (asymmetric) coroutine implementation here:

https://bitbucket.org/markshannon/hotpy_coroutines


As a user, this sounds cool!

~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Coroutines and PEP 380

[Glyph]

On Jan 17, 2012, at 5:03 PM, Mark Shannon wrote:

> Lets start controversially: I don't like PEP 380, I think it's a kludge.

Too late; it's already accepted.  There's not much point in making 
controversial statements about it now.

> I think that CPython should have proper coroutines, rather than add more bits 
> and pieces to generators in an attempt to make them more like coroutines.


By "proper" coroutines, you mean implicit coroutines (cooperative threads) 
rather than explicit coroutines (cooperative generators).  Python has been 
going in the "explicit" direction on this question for a long time.  (And, in 
my opinion, this is the right direction to go, but that's not really relevant 
here.)

I think this discussion would be more suitable for python-ideas though, since 
you have a long row to hoe here.  There's already a PEP - 
http://www.python.org/dev/peps/pep-0219/ - apparently deferred and not 
rejected, which you may want to revisit.

There are several libraries which can give you cooperative threading already; I 
assume you're already aware of greenlet and stackless, but I didn't see what 
advantages your proposed implementation provides over those.  I would guess 
that one of the first things you should address on python-ideas is why adopting 
your implementation would be a better idea than just bundling one of those with 
the standard library :).

-glyph

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 407: New release cycle and introducing long-term support versions

[Antoine Pitrou]

On Tue, 17 Jan 2012 18:29:11 -0500
Terry Reedy  wrote:
> 
> To me, as I understand the proposal, the title is wrong. Our current 
> feather releases already are long-term support versions. They get bugfix 
> releases at close to 6 month intervals for 1 1/2 -2 years and security 
> fixes for 3 years. The only change here is that you propose, for 
> instance, a fixed 6-month interval and 2 year period.
> 
> As I read this, you propose to introduce a new short-term (interim, 
> preview) feature release along with each bugfix release. Each would have 
> all the bugfixes plus a preview of the new features expected to be in 
> the next long-term release. (I know, this is not exactly how you spun it.)

Well, "spinning" is important here. We are not proposing any "preview"
releases. These would have the same issue as alphas or betas: nobody
wants to install them where they could disrupt working applications and
libraries.

What we are proposing are first-class releases that are as robust as
any other (and usable in production). It's really about making feature
releases more frequent, not making previews available during
development.

I agree "long-term" could be misleading as their support duration is
not significantly longer than current feature releases. I chose this
term because it is quite well-known and well-understood, but we could
pick something else ("extended support", "2-year support", etc.).

> There has been discussion on python-ideas about whether new features are 
> or can be considered experimental, or whether there should be an 
> 'experimental' package. An argument against is that long-term production 
> releases should not have experimental features that might go away or 
> have their apis changed.

That's orthogonal to this PEP.
(that said, more frequent feature releases are also a benefit for the
__preview__ proposal, since we could be more reactive changing APIs in
that namespace)

> One problem, at least on Windows, is that short-term releases would 
> almost never have compiled binaries for 3rd-party libraries.

That's a good point, although Py_LIMITED_API will hopefully make things
better in the middle term.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 407: New release cycle and introducing long-term support versions

[Ezio Melotti]

Hi,

On 17/01/2012 22.34, Antoine Pitrou wrote:

[...]

Proposal


Under the proposed scheme, there would be two kinds of feature
versions (sometimes dubbed "minor versions", for example 3.2 or 3.3):
normal feature versions and long-term support (LTS) versions.

Normal feature versions would get either zero or at most one bugfix
release; the latter only if needed to fix critical issues.  Security
fix handling for these branches needs to be decided.


If non-LTS releases won't get bug fixes, a bug that is fixed in 3.3.x 
might not be fixed in 3.4, unless the bug fixes releases are 
synchronized with the new feature releases (see below).



LTS versions would get regular bugfix releases until the next LTS
version is out.  They then would go into security fixes mode, up to a
termination date at the release manager's discretion.

Periodicity
---

A new feature version would be released every X months.  We
tentatively propose X = 6 months.

LTS versions would be one out of N feature versions.  We tentatively
propose N = 4.


If LTS bug fixes releases and feature releases are synchronized, we will 
have something like:


3.3
3.3.1 / 3.4
3.3.2 / 3.5
3.3.3 / 3.6
3.7
3.7.1 / 3.8
...

so every new feature release will have all the bug fixes of the current 
LTS release, plus new features.


With this scheme we will soon run out of 1-digit numbers though.
Currently we already have a 3.x release every ~18 months, so if we keep 
doing that (just every 24 months instead of 18) and introduce the 
feature releases in between under a different versioning scheme, we 
might avoid the problem.


This means:
3.1
... 18 months, N bug fix releases...
3.2
... 18 months, N bug fix releases ...
3.3 LTS
... 24 months, 3 bug fix releases, 3 feature releases ...
3.4 LTS
... 24 months, 3 bug fix releases, 3 feature releases ...
3.5 LTS

In this way we solve the numbering problem and keep a familiar scheme 
(all the 3.x will be LTS and will be released as the same pace as 
before, no need to mark some 3.x as LTS).  OTOH this will make the 
feature releases less "noticeable" and people might just ignore them and 
stick with the LTS releases.  Also we would need to define a versioning 
convention for the feature releases.



[...]

Effect on bugfix cycle
--

The effect on fixing bugs should be minimal with the proposed figures.
The same number of branches would be simultaneously open for regular
maintenance (two until 2.x is terminated, then one).


Wouldn't it still be two?
Bug fixes will go to the last LTS and on default, features only on default.


Effect on workflow
--

The workflow for new features would be the same: developers would only
commit them on the ``default`` branch.

The workflow for bug fixes would be slightly updated: developers would
commit bug fixes to the current LTS branch (for example ``3.3``) and
then merge them into ``default``.


So here the difference is that instead of committing on the previous 
release (what currently is 3.2), we commit it to the previous LTS 
release, ignoring the ones between that and default.



If some critical fixes are needed to a non-LTS version, they can be
grafted from the current LTS branch to the non-LTS branch, just like
fixes are ported from 3.x to 2.7 today.

Effect on the community
---

People who value stability can just synchronize on the LTS releases
which, with the proposed figures, would give a similar support cycle
(both in duration and in stability).


That's why I proposed to keep the same versioning scheme for these 
releases, and have a different numbering for the feature releases.



[...]

Discussion
==

These are open issues that should be worked out during discussion:

* Decide on X (months between feature releases) and N (feature releases
   per LTS release) as defined above.


This doesn't necessarily have to be fixed, especially if we don't change 
the versioning scheme (so we don't need to know that we have a LTS 
release every N releases).



* For given values of X and N, is the no-bugfix-releases policy for
   non-LTS versions feasible?


If LTS bug fix releases and feature releases are synchronized it should 
be feasible.



* Restrict new syntax and similar changes (i.e. everything that was
   prohibited by PEP 3003) to LTS versions?


(I was reading this the other way around, maybe rephrase it to "Allow 
new syntax and similar changes only in LTS versions")



* What is the effect on packagers such as Linux distributions?


* What is the effect on PyPy/Jython/IronPython?  Can they just skip the 
feature releases and focus on the LTS ones?



* How will release version numbers or other identifying and marketing
   material make it clear to users which versions are normal feature
   releases and which are LTS releases?  How do we manage user
   expectations?


This is not an issue with the scheme I proposed.


A community poll or survey to collect opinions from the greater Pyth

Re: [Python-Dev] Backwards incompatible sys.stdout.write() behavior in Python 3 (Was: [Python-ideas] Pythonic buffering in Py3 print())

[Terry Reedy]

On 1/17/2012 5:59 AM, anatoly techtonik wrote:


1. print() buffers output on Python3
2. print() also buffers output on Python2, but only on Linux


No, print() does not buffer output. It merely sends it to a file.


4. print() is not guilty - it is sys.stdout.write() that buffers output


Oh, you already know that 1&2 are false.

So is 4, if interpreted as saying that sys.stdout.write() *will* buffer 
output. sys.stdout can be *any* file-like object. Its .write method 
*may* buffer output, or it *may not*. With IDLE, it does not. We have 
been over this before. At your instigation, the doc has been changed to 
make this clearer. At your request, a new feature has been added to 
force flushing. By most people's standards, you won.


--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 407: New release cycle and introducing long-term support versions

[Jeff Hardy]

On Tue, Jan 17, 2012 at 3:50 PM, Ezio Melotti  wrote:
> * What is the effect on PyPy/Jython/IronPython?  Can they just skip the
> feature releases and focus on the LTS ones?

At least for IronPython it's unlikely we'd be able track the feature
releases. We're still trying to catch up as it is.

Honestly, I don't see the advantages of this. Are there really enough
new features planned that Python needs a full release more than every
18 months?

- Jeff
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Hashing proposal: change only string-only dicts

[martin]

Zitat von Victor Stinner :


Each string would get two hashes: the "public" hash, which is constant
across runs and bugfix releases, and the dict-hash, which is only used
by the dictionary implementation, and only if all keys to the dict are
strings.


The distinction between secret (private, secure) and "public" hash
(deterministic) is not clear to me.


It's not about privacy or security. It's about compatibility. The
dict-hash is only used in the dict implementation, and never exposed,
leaving the tp_hash unmodified.


Example: collections.UserDict implements __hash__() using
hash(self.data).


Are you sure? I only see that used for UserString, not UserDict.


collections.abc.Set computes its hash using hash(x) of each item. Same
question.


The hash of the Set should most certainly use the element's tp_hash.
That *is* the hash of the objects, and it may collide for strings
just fine due to the vulnerability.


If we need to use the secret hash, it should be exposed in Python.


It's not secret, just specific. I don't mind it being exposed. However,
that would be a new feature, which cannot be added in a security fix
or bug fix release.


Which function/method would be used? I suppose that we cannot add
anything to stable releases like 2.7.


Right. Nor do I see any need to expose it. It fixes the vulnerability
just fine without being exposed.

Regards,
Martin

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of the fix for the hash collision vulnerability

[martin]

If randomized hash cannot be turned on by default, an alternative is
to switch them off by default, and add an option (command line option,
environment variable, etc.) to enable it.


That won't really fix the problem. If people install a new release because
it fixes a vulnerability, it better does so.


In 2003, Python was not seen as vulnerable. Maybe because the hash
function is different than Perl hash function, or because nobody tried
to generate collisions. Today it is clear that Python is vulnerable
(64 bits version is also affected), and it's really fast to generate
collisions using the right algorithm.


There is the common vulnerability to the threat of confusing threats
with vulnerabilities [1]. Python was vulnerable all the time, and nobody
claimed otherwise. It's just that nobody saw it as a threat. I still
don't see it as a practical threat, as there are many ways that people
use in practice to protect against this threat already. But I understand
that others feel threatened now.


Why is it so long to fix the vulnerability in Python, whereas it was
fixed quickly in Ruby? (they chose to use a randomized hash)


Because the risk of breakage for Python is much higher than it is for Ruby.

Regards,
Martin

[1] http://jps.anl.gov/Volume4_iss2/Paper3-RGJohnston.pdf

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] cpython: Refactored logging rotating handlers for improved flexibility.

[Éric Araujo]

Hi,

> changeset:   57295c4d81ac
> user:Vinay Sajip 
> date:Wed Jan 04 12:02:26 2012 +
> summary:
>   Refactored logging rotating handlers for improved flexibility.

> diff --git a/Doc/howto/logging-cookbook.rst b/Doc/howto/logging-cookbook.rst
> --- a/Doc/howto/logging-cookbook.rst
> +++ b/Doc/howto/logging-cookbook.rst
> [snip]
> +These are not “true” .gz files, as they are bare compressed data, with no
> +“container” such as you’d find in an actual gzip file. This snippet is just
> +for illustration purposes.

I believe using the right characters for quote marks will upset Latex
and thus PDF generation, so the docs use ASCII straight quote marks.

> diff --git a/Doc/library/logging.handlers.rst 
> b/Doc/library/logging.handlers.rst
> --- a/Doc/library/logging.handlers.rst
> +++ b/Doc/library/logging.handlers.rst
> [snip]
> +   .. method:: BaseRotatingHandler.rotation_filename(default_name)
> +
> +  Modify the filename of a log file when rotating.
> +
> +  This is provided so that a custom filename can be provided.
> +
> +  The default implementation calls the 'namer' attribute of the handler,
> +  if it's callable, passing the default name to it. If the attribute 
> isn't
> +  callable (the default is `None`), the name is returned unchanged.

Should be ``None``.

Regards
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Hashing proposal: change only string-only dicts

[Martin v. Löwis]

Am 17.01.2012 22:26, schrieb Antoine Pitrou:
> On Tue, 17 Jan 2012 21:59:28 +0100
> "Martin v. Löwis"  wrote:
>> I'd like to propose a different approach to seeding the string hashes:
>> only do so for dictionaries involving only strings, and leave the
>> tp_hash slot of strings unchanged.
> 
> I think Python 3 would be better with a clean fix (all hashes
> randomized).
> Now for Python 2... The problem with this idea is that it only
> addresses str dicts. Unicode dicts, and any other dicts, are left
> vulnerable.

No, you misunderstood. I meant to propose that this applies to both
kinds of string (unicode and byte strings); for 2.x also dictionaries
including a mix of them.

> Only 2 bits are used in ob_sstate, meaning 30 are left. These 30 bits
> could cache a "hash perturbation" computed from the string and the
> random bits:
> 
> - hash() would use ob_shash
> - dict_lookup() would use ((ob_shash * 103) ^ (ob_sstate & ~3))
> 
> This way, you cache almost all computations, adding only a computation
> and a couple logical ops when looking up a string in a dict.

That's a good idea. For Unicode, it might be best to add another slot
into the object, even though this increases the object size.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 407: New release cycle and introducing long-term support versions

[Stephen J. Turnbull]

Executive summary:

My take is "show us the additional resources, and don't be stingy!"
Sorry, Antoine, I agree with your goals, but I think you are too
optimistic about the positive effects and way too optimistic about the
costs.

Antoine Pitrou writes:

 > Finding a release cycle for an open-source project is a delicate
 > exercise in managing mutually contradicting constraints: developer
 > manpower,

This increases the demand for developer manpower somewhat.

 > availability of release management volunteers,

Dramatic increase here.  It may look like RM is not so demanding --
run a few scripts to put out the alphas/betas/releases.  But the RM
needs to stay on top of breaking news, make decisions.  That takes
time, interrupts other work, etc.

 > ease of maintenance for users and third-party packagers,

Dunno about users, but 3rd party packagers will also have more work to
do, or will have to tell their users "we only promise compatibility
with LTS releases."

 > quick availability of new features (and behavioural changes),

These are already *available*, just not *tested*.

Since testing is the bottleneck on what users consider to be
"available for me", you cannot decrease the amount of testing (alpha,
beta releases) by anywhere near the amount you're increasing
frequency, or you're just producing "as is" snapshots.  Percentage of
time in feature freeze goes way up, features get introduced all at
once just before the next release, schedule slippage is inevitable on
some releases.

 > availability of bug fixes without pulling in new features or
 > behavioural changes.

Sounds like a slight further increase in demand for RM, and as
described a dramatic decrease in the bugfixing for throw-away releases.

 > The current release cycle errs on the conservative side.

What evidence do you have for that, besides people who aren't RMs
wishing that somebody else would do more RM work?

 > More feature releases might mean more stress on the development and
 > release management teams.  This is quantitatively alleviated by the
 > smaller number of pre-release versions; and qualitatively by the
 > lesser amount of disruptive changes (meaning less potential for
 > breakage).

Way optimistic IMO (theoretical, admitted, but I do release management
for a less well-organized project, and I teach in a business school,
FWIW).

 > The shorter feature freeze period (after the first beta build until
 > the final release) is easier to accept.

But you need to look at total time in feature freeze over the LTS
cycle, not just before each throw-away release.

 > The rush for adding features just before feature freeze should also
 > be much smaller.

This doesn't depend on the length of time in feature freeze per
release, it depends on the fraction of time in feature freeze over the
cycle.  Given your quality goals, this will go way up.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 407: New release cycle and introducing long-term support versions

[Terry Reedy]

On 1/17/2012 6:42 PM, Antoine Pitrou wrote:

On Tue, 17 Jan 2012 18:29:11 -0500
Terry Reedy  wrote:


To me, as I understand the proposal, the title is wrong. Our current
feather releases already are long-term support versions. They get bugfix
releases at close to 6 month intervals for 1 1/2 -2 years and security
fixes for 3 years. The only change here is that you propose, for
instance, a fixed 6-month interval and 2 year period.

As I read this, you propose to introduce a new short-term (interim,
preview) feature release along with each bugfix release. Each would have
all the bugfixes plus a preview of the new features expected to be in
the next long-term release. (I know, this is not exactly how you spun it.)


The main point of my comment is that the new thing you are introducing 
is not long-term supported versions but short term unsupported versions.



Well, "spinning" is important here. We are not proposing any "preview"
releases. These would have the same issue as alphas or betas: nobody


I said nothing about quality. We aim to keep default in near-release 
condition and seem to be getting better. The new unicode is still 
getting polished a bit, it seems, after 3 months, but that is fairly 
unusual.



wants to install them where they could disrupt working applications and
libraries.

What we are proposing are first-class releases that are as robust as
any other (and usable in production).


But I am dubious that releases that are obsolete in 6 months and lack 
3rd party support will see much production use.



It's really about making feature releases more frequent,

> not making previews available during development.

Given the difficulty of making a complete windows build, it would be 
nice to have one made available every 6 months, regardless of how it is 
labeled.


I believe that some people will see and use good-for-6-months releases 
as previews of the new features that will be in the 'real', normal, 
bug-fix supported, long-term releases.


Every release is a snapshot of a continuous process, with some extra 
effort made to tie up some (but not all) of the loose ends.


--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Hashing proposal: change only string-only dicts

[Gregory P. Smith]

On Tue, Jan 17, 2012 at 12:59 PM, "Martin v. Löwis" wrote:

> I'd like to propose a different approach to seeding the string hashes:
> only do so for dictionaries involving only strings, and leave the
> tp_hash slot of strings unchanged.
>
> Each string would get two hashes: the "public" hash, which is constant
> across runs and bugfix releases, and the dict-hash, which is only used
> by the dictionary implementation, and only if all keys to the dict are
> strings. In order to allow caching of the hash, all dicts should use
> the same hash (if caching wasn't necessary, each dict could use its own
> seed).
>
> There are several variants of that approach wrt. caching of the hash
> 1. add an additional field to all string objects, to cache the second
>   hash value.
>

yuck, our objects are large enough as it is.


>   a) variant: in 3.3, drop the extra field, and declare that hashes
>   may change across runs
>

+1 Absolutely.  We can and should make 3.3 change hashes across runs
(behavior that can be disabled via a flag or environment variable).

I think the issue of doctests and such breaking even in 2.7 due to hash
order changes is a being overblown.  Code like that has already needs to
fix its tests at least once when they want tests to pass on on both 32-bit
and 64-bit python VMs (they have different hashes).  Do we have _any_
measure of how big a deal this will be before going too far here?

-gps
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of the fix for the hash collision vulnerability

[Gregory P. Smith]

On Tue, Jan 17, 2012 at 12:52 PM, "Martin v. Löwis" wrote:

> > I plan to commit my fix to Python 3.3 if it is accepted. Then write a
> > simplified version to Python 3.2 and backport it to 3.1.
>
> I'm opposed to any change to the hash values of strings in maintenance
> releases, so I guess I'm opposed to your patch in principle.
>

Please at least consider his patch for 3.3 onwards then.  Changing the hash
seed per interpreter instance / process is the right thing to do going
forward.

What to do on maintenance releases is a separate discussion.

-gps
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of the fix for the hash collision vulnerability

[Martin v. Löwis]

Am 18.01.2012 07:06, schrieb Gregory P. Smith:
> 
> On Tue, Jan 17, 2012 at 12:52 PM, "Martin v. Löwis"  > wrote:
> 
> > I plan to commit my fix to Python 3.3 if it is accepted. Then write a
> > simplified version to Python 3.2 and backport it to 3.1.
> 
> I'm opposed to any change to the hash values of strings in maintenance
> releases, so I guess I'm opposed to your patch in principle.
> 
> 
> Please at least consider his patch for 3.3 onwards then.  Changing the
> hash seed per interpreter instance / process is the right thing to do
> going forward.

For 3.3 onwards, I'm skeptical whether all this configuration support is
really necessary. I think a much smaller patch which leaves no choice
would be more appropriate.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Hashing proposal: change only string-only dicts

[Martin v. Löwis]

> +1 Absolutely.  We can and should make 3.3 change hashes across runs
> (behavior that can be disabled via a flag or environment variable).
> 
> I think the issue of doctests and such breaking even in 2.7 due to hash
> order changes is a being overblown.  Code like that has already needs to
> fix its tests at least once when they want tests to pass on on both
> 32-bit and 64-bit python VMs (they have different hashes).  Do we have
> _any_ measure of how big a deal this will be before going too far here?

My concern is not about breaking doctests: this proposal will also break
them. My concern is about applications that assume that hash(s) is
stable across runs, and we do have reports that it will break
applications.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 407: New release cycle and introducing long-term support versions

[Georg Brandl]

Am 18.01.2012 05:32, schrieb Terry Reedy:
> On 1/17/2012 6:42 PM, Antoine Pitrou wrote:
>> On Tue, 17 Jan 2012 18:29:11 -0500
>> Terry Reedy  wrote:
>>>
>>> To me, as I understand the proposal, the title is wrong. Our current
>>> feather releases already are long-term support versions. They get bugfix
>>> releases at close to 6 month intervals for 1 1/2 -2 years and security
>>> fixes for 3 years. The only change here is that you propose, for
>>> instance, a fixed 6-month interval and 2 year period.
>>>
>>> As I read this, you propose to introduce a new short-term (interim,
>>> preview) feature release along with each bugfix release. Each would have
>>> all the bugfixes plus a preview of the new features expected to be in
>>> the next long-term release. (I know, this is not exactly how you spun it.)
> 
> The main point of my comment is that the new thing you are introducing 
> is not long-term supported versions but short term unsupported versions.

That is really a matter of perspective.  For the proposed cycle, there
would be more regular version than LTS versions, so they are the exception
and get the special name.  (And at the same time, the name is already
established and people probably grasp instantly what it means.)

>> Well, "spinning" is important here. We are not proposing any "preview"
>> releases. These would have the same issue as alphas or betas: nobody
> 
> I said nothing about quality. We aim to keep default in near-release 
> condition and seem to be getting better. The new unicode is still 
> getting polished a bit, it seems, after 3 months, but that is fairly 
> unusual.
> 
>> wants to install them where they could disrupt working applications and
>> libraries.
>>
>> What we are proposing are first-class releases that are as robust as
>> any other (and usable in production).
> 
> But I am dubious that releases that are obsolete in 6 months and lack 
> 3rd party support will see much production use.

Whether people would use the releases is probably something that only
they can tell us -- that's why a community survey is mentioned in the
PEP.

Not sure what you mean by lacking 3rd party support.

>> It's really about making feature releases more frequent,
>  > not making previews available during development.
> 
> Given the difficulty of making a complete windows build, it would be 
> nice to have one made available every 6 months, regardless of how it is 
> labeled.
> 
> I believe that some people will see and use good-for-6-months releases 
> as previews of the new features that will be in the 'real', normal, 
> bug-fix supported, long-term releases.

Maybe they will.  That's another thing that is made clear in the PEP:
for one group of people (those preferring stability over long time),
nothing much changes, except that the release period is a little longer,
and there are these "previews" as you call them.

Georg

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 407: New release cycle and introducing long-term support versions

[Paul Moore]

On 18 January 2012 04:32, Terry Reedy  wrote:
>> It's really about making feature releases more frequent,
>
>> not making previews available during development.
>
> Given the difficulty of making a complete windows build, it would be nice to
> have one made available every 6 months, regardless of how it is labeled.
>
> I believe that some people will see and use good-for-6-months releases as
> previews of the new features that will be in the 'real', normal, bug-fix
> supported, long-term releases.

I'd love to see 6-monthly releases, including Windows binaries, and
binary builds of all packages that needed a compiler to build. Oh, and
a pony every LTS release :-)

Seriously, this proposal doesn't really acknowledge the amount of work
by other people that would be needed for a 6-month release to be
*usable* in normal cases (by Windows users, at least). It's usually
some months after a release on the current schedule that Windows
binaries have appeared for everything I use regularly.

I could easily imagine 3rd-party developers tending to only focus on
LTS releases, making the release cycle effectively *slower* for me,
rather than faster.

Paul

PS Things that might help improve this: (1) PY_LIMITED_API, and (2)
support in packaging for binary releases, including a way to force
installation of a binary release on the "wrong" version (so that
developers don't have to repackage and publish identical binaries
every 6 months).
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 407: New release cycle and introducing long-term support versions

[Georg Brandl]

Am 18.01.2012 01:24, schrieb Jeff Hardy:
> On Tue, Jan 17, 2012 at 3:50 PM, Ezio Melotti  wrote:
>> * What is the effect on PyPy/Jython/IronPython?  Can they just skip the
>> feature releases and focus on the LTS ones?
> 
> At least for IronPython it's unlikely we'd be able track the feature
> releases. We're still trying to catch up as it is.
> 
> Honestly, I don't see the advantages of this. Are there really enough
> new features planned that Python needs a full release more than every
> 18 months?

Yes, we think so.  (What is a non-full release, by the way?)

The main reason is changes in the library.  We have been getting complaints
about the standard library bitrotting for years now, and one of the main
reasons it's so hard to a) get decent code into the stdlib and b) keep it
maintained is that the release cycles are so long.  It's a tough thing for
contributors to accept that the feature you've just implemented will only
be in a stable release in 16 months.

If the stdlib does not get more reactive, it might just as well be cropped
down to a bare core, because 3rd-party libraries do everything as well and
do it before we do.  But you're right that if Python came without batteries,
the current release cycle would be fine.

(Another, more far-reaching proposal, has been to move the stdlib out of
the cpython repo and share a new repo with Jython/IronPython/PyPy.  It could
then also be released separately from the core.  But this is much more work
than the current proposal.)

Georg

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 407: New release cycle and introducing long-term support versions

[Paul Moore]

On 18 January 2012 07:46, Georg Brandl  wrote:
>> But I am dubious that releases that are obsolete in 6 months and lack
>> 3rd party support will see much production use.
>
> Whether people would use the releases is probably something that only
> they can tell us -- that's why a community survey is mentioned in the
> PEP.

The class of people who we need to consider carefully is those who
want to use the latest release, but are limited by the need for other
parties to release stuff that works with that release (usually, this
means Windows binaries of extensions, or platform vendor packaged
releases of modules/packages). For them, if the other parties focus on
LTS releases (as is possible, certainly) the release cycle became
slower, going from 18 months to 24.

> Not sure what you mean by lacking 3rd party support.

I take it as meaning that the people who release Windows binaries on
PyPI, and vendors who package up PyPI distributions in their own
distribution format. Lacking support in the sense that these people
might well decide that a 6 month cycle is too fast (too much work) and
explicitly decide to focus only on LTS releases.

Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 407: New release cycle and introducing long-term support versions

[Georg Brandl]

Am 18.01.2012 00:50, schrieb Ezio Melotti:
> Hi,
> 
> On 17/01/2012 22.34, Antoine Pitrou wrote:
>> [...]
>>
>> Proposal
>> 
>>
>> Under the proposed scheme, there would be two kinds of feature
>> versions (sometimes dubbed "minor versions", for example 3.2 or 3.3):
>> normal feature versions and long-term support (LTS) versions.
>>
>> Normal feature versions would get either zero or at most one bugfix
>> release; the latter only if needed to fix critical issues.  Security
>> fix handling for these branches needs to be decided.
> 
> If non-LTS releases won't get bug fixes, a bug that is fixed in 3.3.x 
> might not be fixed in 3.4, unless the bug fixes releases are 
> synchronized with the new feature releases (see below).

That's already the case today.  3.2.5 might be released before 3.3.1 and
therefore include bugfixes that 3.3.0 doesn't.  True, there will be a
3.3.1 afterwards that does include it, but in the new case, there will be
a new feature release instead.

>> LTS versions would get regular bugfix releases until the next LTS
>> version is out.  They then would go into security fixes mode, up to a
>> termination date at the release manager's discretion.
>>
>> Periodicity
>> ---
>>
>> A new feature version would be released every X months.  We
>> tentatively propose X = 6 months.
>>
>> LTS versions would be one out of N feature versions.  We tentatively
>> propose N = 4.
> 
> If LTS bug fixes releases and feature releases are synchronized, we will 
> have something like:
> 
> 3.3
> 3.3.1 / 3.4
> 3.3.2 / 3.5
> 3.3.3 / 3.6
> 3.7
> 3.7.1 / 3.8
> ...
> 
> so every new feature release will have all the bug fixes of the current 
> LTS release, plus new features.
> 
> With this scheme we will soon run out of 1-digit numbers though.
> Currently we already have a 3.x release every ~18 months, so if we keep 
> doing that (just every 24 months instead of 18) and introduce the 
> feature releases in between under a different versioning scheme, we 
> might avoid the problem.
> 
> This means:
> 3.1
> ... 18 months, N bug fix releases...
> 3.2
> ... 18 months, N bug fix releases ...
> 3.3 LTS
> ... 24 months, 3 bug fix releases, 3 feature releases ...
> 3.4 LTS
> ... 24 months, 3 bug fix releases, 3 feature releases ...
> 3.5 LTS
> 
> In this way we solve the numbering problem and keep a familiar scheme 
> (all the 3.x will be LTS and will be released as the same pace as 
> before, no need to mark some 3.x as LTS).  OTOH this will make the 
> feature releases less "noticeable" and people might just ignore them and 
> stick with the LTS releases.  Also we would need to define a versioning 
> convention for the feature releases.

Let's see how Guido feels about 3.10 first.

>> [...]
>>
>> Effect on bugfix cycle
>> --
>>
>> The effect on fixing bugs should be minimal with the proposed figures.
>> The same number of branches would be simultaneously open for regular
>> maintenance (two until 2.x is terminated, then one).
> 
> Wouldn't it still be two?
> Bug fixes will go to the last LTS and on default, features only on default.

"Maintenance" excludes the feature development branch here.  Will clarify.

>> Effect on workflow
>> --
>>
>> The workflow for new features would be the same: developers would only
>> commit them on the ``default`` branch.
>>
>> The workflow for bug fixes would be slightly updated: developers would
>> commit bug fixes to the current LTS branch (for example ``3.3``) and
>> then merge them into ``default``.
> 
> So here the difference is that instead of committing on the previous 
> release (what currently is 3.2), we commit it to the previous LTS 
> release, ignoring the ones between that and default.

Yes.

>> If some critical fixes are needed to a non-LTS version, they can be
>> grafted from the current LTS branch to the non-LTS branch, just like
>> fixes are ported from 3.x to 2.7 today.
>>
>> Effect on the community
>> ---
>>
>> People who value stability can just synchronize on the LTS releases
>> which, with the proposed figures, would give a similar support cycle
>> (both in duration and in stability).
> 
> That's why I proposed to keep the same versioning scheme for these 
> releases, and have a different numbering for the feature releases.
> 
>> [...]
>>
>> Discussion
>> ==
>>
>> These are open issues that should be worked out during discussion:
>>
>> * Decide on X (months between feature releases) and N (feature releases
>>per LTS release) as defined above.
> 
> This doesn't necessarily have to be fixed, especially if we don't change 
> the versioning scheme (so we don't need to know that we have a LTS 
> release every N releases).

For these relatively short times (X = 6 months), I feel it is important
to fix the time spans to have predictability for our developers.

Georg

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/list