[Python-Dev] PEP 484 syntax: NONONONONONONO!

[Benjamin]

The proposed syntax is abominable. It's the opposite of readable.

The function annotation syntax is ugly, but potentially useful for things
like documentation. While it may very well have been created with the idea
of type-checking, actually using it for such quickly turns into an
unreadable morass of information that is far more difficult for human
brains to parse, which makes this usage the antithesis of pythonic.

I much prefer the idea of a 'where' keyword to denote typing, as discussed
http://aroberge.blogspot.com/2015/01/type-hinting-in-python-focus-on.html,
but I think a refinement of their idea would prove even better:

def retry(callback, timeout, retries=None) where
callback is Callable[AnyArgs, Any[int, None]],
timeout is Any[int, None],
retries is in [int, None], # 'is in' construct is more readable,
dunno about complexity
return is None:
pass

def greeting(name) where name is str, return is str:
return 'Hello ' + name

x, y = [], [] where x, y is List[Employee], List[Manager]

To me, this orders of magnitude more readable than the proposed nonsense.

PS. Obviously the 8-space indent above would only a convention, not
requirement.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] dir() and __all__

[Benjamin]

On Feb 15, 9:18 pm, "Raymond Hettinger" <[EMAIL PROTECTED]> wrote:
> [Raymond]
>
> >> Should dir(module) use __all__ when defined?
>
> [GvR]
>
> > It's not consistent with what dir() of a class or instance does though.
>
> > -1.
>
> Perhaps there is another solution. Have dir() exclude objects
> which are modules.  For example, dir(logging) would exclude
> sys, os, types, time, string, cStringIO, and traceback.
Are you proposing just a list those modules or all module objects?
Excluding all modules would endanger __init__.py modules which
imported modules from their package.
>
> Raymond
> ___
> Python-Dev mailing list
> [EMAIL PROTECTED]://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:http://mail.python.org/mailman/options/python-dev/python-dev2-garchiv...
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Fix Unicode-disabled build of Python 2.7

[Benjamin Peterson]

If Serhiy wants to spend his time supporting this arcane feature, he can
do that. It doesn't really seem worth risking regressions to do this,
though.

On Tue, Jun 24, 2014, at 01:55, Victor Stinner wrote:
> Hi,
> 
> I don't know anyone building Python without Unicode. I would prefer to
> modify configure to raise an error, and drop #ifdef in the code. (Stop
> supporting building Python 2 without Unicode.)
> 
> Building Python 2 without Unicode support is not an innocent change.
> Python is moving strongly to Unicode: Python 3 uses Unicode by
> default. So to me it sounds really weird to work on building Python 2
> without Unicode support. It means that you may have "Python 2" and
> "Python 2 without Unicode" which are not exactly the same language.
> IMO u"unicode" is part of the Python 2 language.
> 
> --disable-unicode is an old option added while Python 1.5 was very
> slowly moving to Unicode.
> 
> I have the same opinion on --without-thread option (we should stop
> supporting it, this option is useless). I worked in the embedded
> world, Python used for the UI of a TV set top box. Even if the
> hardware was slow and old, Python was compiled with threads and
> Unicode. Unicode was mandatory to handle correctly letters with
> diacritics, threads were used to handle network and D-Bus for
> examples.
> 
> Victor
> 
> 
> 2014-06-24 10:22 GMT+02:00 Serhiy Storchaka :
> > I submitted a number of patches which fixes currently broken
> > Unicode-disabled build of Python 2.7 (built with --disable-unicode configure
> > option). I suppose this was broken in 2.7 when C implementation of the io
> > module was introduced.
> >
> > http://bugs.python.org/issue21833 -- main patch which fixes the io module
> > and adds helpers for testing.
> >
> > http://bugs.python.org/issue21834 -- a lot of minor fixes for tests.
> >
> > Following issues fix different modules and related tests:
> >
> > http://bugs.python.org/issue21854 -- cookielib
> > http://bugs.python.org/issue21838 -- ctypes
> > http://bugs.python.org/issue21855 -- decimal
> > http://bugs.python.org/issue21839 -- distutils
> > http://bugs.python.org/issue21843 -- doctest
> > http://bugs.python.org/issue21851 -- gettext
> > http://bugs.python.org/issue21844 -- HTMLParser
> > http://bugs.python.org/issue21850 -- httplib and SimpleHTTPServer
> > http://bugs.python.org/issue21842 -- IDLE
> > http://bugs.python.org/issue21853 -- inspect
> > http://bugs.python.org/issue21848 -- logging
> > http://bugs.python.org/issue21849 -- multiprocessing
> > http://bugs.python.org/issue21852 -- optparse
> > http://bugs.python.org/issue21840 -- os.path
> > http://bugs.python.org/issue21845 -- plistlib
> > http://bugs.python.org/issue21836 -- sqlite3
> > http://bugs.python.org/issue21837 -- tarfile
> > http://bugs.python.org/issue21835 -- Tkinter
> > http://bugs.python.org/issue21847 -- xmlrpc
> > http://bugs.python.org/issue21841 -- xml.sax
> > http://bugs.python.org/issue21846 -- zipfile
> >
> > Most fixes are trivial and are only several lines of a code.
> >
> > ___
> > Python-Dev mailing list
> > Python-Dev@python.org
> > https://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> > https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/benjamin%40python.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cpython (3.3): Closes #20872: dbm/gdbm/ndbm close methods are not documented

[Benjamin Peterson]

On Wed, Jun 25, 2014, at 23:38, Ned Deily wrote:
> In article <53ab53a7.6050...@jcea.es>, Jesus Cea  wrote:
> 
> > On 25/06/14 20:35, Ned Deily wrote:
> > > The 3.3 branch is open only to security fixes. Please don't backport 
> > > other patches to there.
> > > 
> > > https://docs.python.org/devguide/devcycle.html#summary
> > 
> > Ned, I am aware. It is a doc-only fix, like fixing a typo or correcting
> > an incorrect statement. It that is against policy, let me know.
> 
> My understanding is that doc changes are treated the same as any other 
> code changes.  As you noticed, after a release leaves maintenance mode, 
> its documentation is no longer updated on the web site.

To echo Ned, committing a doc change to 3.3 isn't the end of the world.
We just want to make sure energy is focused on the 3 branches we do
fully maintain.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] C version of functools.lru_cache

[Benjamin Peterson]

You might look at https://bugs.python.org/issue14373

On Thu, Jun 26, 2014, at 08:38, Peter Brady wrote:
> Hello python devs,
> 
> I was recently in need of some faster caching and thought this would be a
> good opportunity to familiarize myself with the Python/C api so I wrote a
> C
> extension for the lru_cache in functools.  The source is at
> https://github.com/pbrady/fastcache.git and I've posted it as a package
> on
> PyPI (fastcache).  There are some simple benchmarks on the github page
> showing about 9x speedup.  I would like to submit this for incorporation
> into the standard library.  Is there any interest in this? I suspect it
> probably requires some changes/cleanup especially since I haven't
> addressed
> thread-safety at all.
> 
> Thanks,
> Peter.
> 
> P.S. This was the motivation for the faster caching
> https://github.com/sympy/sympy/pull/7464.
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/benjamin%40python.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

[Benjamin Peterson]

On Thu, Jun 26, 2014, at 17:07, Paul Sokolovsky wrote:
> 
> With my MicroPython hat on, os.scandir() would make things only worse.
> With current interface, one can either have inefficient implementation
> (like CPython chose) or efficient implementation (like MicroPython
> chose) - all transparently. os.scandir() supposedly opens up efficient
> implementation for everyone, but at the price of bloating API and
> introducing heavy-weight objects to wrap info. PEP calls it
> "lightweight DirEntry objects", but that cannot be true, because all
> Python objects are heavy-weight, especially those which have methods.

Why do you think methods make an object more heavyweight? namedtuples
have methods.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] buildbot.python.org down?

[Benjamin Peterson]

On Fri, Jun 27, 2014, at 02:14, Ned Deily wrote:
> The buildbot web site seems to have been down for some hours and still 
> is as of 0915 UTC.  I'm not sure who is watching over it but I'll ping 
> the infrastructure team as well.

Fixed. The VM crashed, and Ernest rebooted it.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] [RELEASE] Python 2.7.8

[Benjamin Peterson]

Greetings,
I have the distinct privilege of informing you that the latest release
of the Python 2.7 series, 2.7.8, has been released and is available for
download. 2.7.8 contains several important regression fixes and security
changes:
  - The openssl version bundled in the Windows installer has been
  updated.
  - A regression in the mimetypes module on Windows has been fixed. [1]
  - A possible overflow in the buffer type has been fixed. [2]
  - A bug in the CGIHTTPServer module which allows arbitrary execution
  of code in the server root has been patched. [3]
  - A regression in the handling of UNC paths in os.path.join has been
  fixed. [4]

Downloads of 2.7.8 are at

https://www.python.org/download/releases/2.7.8/

The full changelog is located at

http://hg.python.org/cpython/raw-file/v2.7.8/Misc/NEWS

This is a production release. As always, please report bugs to

http://bugs.python.org/

Till next time,
Benjamin Peterson
2.7 Release Manager
(on behalf of all of Python's contributors)

[1] http://bugs.python.org/issue21652
[2] http://bugs.python.org/issue21831
[3] http://bugs.python.org/issue21766
[4] http://bugs.python.org/issue21672
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] == on object tests identity in 3.x

[Benjamin Peterson]

On Mon, Jul 7, 2014, at 04:22, Andreas Maier wrote:
> While discussing Python issue #12067 
> (http://bugs.python.org/issue12067#msg222442), I learned that Python 3.4 
> implements '==' and '!=' on the object type such that if no special 
> equality test operations are implemented in derived classes, there is a 
> default implementation that tests for identity (as opposed to equality 
> of the values).
> 
> The relevant code is in function do_richcompare() in Objects/object.c.
> 
> IMHO, that default implementation contradicts the definition that '==' 
> and '!=' test for equality of the values of an object.
> 
> Python 2.x does not seem to have such a default implementation; == and 
> != raise an exception if attempted on objects that don't implement 
> equality in derived classes.

Why do you think that?

% python
Python 2.7.6 (default, May 29 2014, 22:22:15) 
[GCC 4.7.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> class x(object): pass
... 
>>> class y(object): pass
... 
>>> x != y
True
>>> x == y
False
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] buildbot.python.org down again?

[Benjamin Peterson]

On Mon, Jul 7, 2014, at 08:44, Guido van Rossum wrote:
> It would still be nice to know who "the appropriate persons" are. Too
> much
> of our infrastructure seems to be maintained by house elves or the ITA.

:) Is ITA "International Trombone Association"?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] == on object tests identity in 3.x

[Benjamin Peterson]

On Mon, Jul 7, 2014, at 16:36, Andreas Maier wrote:
> Am 2014-07-07 19:43, schrieb Ethan Furman:
> > On 07/07/2014 09:56 AM, Andreas Maier wrote:
> >> Am 07.07.2014 17:55, schrieb Ethan Furman:
> >>> On 07/07/2014 04:22 AM, Andreas Maier wrote:
> 
>  Where is the discrepancy between the documentation of == and its
>  default implementation on object documented?
> >>>
> >>> There's seems to be no discrepancy (at least, you have not shown it),
> >>
> >> The documentation states consistently that == tests the equality of 
> >> the value of an object. The default implementation
> >> of == in both 2.x and 3.x tests the object identity. Is that not a 
> >> discrepancy?
> >
> > One could say that the value of an object is the object itself. Since 
> > different objects are different, then they are not equal.
> >
> >>> but to answer the question about why the default equals operation is an
> >>> identity test:
> >>>
> >>>- all objects should be equal to themselves (there is only one that
> >>> isn't, and it's weird)
> >>
> >> I agree. But that is not a reason to conclude that different objects 
> >> (as per their identity) should be unequal. Which is
> >> what the default implementation does.
> >
> > Python cannot know which values are important in an equality test, and 
> > which are not.  So it refuses to guess.
> >
> Well, one could argue that using the address of an object for its value 
> equality test is pretty close to guessing, considering that given a 
> sensible definition of value equality, objects of different identity can 
> very well be equal but will always be considered unequal based on the 
> address.

Probably the best argument for the behavior is that "x is y" should
imply "x == y", which preludes raising an exception. No such invariant
is desired for ordering, so default implementations of < and > are not
provided in Python 3.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] https:bugs.python.org -- Untrusted Connection (Firefox)

[Benjamin Peterson]

It uses a CACert certificate, which your system probably doesn't trust.

On Mon, Aug 18, 2014, at 13:12, Terry Reedy wrote:
> Firefox does not want to connect to https:bugs.python.org. Plain 
> bugs.python.org works fine. Has the certificate expired?
> 
> -- 
> Terry Jan Reedy
> 
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/benjamin%40python.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Bytes path support

[Benjamin Peterson]

On Tue, Aug 19, 2014, at 10:31, Ben Hoyt wrote:
> > The official policy is that we want them [support for bytes paths in stdlib 
> > functions] to go away, but reality so far has not budged. We will continue 
> > to hold our breath though. :-)
> 
> Does that mean that new APIs should explicitly not support bytes? I'm
> thinking of os.scandir() (PEP 471), which I'm implementing at the
> moment. I was originally going to make it support bytes so it was
> compatible with listdir, but maybe that's a bad idea. Bytes paths are
> essentially broken on Windows.

Bytes paths are "essential" on Unix, though, so I don't think we should
create new low-level APIs that don't support bytes.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Bytes path support

[Benjamin Peterson]

On Tue, Aug 19, 2014, at 10:43, Ben Hoyt wrote:
> >> > The official policy is that we want them [support for bytes paths in 
> >> > stdlib functions] to go away, but reality so far has not budged. We will 
> >> > continue to hold our breath though. :-)
> >>
> >> Does that mean that new APIs should explicitly not support bytes? I'm
> >> thinking of os.scandir() (PEP 471), which I'm implementing at the
> >> moment. I was originally going to make it support bytes so it was
> >> compatible with listdir, but maybe that's a bad idea. Bytes paths are
> >> essentially broken on Windows.
> >
> > Bytes paths are "essential" on Unix, though, so I don't think we should
> > create new low-level APIs that don't support bytes.
> 
> Fair enough. I don't quite understand, though -- why is the "official
> policy" to kill something that's "essential" on *nix?

Well, notice the official policy is desperately *wanting* them to go
away with the implication that we grudgingly bow to reality. :)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] https:bugs.python.org -- Untrusted Connection (Firefox)

[Benjamin Peterson]

On Thu, Aug 21, 2014, at 09:48, Ryan Hiebert wrote:
> 
> > On Aug 21, 2014, at 11:29 AM, Martin v. Löwis  wrote:
> > 
> > Am 21.08.14 17:44, schrieb Nick Coghlan:
> >> I've now raised this issue with the infrastructure team. The current
> >> hosting arrangements for bugs.python.org were put in place when the
> >> PSF didn't have any on-call system administrators of its own, but now
> >> that we do, it may be time to migrate that service to a location where
> >> we can switch to a more appropriate SSL certificate.
> > 
> > Just to relay Noah's response: it's actually not the hosting that
> > prevents installation of a proper certificate, it's the limitation
> > that the certificate we could deploy would include "python.org" as
> > a server name, which is considered risky regardless of where the
> > service is hosted. There are solutions to that as well, of course.
> 
> That sounds like a limitation I’ve seen with StartSSL. Perhaps there’s a
> certificate authority that would be willing to sponsor a certificate for
> Python without this annoying limitation?

Perhaps some board members could comment, but I hope the PSF could just
pay a few hundred a year for a proper certificate.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 477: selected ensurepip backports for Python 2.7

[Benjamin Peterson]

On Sun, Aug 31, 2014, at 16:17, Antoine Pitrou wrote:
> On Mon, 1 Sep 2014 08:00:14 +1000
> Nick Coghlan  wrote:
> > 
> > That part of the proposal proved to be controversial, so we dropped it from
> > the original PEP in order to focus on meeting the Python 3.4 specific
> > release deadlines. This also had the benefit of working out the kinks in
> > the bootstrapping processing as part of the Python 3.4 release cycle.
> > 
> > However, we still think we should start providing pip by default to Python
> > 2.7 users as well, at least as part of the Windows and Mac OS X installers.
> 
> I don't agree with this. pip is simply not part of the 2.7 feature set.
> If you add pip to a bugfix version, then you have bugfix versions which
> are more featureful than others, which makes things more complicated to
> explain.

2.7.x has been and will be alive for so long that will already have to
explain that sort thing; i.e. PEP 466 and why different bugfix releases
support different versions of dependency libraries.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Daily reference leaks (640c575ab3e1): sum=151940

[Benjamin Peterson]

The codecs one is https://bugs.python.org/issue22166

On Mon, Sep 1, 2014, at 16:16, Antoine Pitrou wrote:
> 
> Is anyone working on those?
> 
> 
> 
> On Mon, 01 Sep 2014 10:41:45 +0200
> solip...@pitrou.net wrote:
> > results for 640c575ab3e1 on branch "default"
> > 
> > 
> > test_codecs leaked [5825, 5825, 5825] references, sum=17475
> > test_codecs leaked [1172, 1174, 1174] memory blocks, sum=3520
> > test_collections leaked [0, 2, 0] references, sum=2
> > test_distutils leaked [37735, 37735, 37735] references, sum=113205
> > test_distutils leaked [5909, 5911, 5911] memory blocks, sum=17731
> > test_functools leaked [0, 0, 3] memory blocks, sum=3
> > test_site leaked [0, 0, 2] references, sum=2
> > test_site leaked [0, 0, 2] memory blocks, sum=2
> > 
> > 
> > Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', 
> > '3:3:/home/antoine/cpython/refleaks/reflogHPiXJp', '-x']
> 
> 
> 
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/benjamin%40python.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 476: Enabling certificate validation by default!

[Benjamin Peterson]

On Wed, Sep 3, 2014, at 13:37, Alex Gaynor wrote:
> Guido van Rossum  python.org> writes:
> 
> > OK, that changes my position for 2.7 (but not for 3.5). I had assumed there
> > was a way to disable the cert check by changing one parameter to the
> > urlopen() call. (And I had wanted to add that there should be a clear FAQ
> > about the subject.) If this isn't possible that changes the situation. (But 
> > I
> > still think that once we do have that simple change option we should do it,
> > in a later 2.7 upgrade.) I apologize for speaking before I had read all
> > facts, and I'll await what you and Nick come up with.
> > --Guido
> 
> This probably doesn't surprise anyone, but I'm more than happy to do the
> back-
> porting work for httplib, and any other modules which need SSLContext
> support;
> does this require an additional PEP, or does it fit under PEP466 or
> PEP476?

I imagine you could put it into 476.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 477: selected ensurepip backports for Python 2.7

[Benjamin Peterson]

On Wed, Sep 3, 2014, at 17:03, Nick Coghlan wrote:
> On 1 September 2014 08:00, Nick Coghlan  wrote:
> > Earlier versions of PEP 453 proposed bootstrapping pip into a Python 2.7
> > maintenance release in addition to including it with Python 3.4.
> >
> > That part of the proposal proved to be controversial, so we dropped it from
> > the original PEP in order to focus on meeting the Python 3.4 specific
> > release deadlines. This also had the benefit of working out the kinks in the
> > bootstrapping processing as part of the Python 3.4 release cycle.
> >
> > However, we still think we should start providing pip by default to Python
> > 2.7 users as well, at least as part of the Windows and Mac OS X installers.
> 
> I suggested off-list that, as Python 2.7 release manager, Benjamin
> would be a good candidate for BDFL-Delegate for this proposal, and
> both Guido and Benjamin agreed. The PEP has been updated accordingly:
> http://hg.python.org/peps/rev/909954e4ba43 (I also fixed a
> copy-and-paste error in the PEP status!)

... and accepted. Let's make our users' lives' better!.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] new hg.python.org server

[Benjamin Peterson]

I just switched hg.python.org from a OSUOSL VM to a Rackspace VM. The
new VM is a bit beefier and has what I think is better network
connectivity, so hopefully that will improving the speed of repository
operations. We also now support HTTPS for repository browsing and
cloning, so update all your links to https://hg.python.org! IPv6 support
has also returned for those who like that sort of thing.

Note the host keys changed, so you'll probably have to futz with
known_hosts to quiet ssh down. I apologize, but I noticed that that the
current RSA host key is 1024 bits, so I decided to upgrade it to 2048
during the transition.

Thanks to Donald Stufft for helping me set this up.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [python-committers] new hg.python.org server

[Benjamin Peterson]

On Fri, Sep 12, 2014, at 21:52, Shorya Raj wrote:
> Just wondering - are there any sys-adminy sort of tasks that could be
> completed? I mean, I have some (note, some) experience doing this, and I
> wouldn't mind helping out (I inquired in the buildbot thread as well, but
> there wasn't much of a response).

Well, hg.python.org is basically done now. The main thing now is
understanding how other services (planet.python.org, bugs.python.org)
are setup and moving them to config management.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Semi-official read-only Github mirror of the CPython Mercurial repository

[Benjamin Peterson]

Eli,
Thanks for setting this up. People are evidently finding it quite useful
and are wondering if it could be more frequently run. We don't want you
to have to absorb the bandwidth costs yourself, though. Is the code you
use available somewhere? It shouldn't be hard to set up the cron job on
PSF infrastructure.

On Sat, Jul 12, 2014, at 09:15, Eli Bendersky wrote:
> Just a quick update on this. I've finally found time to set up a VPS at
> DigitalOcean of myself, and I'm moving the cronjob for updating the
> Github
> mirrors to it. This lets me ramp up the update frequency. For now I'll
> set
> it to every 4 hours, but in the future I may make it even more frequent.
> Hopefully this will not overrun my bandwidth allocation :)
> 
> The CPython mirror (https://github.com/python/cpython) has been pretty
> popular so far, with over 70 forks.
> 
> Eli
> 
> 
> 
> On Mon, Sep 30, 2013 at 6:09 AM, Eli Bendersky  wrote:
> 
> > Hi all,
> >
> > https://github.com/python/cpython is now live as a semi-official, *read
> > only* Github mirror of the CPython Mercurial repository. Let me know if you
> > have any problems/concerns.
> >
> > I still haven't decided how often to update it (considering either just N
> > times a day, or maybe use a Hg hook for batching). Suggestions are welcome.
> >
> > The methodology I used to create it is via hg-fast-export. I also tried to
> > pack and gc the git repo as much as possible before the initial Github push
> > - it went down from almost ~2GB to ~200MB (so this is the size of a fresh
> > clone right now).
> >
> > Eli
> >
> > P.S. thanks Jesse for the keys to https://github.com/python
> >
> >
> >
> >
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/benjamin%40python.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Idea to support lazy loaded names.

[Benjamin Peterson]

On Mon, Oct 6, 2014, at 16:36, Brian Allen Vanderburg II wrote:
> I've got an idea for Python but don't know where to post it.

python-id...@python.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Fixing 2.7.x

[Benjamin Peterson]

On Mon, Oct 6, 2014, at 19:13, Ned Deily wrote:
> In article 
> ,
>  Zachary Ware  wrote:
> > On Mon, Oct 6, 2014 at 12:24 PM, Ned Deily  wrote:
> > > 3. security: "fixing issues exploitable by attackers such as crashes,
> > > privilege escalation and, optionally, other issues such as denial of
> > > service attacks. Any other changes are not considered a security risk
> > > and thus not backported to a security branch."
> > >= 3.2.x and 3.3.x
> > 3.1 is still in this category, is it not?  According to PEP 375, it's
> > a few months past due for its last release.
> > http://legacy.python.org/dev/peps/pep-0375/#maintenance-releases
> 
> I don't think Benjamin was planning any further security releases.  But, 
> in any case, the PEP should be updated, I guess.  Benjamin?

Oh, yeah, I'm really tired of 3.1, so it's over.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Sad status of Python 3.x buildbots

[Benjamin Peterson]

It's https://bugs.python.org/issue19884

On Fri, Oct 10, 2014, at 12:08, R. David Murray wrote:
> On Fri, 10 Oct 2014 18:00:06 +0200, Jesus Cea  wrote:
> > On 10/10/14 17:56, Chris Angelico wrote:
> > > Could I write a little
> > > monitor at my end that asks every hour if my buildbots can be seen?
> > 
> > AFAIK maintainers already get an email if the buildbot vanishes long
> > enough. I am more interested in getting an email when my buildbot is
> > consistently red because somebody committed something it doesn't like
> > two months ago...
> 
> I think they are supposed to, but I've never gotten one, not even when
> my gentoo buildbots suffered a hardware failure.  A month or so later
> Antoine emailed me, and I told him to remove them at least for now,
> since I don't currently have replacement hardware.  (I'm hoping to fix
> that, but I have to find the time...)
> 
> That said, there has never as far as I know been an active hook to email
> the maintainer when the buildbot is red.  I don't know if buildbot
> supports that or not, so that would be the first thing to check.
> 
> --David
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/benjamin%40python.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] 2.7.9 schedule

[Benjamin Peterson]

Multiple RGCs are not unlikely given.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 2.7.9 schedule

[Benjamin Peterson]

Oops, I hit "send" a bit early on that...

On Fri, Oct 31, 2014, at 19:46, Benjamin Peterson wrote:
> Multiple RGCs are not unlikely given.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] 2.7.9 schedule redux

[Benjamin Peterson]

I'm updating the 2.7 release PEP with the following release dates

2.7.9rc1 Nov 22
2.7.9 final December 5

I expect PEPs 476 and 477 can be done by then?

Given that 2.7.9 has had some large changes, having multiple rcs is not
unlikely. Hopefully, we can get a final out by 2015, though.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] cpython (2.7): Issue #22773: fix failing test with old readline versions due to issue #19884.

[Benjamin Peterson]

On Tue, Nov 4, 2014, at 09:55, antoine.pitrou wrote:
> https://hg.python.org/cpython/rev/eba6e68e818c
> changeset:   93382:eba6e68e818c
> branch:  2.7
> parent:  93379:e54d0b197c82
> user:Antoine Pitrou 
> date:Tue Nov 04 14:52:10 2014 +0100
> summary:
>   Issue #22773: fix failing test with old readline versions due to issue
>   #19884.
> 
> files:
>   Lib/test/test_readline.py |  4 
>   Modules/readline.c|  3 +++
>   2 files changed, 7 insertions(+), 0 deletions(-)
> 
> 
> diff --git a/Lib/test/test_readline.py b/Lib/test/test_readline.py
> --- a/Lib/test/test_readline.py
> +++ b/Lib/test/test_readline.py
> @@ -43,6 +43,10 @@
>  
>  
>  class TestReadline(unittest.TestCase):
> +
> +@unittest.skipIf(readline._READLINE_VERSION < 0x0600

Shouldn't this use the runtime version?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] cpython (2.7): #22650: test suite: load Unicode test data files from www.pythontest.net

[Benjamin Peterson]

On Thu, Nov 6, 2014, at 09:39, Brett Cannon wrote:
> What is pythontest.net? Is it something we control, and if so how do we
> add
> things to it for tests? Did I miss an email on python-dev or
> python-committers about this?

See https://bugs.python.org/issue22650

> 
> On Thu Nov 06 2014 at 8:57:22 AM georg.brandl
> 
> wrote:
> 
> > https://hg.python.org/cpython/rev/0af36ea1d010
> > changeset:   93417:0af36ea1d010
> > branch:  2.7
> > parent:  93401:3e8d3c4bc17e
> > user:Georg Brandl 
> > date:Thu Nov 06 14:37:49 2014 +0100
> > summary:
> >   #22650: test suite: load Unicode test data files from www.pythontest.net
> >
> > files:
> >   Lib/test/test_codecmaps_cn.py  |   8 +++-
> >   Lib/test/test_codecmaps_hk.py  |   2 +-
> >   Lib/test/test_codecmaps_jp.py  |  12 +---
> >   Lib/test/test_codecmaps_kr.py  |   8 +++-
> >   Lib/test/test_codecmaps_tw.py  |   6 ++
> >   Lib/test/test_normalization.py |   2 +-
> >   6 files changed, 15 insertions(+), 23 deletions(-)
> >
> >
> > diff --git a/Lib/test/test_codecmaps_cn.py b/Lib/test/test_codecmaps_cn.py
> > --- a/Lib/test/test_codecmaps_cn.py
> > +++ b/Lib/test/test_codecmaps_cn.py
> > @@ -10,19 +10,17 @@
> >  class TestGB2312Map(test_multibytecodec_support.TestBase_Mapping,
> > unittest.TestCase):
> >  encoding = 'gb2312'
> > -mapfileurl = 'http://people.freebsd.org/~perky/i18n/EUC-CN.TXT'
> > +mapfileurl = 'http://www.pythontest.net/unicode/EUC-CN.TXT'
> >
> >  class TestGBKMap(test_multibytecodec_support.TestBase_Mapping,
> > unittest.TestCase):
> >  encoding = 'gbk'
> > -mapfileurl = 'http://www.unicode.org/Public/MAPPINGS/VENDORS/' \
> > - 'MICSFT/WINDOWS/CP936.TXT'
> > +mapfileurl = 'http://www.pythontest.net/unicode/CP936.TXT'
> >
> >  class TestGB18030Map(test_multibytecodec_support.TestBase_Mapping,
> >   unittest.TestCase):
> >  encoding = 'gb18030'
> > -mapfileurl = 'http://source.icu-project.org/repos/icu/data/' \
> > - 'trunk/charset/data/xml/gb-18030-2000.xml'
> > +mapfileurl = 'http://www.pythontest.net/unicode/gb-18030-2000.xml'
> >
> >
> >  def test_main():
> > diff --git a/Lib/test/test_codecmaps_hk.py b/Lib/test/test_codecmaps_hk.py
> > --- a/Lib/test/test_codecmaps_hk.py
> > +++ b/Lib/test/test_codecmaps_hk.py
> > @@ -10,7 +10,7 @@
> >  class TestBig5HKSCSMap(test_multibytecodec_support.TestBase_Mapping,
> > unittest.TestCase):
> >  encoding = 'big5hkscs'
> > -mapfileurl = 'http://people.freebsd.org/~
> > perky/i18n/BIG5HKSCS-2004.TXT'
> > +mapfileurl = 'http://www.pythontest.net/unicode/BIG5HKSCS-2004.TXT'
> >
> >  def test_main():
> >  test_support.run_unittest(__name__)
> > diff --git a/Lib/test/test_codecmaps_jp.py b/Lib/test/test_codecmaps_jp.py
> > --- a/Lib/test/test_codecmaps_jp.py
> > +++ b/Lib/test/test_codecmaps_jp.py
> > @@ -10,8 +10,7 @@
> >  class TestCP932Map(test_multibytecodec_support.TestBase_Mapping,
> > unittest.TestCase):
> >  encoding = 'cp932'
> > -mapfileurl = 'http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/'
> > \
> > - 'WINDOWS/CP932.TXT'
> > +mapfileurl = 'http://www.pythontest.net/unicode/CP932.TXT'
> >  supmaps = [
> >  ('\x80', u'\u0080'),
> >  ('\xa0', u'\uf8f0'),
> > @@ -27,15 +26,14 @@
> >   unittest.TestCase):
> >  encoding = 'euc_jp'
> >  mapfilename = 'EUC-JP.TXT'
> > -mapfileurl = 'http://people.freebsd.org/~perky/i18n/EUC-JP.TXT'
> > +mapfileurl = 'http://www.pythontest.net/unicode/EUC-JP.TXT'
> >
> >
> >  class TestSJISCOMPATMap(test_multibytecodec_support.TestBase_Mapping,
> >  unittest.TestCase):
> >  encoding = 'shift_jis'
> >  mapfilename = 'SHIFTJIS.TXT'
> > -mapfileurl = 'http://www.unicode.org/Public/MAPPINGS/OBSOLETE' \
> > - '/EASTASIA/JIS/SHIFTJIS.TXT'
> > +mapfileurl = 'http://www.pythontest.net/unicode/SHIFTJIS.TXT'
> >  pass_enctest = [
> >  ('\x81_', u'\\'),
> >  ]
> > @@ -49,14 +47,14 @@
> >   unittest.TestCase):
> >  encoding = 'euc_jisx0213'
> >  mapfilename = 'EUC-JISX0213.TXT'
> > -mapfileurl = 'http://people.freebsd.org/~perky/i18n/EUC-JISX0213.TXT'
> > +mapfileurl = 'http://www.pythontest.net/unicode/EUC-JISX0213.TXT'
> >
> >
> >  class TestSJISX0213Map(test_multibytecodec_support.TestBase_Mapping,
> > unittest.TestCase):
> >  encoding = 'shift_jisx0213'
> >  mapfilename = 'SHIFT_JISX0213.TXT'
> > -mapfileurl = 'http://people.freebsd.org/~
> > perky/i18n/SHIFT_JISX0213.TXT'
> > +mapfileurl = 'http://www.pythontest.net/unicode/SHIFT_JISX0213.TXT'
> >
> >
> >  def test_main():
> > diff --git a/Lib/test/test_codecmaps_kr.py b/Lib/test/test_codecmaps_kr.py
> > --- a/Lib/test/test_codecmaps_kr.py
> > +++ b/Lib/test/

Re: [Python-Dev] [Python-checkins] cpython (2.7): #22650: test suite: load Unicode test data files from www.pythontest.net

[Benjamin Peterson]

On Thu, Nov 6, 2014, at 10:41, Brett Cannon wrote:
> Ah, cool! Just an FYI, the index.html file is not being served for me.

At the moment it's only served on www.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] OneGet provider for Python

[Benjamin Peterson]

On Sat, Nov 15, 2014, at 05:54, Nathaniel Smith wrote:
> On 15 Nov 2014 10:10, "Paul Moore"  wrote:
> >
> > > Incidentally, it would be really useful if python.org provided stable
> > > url's that always redirected to the latest .msi installers, for
> > > bootstrapping purposes. I'd prefer to not rely on chocolatey (or on
> > > scraping the web site) for this.
> >
> > https://www.python.org/ftp/python/$ver/python-$ver.msi
> > https://www.python.org/ftp/python/$ver/python-$ver.amd64.msi
> 
> Right, but what's the URL for "the latest 2.7.x release" or "the latest
> 3.x.x release"?

The website has an API you know.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators

[Benjamin Peterson]

On Wed, Nov 19, 2014, at 15:10, Guido van Rossum wrote:
> There's a new PEP proposing to change how to treat StopIteration bubbling
> up out of a generator frame (not caused by a return from the frame). The
> proposal is to replace such a StopIteration with a RuntimeError (chained
> to
> the original StopIteration), so that only *returning* from a generator
> (or
> falling off the end) causes the iteration to terminate.
> 
> The proposal unifies the behavior of list comprehensions and generator
> expressions along the lines I had originally in mind when they were
> introduced. It renders useless/illegal certain hacks that have crept into
> some folks' arsenal of obfuscated Python tools.
> 
> In Python 3.5 the proposed change is conditional on:
> 
> from __future__ import replace_stopiteration_in_generators

Drive-by comment: This seems like a terribly awkward name. Could a
shorter and sweeter name not be found?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?

[Benjamin Peterson]

On Fri, Nov 21, 2014, at 07:36, Nick Coghlan wrote:
> For those that aren't aware, PEP 474 is a PEP I wrote a while back
> suggesting we set up a "forge.python.org" service that provides easier
> management of Mercurial repos that don't have the complex branching
> requirements of the main CPython repo. Think repos like the PEPs repo,
> or the developer guide repo.
> 
> The primary objective of the PEP is to enable an online editing + pull
> request style workflow for these pure documentation projects that only
> have a single active branch.
> 
> I'd been taking "must be hosted in PSF infrastructure" as a hard
> requirement, but MAL pointed out earlier this evening that in the age
> of DVCS's, that requirement may not make sense: if you avoid tightly
> coupling your automation to a particular DVCS host's infrastructure,
> then reverting back to self-hosting (if that becomes necessary for
> some reason) is mostly just a matter of "hg push".
> 
> If that "must be self-hosted" constraint is removed, then the obvious
> candidate for Mercurial hosting that supports online editing + pull
> requests is the PSF's BitBucket account.
> 
> There'd still be some work in such a change to make sure we didn't
> break automated regeneration of associated site elements, but that's
> still a lot simpler than adding an entirely new piece of
> infrastructure.
> 
> If folks are amenable to that variant of the idea, I'll undefer PEP
> 474 and revise it accordingly, with the developer guide and the PEP's
> repo as the initially proposed candidates for transfer.

I hate to say this, but if we're going to have "doc" repos hosted in a
different place than code, we might as bite the bullet and move them to
Git + GitHub. That would surely maximize the community size + ease of
use objective function.

Otherwise, moving the hosting will probably be more trouble
(administratively + infra) than it's worth.

I know it's hard to know, but do we have a lot of people comping at the
bit to submit PRs to the devguide repo?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?

[Benjamin Peterson]

On Fri, Nov 21, 2014, at 11:00, Donald Stufft wrote:
> 
> > On Nov 21, 2014, at 10:26 AM, Barry Warsaw  wrote:
> > 
> > On Nov 21, 2014, at 10:36 PM, Nick Coghlan wrote:
> > 
> >> I'd been taking "must be hosted in PSF infrastructure" as a hard
> >> requirement, but MAL pointed out earlier this evening that in the age
> >> of DVCS's, that requirement may not make sense: if you avoid tightly
> >> coupling your automation to a particular DVCS host's infrastructure,
> >> then reverting back to self-hosting (if that becomes necessary for
> >> some reason) is mostly just a matter of "hg push".
> >> 
> >> If that "must be self-hosted" constraint is removed, then the obvious
> >> candidate for Mercurial hosting that supports online editing + pull
> >> requests is the PSF's BitBucket account.
> > 
> > For the record, I object to moving *official* PSF resources to proprietary,
> > closed-source infrastructure that we do not control or have access to[*].
> > 
> > As nice and friendly as BitBucket or any other code hosting source is today,
> > there are many reasons why I think this is a bad idea for *official*
> > branches.  We are beholden to their policies and operations, which may not
> > align with PSF policies or principles today or in the future.  We will not 
> > be
> > able to customize the experience for our own needs.  We will not have access
> > to the underlying resources should we need them for any purpose.  We cannot
> > take action ourselves if some problem occurs, e.g. banning an offensive 
> > user.
> > 
> > You're right that in a world of dvcs, branches can be mirrored anywhere.  
> > For
> > that reason, I have no problem allowing developers to use non-PSF controlled
> > resources *unofficially* if it makes their work easier and doesn't conflict
> > with their own principles.  However, in such cases, I still believe that the
> > official, master, blessed repositories remain on PSF controlled
> > infrastructure.  Surely that too is possible in the world of dvcs, right?
> > 
> > Cheers,
> > -Barry
> > 
> > [*] Please note that I am not objecting to our use of lower-level resources
> > donated by our generous sponsors.  It's a fine line perhaps, but I have no
> > problem with a wiki running on a VM hosted on some donated hardware, since 
> > we
> > still have full access to the machine, the OS, and the application software.
> 
> Personally I care less about proprietary and closed-source and care a lot
> more
> about lock-in. Thus my big problem using Bitbucket for these things is
> that if
> we ever want to *leave* bitbucket it becomes a lot harder because you
> have a
> bunch of links and such pointing at bitbucket instead of a python.org
> domain.
> They do offer a redirect feature but that is dependent on them not taking
> that
> away in the future. (They also offer a CNAME feature but if you use it
> you lose
> the ability to use TLS, which is also a non starter for me). Sadly this
> also
> leaves out my favorite host site of Github :/. Something like Github
> Enterprise
> or Atlassian stash which are able to be migrated away from are better in
> this
> regards.
> 
> Ironically we do use a propetiary/closed-source/hosted solution for
> https://status.python.org/ but it’s not terribly difficult to migrate
> away from
> that if we ever wanted to.

The more significant one is probably Fastly.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] 2.7.9 delayed

[Benjamin Peterson]

As you may have noticed, 2.7.9rc1 wasn't tagged today. I've been much
busier than I expected to be, so I haven't had the time to fix up some
2.7.9 lose ends. Hopefully, the delay won't be more than a few days.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?

[Benjamin Peterson]

On Sun, Nov 23, 2014, at 01:25, Nick Coghlan wrote:
> On 23 November 2014 at 16:03, Donald Stufft  wrote:
> >> On Nov 23, 2014, at 12:59 AM, Nick Coghlan  wrote:
> >>
> >> Note that if folks prefer Git, BitBucket supports both. I would object
> >> strongly to unilaterally forcing existing contributors to switch from
> >> Mercurial to git.
> >
> > Going to all the trouble to move to an external repository and choosing the
> > least popular option out of the two main ones seems like a bad idea in
> > general.
> 
> Moving repos from hg.python.org to bitbucket.org is just a matter of
> switching some URLs around, and changing some backend systems to cope
> with the new location. The end result should be to make life better
> for existing contributors *and* new contributors using the web UI, and
> be largely transparent to folks using command line tools.
> 
> By contrast, proposals to switch from Mercurial to Git impose a
> *massive* burden on contributors that don't already know git.

But how many people are there who will have this massive burden imposed
on them? I imagine there's few among us who haven't had to learn git for
our job or other projects.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] ubuntu buildbot

[Benjamin Peterson]

Hi David,
I noticed you run the "Builder x86 Ubuntu Shared" buildbot. It seems
it's running a very old version of Ubuntu. Is there any chance of
getting that updated?

Regards,
Benjamin
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] ubuntu buildbot

[Benjamin Peterson]

On Mon, Nov 24, 2014, at 00:33, David Bolen wrote:
> Yeah, it definitely needs it.  Historically it was intentional as my own
> servers were all on 8.04, but the last of those moved 12.04 last year.
> 
> I think there's already a 12.04 buildbot, so perhaps 14.04 would be
> better?  I do prefer sticking with an LTS.

14.04 would be good.

> 
> It'll need to move to 64-bit given the hosting environment, at least for
> the kernel.  I could do 32-bit userspace via multiarch if keeping a
> 32-bit
> buildbot in the mix is attractive.

It'd be nice to keep a 32-bit bot around.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cpython (3.4): handle errors without a reason attribute

[Benjamin Peterson]

On Tue, Nov 25, 2014, at 19:16, Antoine Pitrou wrote:
> On Wed, 26 Nov 2014 00:06:06 +
> benjamin.peterson  wrote:
> 
> > https://hg.python.org/cpython/rev/e635c3ba75c8
> > changeset:   93591:e635c3ba75c8
> > branch:  3.4
> > user:Benjamin Peterson 
> > date:Tue Nov 25 15:43:58 2014 -0600
> > summary:
> >   handle errors without a reason attribute
> > 
> > files:
> >   Lib/test/support/__init__.py |  2 +-
> >   1 files changed, 1 insertions(+), 1 deletions(-)
> > 
> > 
> > diff --git a/Lib/test/support/__init__.py b/Lib/test/support/__init__.py
> > --- a/Lib/test/support/__init__.py
> > +++ b/Lib/test/support/__init__.py
> > @@ -698,7 +698,7 @@
> >  try:
> >  f(*args, **kwargs)
> >  except IOError as e:
> > -if e.reason == "CERTIFICATE_VERIFY_FAILED":
> > +if "CERTIFICATE_VERIFY_FAILED" in str(e):
> 
> You should be able to keep the e.reason test if you only catch SSLError.

Unfortunately, test_robotparser seems to manage to raise a cert
validation error without a reason.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] [RELEASE] Python 2.7.9 release candidate 1

[Benjamin Peterson]

I'm pleased to announce the first release candidate of Python 2.7.9,
which will be the next bugfix release in the Python 2.7 series. Despite
technically being a maintenance release, Python 2.7.9 will include
several majors changes from 2.7.8:
- The "ensurepip" module has been backported to Python 2.7.
- Python 3's ssl module has been backported to Python 2.7.
- HTTPS certificates are now verified by default using the system's
certificate store.
- SSLv3 has been disabled by default due to the POODLE attack.

Downloads are at
https://www.python.org/downloads/release/python-279rc1/

Application and library authors are encouraged test Python 2.7.9 release
candidate 1 with their code. This is especially important for 2.7.9 due
to significant changes mentioned above.

Please report bugs to
https://bugs.python.org/

Python 2.7.9 final is currently scheduled for December 10th.

Enjoy,
Benjamin
2.7 release manager
on behalf on python-dev and all of Python's contributors
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github

[Benjamin Peterson]

On Sun, Nov 30, 2014, at 11:45, Donald Stufft wrote:
> 
> > On Nov 30, 2014, at 11:28 AM, Brett Cannon  wrote:
> > 
> > 
> > 
> > On Sat Nov 29 2014 at 7:16:34 PM Alex Gaynor  > > wrote:
> > Donald Stufft  stufft.io > writes:
> > 
> > >
> > > [words words words]
> > >
> > 
> > I strongly support this PEP. I'd like to share two pieces of information. 
> > Both
> > of these are personal anecdotes:
> > 
> > For the past several years, I've been a contributor on two major projects 
> > using
> > mercurial, CPython and PyPy. PyPy has a strong culture of in-repo branching,
> > basically all contributors regularly make branches in the main repo for 
> > their
> > work, and we're very free in giving people commit rights, so almost everyone
> > working on PyPy in any way has this level of access. This workflow works 
> > ok. I
> > don't love it as much as git, but it's fine, it's not an impediment to my 
> > work.
> > 
> > By contrast, CPython does not have this type of workflow, there are almost 
> > no
> > in-tree branches besides the 2.7, 3.4, etc. ones. Despite being a regular hg
> > user for years, I have no idea how to create a local-only branch, or a 
> > branch
> > which is pushed to a remote (to use the git term). I also don't generally
> > commit my own work to CPython, even though I have push privledges,
> > because I
> > prefer to *always* get code review on my work. As a result, I use a git 
> > mirror
> > of CPython to do all my work, and generate patches from that.
> > 
> > The conclusion I draw from this is that hg's workflow is probably fine if
> > you're a committer on the project, or don't ever need to maintain multiple
> > patches concurrently (and thus can just leave everything uncommitted in the
> > repo). However, the hg workflow seems extremely defficient at non-committer
> > contributors.
> > 
> > One way to come close to that using hg is to have your own clone that you 
> > never push to hg.python.org/cpython  (e.g. 
> > cloning the Bitbucket clone of cpython or hosting on hg.python.org 
> >  a personal clone). You can then specify the repo 
> > and branch on the issue tracker to generate your patch: 
> > https://docs.python.org/devguide/triaging.html#mercurial-repository 
> >  . 
> > After that it's just like any other patch workflow for core devs. It's not 
> > quite as nice as maybe using named branches where you can just do a final 
> > hg merge/commit to get your changes committed, but if you're not going to 
> > commit your branches then you might as well get the automatic patch 
> > generation perk in the issue tracker rather than using git (unless there is 
> > some other git feature you use that you can't get in hg).
> 
> This doesn’t really work in a Pull Request workflow though I think is the
> point.
> Uploading patches is it’s own kind of terrible but yes if you’re just
> uploading
> patches you can probably do that. I don’t make branches in Mercurial
> because
> i’m afraid I’m going to push a permanent branch to hg.python.org
>  and screw
> something up.

Don't worry; we have a hook for that.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 2/3 porting HOWTO has been updated

[Benjamin Peterson]

On Fri, Dec 5, 2014, at 16:04, Brett Cannon wrote:
> It now promotes using tooling as much as possible to automate the process
> of making code by Python 2/3 source-compatible:
> https://docs.python.org/3.5/howto/pyporting.html

Are you going to update the 2.7 copy of the howto, too?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 2/3 porting HOWTO has been updated

[Benjamin Peterson]

On Fri, Dec 5, 2014, at 18:16, Donald Stufft wrote:
> > 
> > On Dec 5, 2014, at 6:10 PM, Brett Cannon  wrote:
> > 
> > 
> > 
> > On Fri Dec 05 2014 at 4:07:46 PM Benjamin Peterson  > <mailto:benja...@python.org>> wrote:
> > 
> > 
> > On Fri, Dec 5, 2014, at 16:04, Brett Cannon wrote:
> > > It now promotes using tooling as much as possible to automate the process
> > > of making code by Python 2/3 source-compatible:
> > > https://docs.python.org/3.5/howto/pyporting.html 
> > > <https://docs.python.org/3.5/howto/pyporting.html>
> > 
> > Are you going to update the 2.7 copy of the howto, too?
> > 
> > Have not decided yet. All the Google searches I have tried that bring up 
> > the HOWTO use the Python 3 version. Plus I know people are going to find 
> > mistakes that require fixing so I would rather wait until it stabilizes 
> > before I bother backporting to 2.7. 
> > ___
> > Python-Dev mailing list
> > Python-Dev@python.org
> > https://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe: 
> > https://mail.python.org/mailman/options/python-dev/donald%40stufft.io
> 
> 
> Do we need to update it? Can it just redirect to the 3 version?

Technically, yes, of course. However, that would unexpected take you out
of the Python 2 docs "context". Also, that doesn't solve the problem for
the downloadable versions of the docs.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] [RELEASE] Python 2.7.9

[Benjamin Peterson]

It is my pleasure to announce the release of Python 2.7.9, a new bugfix
release in the Python 2.7 series. Despite technically being a
maintenance release, Python 2.7.9 includes several majors changes from
2.7.8:

- The "ensurepip" module has been backported to Python 2.7
- Python 3's ssl module has been backported to Python 2.7.
- HTTPS certificates are now verified by default using the system's
certificate store.
- SSLv3 has been disabled by default due to the POODLE attack.

Downloads are at
https://www.python.org/downloads/release/python-279/

Please report bugs to
https://bugs.python.org/

I would  like to thank the people who made the above security and
usability improvements listed above possible. Among others, Alex Gaynor,
David Reid, Nick Coghlan, and Donald Stufft wrote many PEPs and a lot of
code to bring those features to 2.7.9. Thank you.

Enjoy,
Benjamin
2.7 release manager
on behalf on python-dev and all of Python's contributors
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] libffi embedded in CPython

[Benjamin Peterson]

On Thu, Dec 18, 2014, at 14:13, Maciej Fijalkowski wrote:
> After reading this http://bugs.python.org/issue23085 and remembering
> struggling having our own patches into cpython's libffi (but not into
> libffi itself), I wonder, is there any reason any more for libffi
> being included in CPython?

It has some sort of Windows related patches. No one seems to know
whether they're still needed for newer libffi. Unfortunately, ctypes
doesn't currently have a maintainer.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] libffi embedded in CPython

[Benjamin Peterson]

On Thu, Dec 18, 2014, at 14:50, Maciej Fijalkowski wrote:
> well, the problem is essentially that libffi gets patched (e.g. for
> ARM) and it does not make it's way to CPython quickly. This is
> unlikely to be a security issue (for a variety of reasons, including
> ctypes), but it's still an issue I think. Segfaults related to e.g.
> stack alignment are hard to debug

Certainly it's a suboptimal situation, but resolving it requires someone
to figure out whether we still need/want whatever patches are in there.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] libffi embedded in CPython

[Benjamin Peterson]

On Thu, Dec 18, 2014, at 14:50, Maciej Fijalkowski wrote:
> well, the problem is essentially that libffi gets patched (e.g. for
> ARM) and it does not make it's way to CPython quickly. This is
> unlikely to be a security issue (for a variety of reasons, including
> ctypes), but it's still an issue I think. Segfaults related to e.g.
> stack alignment are hard to debug

Certainly it's a suboptimal situation, but resolving it requires someone
to figure out whether we still need/want whatever patches are in there.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Redirection of ar.pycon.org

[Benjamin Peterson]

On Thu, Dec 18, 2014, at 10:59, Facundo Batista wrote:
> Hi!
> 
> Don't remember where to ask for changing the redirection of that
> domain name. Somebody knows?

Seems DNS for that is controlled by eGenix, so ccing mal.

(We should move pycon.org DNS to use the PSF's normal DNS
infrastructure.)

> 
> I need for the redirection to be to pycon.python.org.ar (and we'll
> take care of proper year-by-year redirection from there).
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] libffi embedded in CPython

[Benjamin Peterson]

On Thu, Dec 18, 2014, at 15:36, Jim J. Jewett wrote:
> 
> 
> On Thu, Dec 18, 2014, at 14:13, Maciej Fijalkowski wrote:
> > ... http://bugs.python.org/issue23085 ...
> > is there any reason any more for libffi being included in CPython?
> 
> [And why a fork, instead of just treating it as an external dependency]
> 
> Benjamin Peterson responded:
> 
> > It has some sort of Windows related patches. No one seems to know
> > whether they're still needed for newer libffi. Unfortunately, ctypes
> > doesn't currently have a maintainer.
> 
> Are any of the following false?
> 
> (1)  Ideally, we would treat it as an external dependency.
> 
> (2)  At one point, it was intentionally forked to get in needed
> patches, including at least some for 64 bit windows with MSVC.
> 
> (3)  Upstream libffi maintenance has picked back up.
> 
> (4)  Alas, that means the switch merge would not be trivial.
> 
> (5)  In theory, we could now switch to the external version.
> [In particular, does libffi have a release policy such that we
> could assume the newest released version is "safe", so long as
> our integration doesn't break?]
> 
> (6)  By its very nature, libffi changes are risky and undertested.
> At the moment, that is also true of its primary user, ctypes.
> 
> (7)  So a switch is OK in theory, but someone has to do the
> non-trivial testing and merging, and agree to support both libffi
> and and ctypes in the future.  Otherwise, stable wins.
> 
> (8)  The need for future support makes this a bad candidate for
> "patches wanted"/"bug bounty"/GSoC.

Sounds about right.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] cpython (3.4): improve incorrect French (#23109)

[Benjamin Peterson]

On Wed, Dec 24, 2014, at 15:59, Terry Reedy wrote:
> On 12/24/2014 2:59 PM, benjamin.peterson wrote:
> > https://hg.python.org/cpython/rev/2c87dd2d821e
> > changeset:   93958:2c87dd2d821e
> > branch:  3.4
> > parent:  93955:08972a47f710
> > user:Benjamin Peterson 
> > date:Wed Dec 24 13:58:05 2014 -0600
> > summary:
> >improve incorrect French (#23109)
> >
> > Following suggestions from Clément.
> >
> > files:
> >Doc/howto/unicode.rst |  4 ++--
> >1 files changed, 2 insertions(+), 2 deletions(-)
> >
> >
> > diff --git a/Doc/howto/unicode.rst b/Doc/howto/unicode.rst
> > --- a/Doc/howto/unicode.rst
> > +++ b/Doc/howto/unicode.rst
> > @@ -32,8 +32,8 @@
> >   In the mid-1980s an Apple II BASIC program written by a French speaker
> >   might have lines like these::
> >
> > -   PRINT "FICHIER EST COMPLETE."
> > -   PRINT "CARACTERE NON ACCEPTE."
> > +   PRINT "MISE A JOUR TERMINEE"
> > +   PRINT "PARAMETRES ENREGISTRES"
> >
> >   Those messages should contain accents (completé, caractère, accepté),
> 
> It seems that this list should have been changed also, to the words that 
> need accents in the replacement.

Good point. Thank you.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Google search labels Python 2.7 docs as Python 3.4

[Benjamin Peterson]

On Wed, Dec 31, 2014, at 19:32, Ryan Gonzalez wrote:
> Not sure if this is something to post here...but...
> 
> [image: Inline image 1]

That must be some sort of inconsistency in Google's index, since the
actual page's title is correct.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Why are generated files in the repository?

[Benjamin Peterson]

On Mon, Jan 19, 2015, at 19:40, Neil Girdhar wrote:
> I was also wondering why files like Python/graminit.c are in the
> respository?  They generate spurious merge conflicts.

Convenience mostly.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] How do I ensure that my code is being executed?

[Benjamin Peterson]

On Tue, Jan 20, 2015, at 09:51, Brett Cannon wrote:
> This is a mailing to discuss the development *of* Python, not its *use*.
> You should be able to get help from python-list or #python on IRC.

To be fair, he's asking to debug his patch in
https://bugs.python.org/issue2292
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 448 (almost finished!) — Question regarding test_ast

[Benjamin Peterson]

On Tue, Jan 20, 2015, at 11:34, Neil Girdhar wrote:
> My question first:
> test_ast is mostly generated code, but I can't find where it is being
> generated.  I am pretty sure I know how to fix most of the introduced
> problems.  Who is generating test_ast??

It generates itself.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 448 (almost finished!) — Question regarding test_ast

[Benjamin Peterson]

$ ./python Lib/test/test_ast.py -g
exec_results = [
('Module', [('Expr', (1, 0), ('NameConstant', (1, 0), None))]),
('Module', [('FunctionDef', (1, 0), 'f', ('arguments', [], None, [], [],
None, []), [('Pass', (1, 9))], [], None)]),
('Module', [('FunctionDef', (1, 0), 'f', ('arguments', [('arg', (1, 6),
'a', None)], None, [], [], None, []), [('Pass', (1, 10))], [], None)]),
('Module', [('FunctionDef', (1, 0), 'f', ('arguments', [('arg', (1, 6),
'a', None)], None, [], [], None, [('Num', (1, 8), 0)]), [('Pass', (1,
12))], [], None)]),
('Module', [('FunctionDef', (1, 0), 'f', ('arguments', [], ('arg', (1,
7), 'args', None), [], [], None, []), [('Pass', (1, 14))], [], None)]),
('Module', [('FunctionDef', (1, 0), 'f', ('arguments', [], None, [], [],
('arg', (1, 8), 'kwargs', None), []), [('Pass', (1, 17))], [], None)]),


On Tue, Jan 20, 2015, at 12:06, Neil Girdhar wrote:
> Hi Benjamin,
> 
> I'm having trouble finding where it is generating the lines below
> 
>  EVERYTHING BELOW IS GENERATED #
> 
> Neither a call to test_ast nor a make (in case it's generated somewhere
> else) regenerate those lines if they have been removed.
> 
> How were those lines generated?
> 
> Best,
> Neil
> 
> 
> On Tue, Jan 20, 2015 at 11:36 AM, Benjamin Peterson 
> wrote:
> 
> >
> >
> > On Tue, Jan 20, 2015, at 11:34, Neil Girdhar wrote:
> > > My question first:
> > > test_ast is mostly generated code, but I can't find where it is being
> > > generated.  I am pretty sure I know how to fix most of the introduced
> > > problems.  Who is generating test_ast??
> >
> > It generates itself.
> >
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Why are generated files in the repository?

[Benjamin Peterson]

On Sat, Jan 24, 2015, at 03:00, Nick Coghlan wrote:
> On 20 January 2015 at 10:53, Benjamin Peterson 
> wrote:
> >
> >
> > On Mon, Jan 19, 2015, at 19:40, Neil Girdhar wrote:
> >> I was also wondering why files like Python/graminit.c are in the
> >> respository?  They generate spurious merge conflicts.
> >
> > Convenience mostly.
> 
> It also gets us a round a couple of bootstrapping problems, where
> generating some of those files requires a working Python interpreter,
> which you may not have if you just cloned the source tree or unpacked
> the tarball.

We could distribute the generated files in tarballs as part of the
release process.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] (no subject)

[Benjamin Peterson]

On Mon, Feb 9, 2015, at 16:06, Neil Girdhar wrote:
> Hello all,
> 
> The updated PEP 448 (https://www.python.org/dev/peps/pep-0448/) is
> implemented now based on some early work by Thomas Wouters (in 2008) and
> Florian Hahn (2013) and recently completed by Joshua Landau and me.
> 
> The issue tracker http://bugs.python.org/issue2292  has  a working patch.
> Would someone be able to review it?

The PEP is not even accepted.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] (no subject)

[Benjamin Peterson]

On Mon, Feb 9, 2015, at 16:32, Guido van Rossum wrote:
> FWIW, I've encouraged Neil and others to complete this code as a
> prerequisite for a code review (but I can't review it myself). I am
> mildly
> in favor of the PEP -- if the code works and looks maintainable I would
> accept it. (A few things got changed in the PEP as a result of the work.)

In a way, it's a simplification, since functions are now simply called
with a sequence of "generalized arguments"; there's no privileged kwarg
or vararg. Of course, I wonder how much of f(**w, x, y, *k, *b, **d, c)
we would see...
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] (no subject)

[Benjamin Peterson]

On Mon, Feb 9, 2015, at 16:34, Ethan Furman wrote:
> On 02/09/2015 01:28 PM, Benjamin Peterson wrote:
> > On Mon, Feb 9, 2015, at 16:06, Neil Girdhar wrote:
> >>
> >> The updated PEP 448 (https://www.python.org/dev/peps/pep-0448/) is
> >> implemented now based on some early work by Thomas Wouters (in 2008) and
> >> Florian Hahn (2013) and recently completed by Joshua Landau and me.
> >>
> >> The issue tracker http://bugs.python.org/issue2292  has  a working patch.
> >> Would someone be able to review it?
> > 
> > The PEP is not even accepted.
> 
> I believe somebody (Guido?) commented "Why worry about accepting the PEP
> when there's no working patch?"  -- or
> something to that effect.

On the other hand, I'd rather not do detailed reviews of patches that
won't be accepted. :)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] (no subject)

[Benjamin Peterson]

On Mon, Feb 9, 2015, at 17:12, Neil Girdhar wrote:
> Right,
> 
> Just to be clear though:  **-args must follow any *-args and position
> arguments.  So at worst, your example is:
> 
> f(x, y, *k, *b, c,  **w, **d)
> 
> Best,

Ah, I guess I was confused by this sentence in the PEP: " Function calls
currently have the restriction that keyword arguments must follow
positional arguments and ** unpackings must additionally follow *
unpackings."

That suggests that that rule is going to change.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] cpython: Update copyright.

[Benjamin Peterson]

On Tue, Feb 10, 2015, at 23:37, raymond.hettinger wrote:
> https://hg.python.org/cpython/rev/7d826a6b92a1
> changeset:   94582:7d826a6b92a1
> user:Raymond Hettinger 
> date:Tue Feb 10 22:37:22 2015 -0600
> summary:
>   Update copyright.
> 
> files:
>   Modules/_collectionsmodule.c |  2 +-
>   1 files changed, 1 insertions(+), 1 deletions(-)
> 
> 
> diff --git a/Modules/_collectionsmodule.c b/Modules/_collectionsmodule.c
> --- a/Modules/_collectionsmodule.c
> +++ b/Modules/_collectionsmodule.c
> @@ -3,7 +3,7 @@
>  
>  /* collections module implementation of a deque() datatype
> Written and maintained by Raymond D. Hettinger 
> -   Copyright (c) 2004-2014 Python Software Foundation.
> +   Copyright (c) 2004-2015 Python Software Foundation.
> All rights reserved.
>  */

How about simply removing this line? The copyight years are well
documented in the LICENSE and README.txt, and it would save you the
trouble of keeping this up to date.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3.4 RPM on AIX

[Benjamin Peterson]

On Tue, Feb 17, 2015, at 23:06, Blaxton wrote:
> Hi Zach,
> I think it is best to remove the .spec file if it is not supported by
> Python developers.It is misleading to ship unsupported file within source
> tree.

I agree, so I just removed it.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Enum is really serializable using json?

[Benjamin Peterson]

On Sat, Feb 21, 2015, at 13:48, Facundo Batista wrote:
> We have this issue closed as resolved:
> 
>   http://bugs.python.org/issue18264
> 
> It's called "enum.IntEnum is not compatible with JSON serialisation",
> and it looks that after a long conversation they added proper support
> for it in Py3.4.
> 
> However, this is still failing:
> 
> Python 3.4.2 (default, Oct  8 2014, 13:18:07)
> [GCC 4.9.1] on linux
> >>> import enum, json
> >>> Foo = enum.Enum('Foo', 'bar baz')
> >>> json.dumps(Foo.bar)
> Traceback (most recent call last):
> ...
> TypeError:  is not JSON serializable
> 
> Am I failing to understand the bug tracker saying the patch was
> applied for Py3.4?

As the issue title suggests, IntEnum but not Enum supports JSON
serialization.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] ubuntu buildbot

[Benjamin Peterson]

On Wed, Feb 25, 2015, at 22:38, David Bolen wrote:
> On Mon, Nov 24, 2014 at 11:07 AM, Benjamin Peterson 
> wrote:
> >
> > On Mon, Nov 24, 2014, at 00:33, David Bolen wrote:
> >> Yeah, it definitely needs it.  Historically it was intentional as my own
> >> servers were all on 8.04, but the last of those moved 12.04 last year.
> >>
> >> I think there's already a 12.04 buildbot, so perhaps 14.04 would be
> >> better?  I do prefer sticking with an LTS.
> >
> > 14.04 would be good.
> >
> >> It'll need to move to 64-bit given the hosting environment, at least for
> >> the kernel.  I could do 32-bit userspace via multiarch if keeping a 32-bit
> >> buildbot in the mix is attractive.
> >
> > It'd be nice to keep a 32-bit bot around.
> 
> Took a bit longer than anticipated, but the slave upgrade is complete.
> 
> The bolen-ubuntu slave is now a 32-bit Ubuntu 14.04.2 LTS system.
> I've re-run the most recent 2.7, 3.4 and 3.x builds which all pass
> (though with a few new compiler warnings in some cases).

Thank you very much!
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 488: elimination of PYO files

[Benjamin Peterson]

On Fri, Mar 6, 2015, at 13:34, Brett Cannon wrote:
> On Fri, Mar 6, 2015 at 1:27 PM Neil Girdhar 
> wrote:
> 
> > On Fri, Mar 6, 2015 at 1:11 PM, Brett Cannon  wrote:
> >
> >>
> >>
> >> On Fri, Mar 6, 2015 at 1:03 PM Mark Shannon  wrote:
> >>
> >>>
> >>> On 06/03/15 16:34, Brett Cannon wrote:
> >>> > Over on the import-sig I proposed eliminating the concept of .pyo files
> >>> > since they only signify that /some/ optimization took place, not
> >>> > /what/ optimizations took place. Everyone on the SIG was positive with
> >>> > the idea so I wrote a PEP, got positive feedback from the SIG again,
> >>> and
> >>> > so now I present to you PEP 488 for discussion.
> >>> >
> >>> [snip]
> >>>
> >>> Historically -O and -OO have been the antithesis of optimisation, they
> >>> change the behaviour of the program with no noticeable effect on
> >>> performance.
> >>> If a change is to be made, why not just drop .pyo files and be done with
> >>> it?
> >>>
> >>
> >> I disagree with your premise that .pyo files don't have a noticeable
> >> effect on performance. If you don't use asserts a lot then there is no
> >> effect, but if you use them heavily or have them perform expensive
> >> calculations then there is an impact. And the dropping of docstrings does
> >> have an impact on memory usage when you use Python at scale.
> >>
> >> You're also assuming that we will never develop an AST optimizer that
> >> will go beyond what the peepholer can do based on raw bytecode, or
> >> something that involves a bit of calculation and thus something you
> >> wouldn't want to do at startup.
> >>
> >
> > I don't want to speak for him, but you're going to get the best results
> > optimizing ASTs at runtime, which is what I thought he was suggesting.
> > Trying to optimize Python at compile time is setting your sights really
> > low.   You have so little information then.
> >
> 
> OK, I don't want to derail the discussion of the PEP into one over how
> best
> to optimize CPython's performance relative to bytecode vs. runtime like
> PyPy. The point is that we have -O and -OO and people do have uses for
> those flags. People can also do custom optimizations thanks to the
> flexibility of loaders.

I think it would be preferable deprecate -O and -OO and replace them
with flags like --no-docstrings or --no-asserts. Ideally, "optimization"
levels shouldn't change program semantics.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 488: elimination of PYO files

[Benjamin Peterson]

On Fri, Mar 6, 2015, at 15:13, Chris Angelico wrote:
> On Sat, Mar 7, 2015 at 6:09 AM, Benjamin Peterson 
> wrote:
> > I think it would be preferable deprecate -O and -OO and replace them
> > with flags like --no-docstrings or --no-asserts. Ideally, "optimization"
> > levels shouldn't change program semantics.
> 
> Plenty of C compilers have optimization levels that can change
> behaviour (replacing division with mult-by-reciprocal and such), so I
> don't see any reason for Python to object.

Yes, but in the C world, you have pass scary flags like -ffast-math.
Just -O2 generally won't reach outside of standard semantics.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 488: elimination of PYO files

[Benjamin Peterson]

On Fri, Mar 6, 2015, at 15:11, Brett Cannon wrote:
> 
> OK, but that doesn't influence the PEP's goal of dropping .pyo files.

Correct.

> 
> Are you suggesting that the tag be changed to be less specific to
> optimizations and more free-form? Like
> `importlib.cpython-35.__no-asserts_no-docstrings__.pyc`? Otherwise stuff
> like this gets baked into the .pyc file itself instead of the file name,
> but I don't think we should just drop the ability to switch off asserts
> and
> docstrings like Mark seemed to be suggesting.

Basically, though the filename strings could perhaps be more compact.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Missing *-unpacking generalizations (issue 2292)

[Benjamin Peterson]

On Fri, Mar 20, 2015, at 19:16, Neil Girdhar wrote:
> Wow, this is an excellent review.  Thank you.
> 
> My only question is with respect to this:
> 
> I think there ought to be two opcodes; one for unpacking maps in
> function calls and another for literals. The whole function location
> thing is rather hideous.
> 
> What are the two opcodes?  BUILD_MAP_UNPACK and
> BUILD_MAP_UNPACK_WITH_CALL?
> 
> The first takes (n) a number of maps that it will merge, and the second
> does the same but also accepts (function_call_location) for the purpose
> of
> error reporting?

Exactly.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] Daily reference leaks (e10ad4d4d490): sum=333

[Benjamin Peterson]

On Sat, Apr 4, 2015, at 10:33, Brett Cannon wrote:
> Anyone know what is causing the deque leakage?

https://hg.python.org/cpython/rev/3409f4d945e8
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tracker user registration problem

[Benjamin Peterson]

Did you check spam?

On Sat, Apr 11, 2015, at 17:38, Guido van Rossum wrote:
> I am sitting net to Christie Wilson (bobcatfish) who is trying to
> register
> a user account for our issue tracker. However it doesn't seem to work --
> the promised email never arrives. Who can help with this?
> 
> -- 
> --Guido van Rossum (python.org/~guido)
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/benjamin%40python.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492: What is the real goal?

[Oscar Benjamin]

On 30 April 2015 at 09:50, Arnaud Delobelle  wrote:
>>
>> I'm flexible about how we name 'async def' functions.  I like
>> to call them "coroutines", because that's what they are, and
>> that's how asyncio calls them.  It's also convenient to use
>> 'coroutine-object' to explain what is the result of calling
>> a coroutine.
>
> I'd like the object created by an 'async def' statement to be called a
> 'coroutine function' and the result of calling it to be called a
> 'coroutine'.

That would be an improvement over the confusing terminology in the PEP
atm. The PEP proposes to name the inspect functions
inspect.iscoroutine() and inspect.iscoroutinefunction(). According to
the PEP iscoroutine() identifies "coroutine objects" and
iscoroutinefunction() identifies "coroutine functions" -- a term which
is not defined in the PEP but presumably means what the PEP calls a
"coroutine" in the glossary.

Calling the async def function an "async function" and the object it
returns a "coroutine" makes for the clearest terminology IMO (provided
the word coroutine is not also used for anything else). It would help
to prevent both experienced and new users from confusing the two
related but necessarily distinct concepts. Clearly distinct
terminology makes it easier to explain/discuss something if nothing
else because it saves repeating definitions all the time.


--
Oscar
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492: What is the real goal?

[Oscar Benjamin]

On 5 May 2015 at 17:48, Yury Selivanov  wrote:
>
> I've updated the PEP with some fixes of the terminology:
> https://hg.python.org/peps/rev/f156b272f860

Yes that looks better.

> I still think that 'coroutine functions' and 'coroutines'
> is a better pair than 'async functions' and 'coroutines'.

Fair enough. The terminology in the PEP seems consistent now which is
more important than the exact terms used.


--
Oscar
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] coming soon: 2.7.10

[Benjamin Peterson]

In the spirit of regular releases, it's time to release 2.7.10. I'm
going to plan to cut rc1 this weekend with a final in 2 weeks.

I apologize for the short notice; time has crept up on me, and I have
commitments in June that prevent pushing releases into that month.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] [RELEASE] Python 2.7.10 release candidate 1

[Benjamin Peterson]

It is my privilege to announce the first release candidate of 2.7.10,
the next bugfix release in the 2.7 series.

Downloads are at

   https://www.python.org/downloads/release/python-2710rc1/

The full changelog is at

   https://hg.python.org/cpython/raw-file/80ccce248ba2/Misc/NEWS

Please consider testing 2.7.10rc1 with your application and reporting
bugs to

   https://bugs.python.org

Regards,
Benjamin
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Fwd: [docs] Python documentation missing

[Benjamin Peterson]

On Wed, May 13, 2015, at 13:22, Sandro Tosi wrote:
> Hello,
> this happens every time we cut a RC release: the files referenced in
> the download section are missing and (rightfully so) people complain

In this case, it was because the Python 3.5 docs were failing to build.
Should be fixed in ~12 hours.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 552: deterministic pycs

[Benjamin Peterson]

Hello,
I've written a short PEP about an import extension to allow pycs to be
more deterministic by optional replacing the timestamp with a hash of
the source file: https://www.python.org/dev/peps/pep-0552/

Thanks for reading,
Benjamin

P.S. I came up with the idea for this PEP while awake.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 552: deterministic pycs

[Benjamin Peterson]

On Thu, Sep 7, 2017, at 14:00, Antoine Pitrou wrote:
> On Thu, 07 Sep 2017 13:39:21 -0700
> Benjamin Peterson  wrote:
> > Hello,
> > I've written a short PEP about an import extension to allow pycs to be
> > more deterministic by optional replacing the timestamp with a hash of
> > the source file: https://www.python.org/dev/peps/pep-0552/
> 
> Why isn't https://github.com/python/cpython/pull/296 a good enough
> solution to this problem?  It has a simple implementation, and requires
> neither maintaining two different pyc formats nor reading the entire
> source file to check whether the pyc file is up to date.

The main objection to that model is that it requires modifying source
timestamps, which isn't possible for builds on read-only source trees.
This proposal also allows reproducible builds even if the files are
being modified in an edit-run-tests cycle.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 552: deterministic pycs

[Benjamin Peterson]

On Thu, Sep 7, 2017, at 14:19, Freddy Rietdijk wrote:
> > The main objection to that model is that it requires modifying source
> timestamps, which isn't possible for builds on read-only source trees.
> 
> Why not set the source timestamps of the source trees to say 1 first?

If the source-tree is readonly (because you don't want your build system
to modify source files on principal), you cannot do that.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 552: deterministic pycs

[Benjamin Peterson]

On Thu, Sep 7, 2017, at 14:21, Antoine Pitrou wrote:
> On Thu, 07 Sep 2017 14:08:58 -0700
> Benjamin Peterson  wrote:
> > On Thu, Sep 7, 2017, at 14:00, Antoine Pitrou wrote:
> > > On Thu, 07 Sep 2017 13:39:21 -0700
> > > Benjamin Peterson  wrote:  
> > > > Hello,
> > > > I've written a short PEP about an import extension to allow pycs to be
> > > > more deterministic by optional replacing the timestamp with a hash of
> > > > the source file: https://www.python.org/dev/peps/pep-0552/  
> > > 
> > > Why isn't https://github.com/python/cpython/pull/296 a good enough
> > > solution to this problem?  It has a simple implementation, and requires
> > > neither maintaining two different pyc formats nor reading the entire
> > > source file to check whether the pyc file is up to date.  
> > 
> > The main objection to that model is that it requires modifying source
> > timestamps, which isn't possible for builds on read-only source trees.
> 
> Not sure how common that situation is (certainly the source tree wasn't
> read-only when you checked it out or untar'ed it), but isn't it easily
> circumvented by copying the source tree before building?

Well, yes, in these kind of "batch" build situations, copying is
probably fine. However, I want to be able to have pyc determinism even
when developing. Copying the entire source every time I change something
isn't a nice.

> 
> > This proposal also allows reproducible builds even if the files are
> > being modified in an edit-run-tests cycle.
> 
> I don't follow you here.  Could you elaborate?

If you require source timestamps to be fixed and deterministic, Python
won't notice when a file is modified.

The larger point is that while the SOURCE_EPOCH patch will likely work
for Linux distributions, I'm interested in being able to have
deterministic pycs in "normal" Python development workflows.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 552: deterministic pycs

[Benjamin Peterson]

On Thu, Sep 7, 2017, at 14:19, Guido van Rossum wrote:
> Nice one.
> 
> It would be nice to specify the various APIs needed as well.

The compileall and py_compile ones?

> 
> Why do you keep the mtime-based format as an option? (Maybe because it's
> faster? Did you measure it?)

I haven't actually measured anything, but stating a file will definitely
be faster than reading it completely and hashing it. I suppose if the
speed difference between timestamp-based and hash-based pycs turned out
to be small we could feel good about dropping the timestamp format
completely. However, that difference might be hard to determine
definitely as I expect the speed hit will vary widely based on system
parameters such as disk speed and page cache size.

My goal in this PEP was to preserve the current pyc invalidation
behavior, which works well today for many use cases, as the default. The
hash-based pycs are reserved for distribution and other power use cases.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 552: deterministic pycs

[Benjamin Peterson]

On Thu, Sep 7, 2017, at 14:54, Antoine Pitrou wrote:
> On Thu, 07 Sep 2017 14:32:19 -0700
> Benjamin Peterson  wrote:
> > > 
> > > Not sure how common that situation is (certainly the source tree wasn't
> > > read-only when you checked it out or untar'ed it), but isn't it easily
> > > circumvented by copying the source tree before building?  
> > 
> > Well, yes, in these kind of "batch" build situations, copying is
> > probably fine. However, I want to be able to have pyc determinism even
> > when developing. Copying the entire source every time I change something
> > isn't a nice.
> 
> Hmm... Are you developing from a read-only source tree?

No, but the build system is building from one (at least conceptually).

> 
> > The larger point is that while the SOURCE_EPOCH patch will likely work
> > for Linux distributions, I'm interested in being able to have
> > deterministic pycs in "normal" Python development workflows.
> 
> That's an interesting idea, but is there a concrete motivation or is it
> platonical?  After all, if you're changing something in the source tree
> it's expected that the overall "signature" of the build will be
> modified too.

Yes, I have used Bazel to build pycs. Having pycs be deterministic
allows interesting build system optimizations like Bazel distributed
caching to work well for Python.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 552: deterministic pycs

[Benjamin Peterson]

On Thu, Sep 7, 2017, at 14:43, Guido van Rossum wrote:
> On Thu, Sep 7, 2017 at 2:40 PM, Benjamin Peterson 
> wrote:
> 
> >
> >
> > On Thu, Sep 7, 2017, at 14:19, Guido van Rossum wrote:
> > > Nice one.
> > >
> > > It would be nice to specify the various APIs needed as well.
> >
> > The compileall and py_compile ones?
> >
> 
> Yes, and the SipHash mod to specify the key you mentioned.

Done.

> 
> >
> > > Why do you keep the mtime-based format as an option? (Maybe because it's
> > > faster? Did you measure it?)
> >
> > I haven't actually measured anything, but stating a file will definitely
> > be faster than reading it completely and hashing it. I suppose if the
> > speed difference between timestamp-based and hash-based pycs turned out
> > to be small we could feel good about dropping the timestamp format
> > completely. However, that difference might be hard to determine
> > definitely as I expect the speed hit will vary widely based on system
> > parameters such as disk speed and page cache size.
> >
> > My goal in this PEP was to preserve the current pyc invalidation
> > behavior, which works well today for many use cases, as the default. The
> > hash-based pycs are reserved for distribution and other power use cases.
> >
> 
> OK, maybe you can clarify that a bit in the PEP.

I've added a paragraph to the Rationale section.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 552: deterministic pycs

[Benjamin Peterson]

On Thu, Sep 7, 2017, at 16:58, Gregory P. Smith wrote:
> +1 on this PEP.

Thanks!

> Questions:
> 
> Input from OS package distributors would be interesting.  Would they use
> this?  Which way would it impact their startup time (loading the .py file
> vs just statting it.  does that even matter?  source files are often
> eventually loaded for linecache use in tracebacks anyways)?

I an anticipate distributors will use the mode where the pyc is simply
trusted and the source file isn't hashed. That would make the io
overhead identical to today.

> 
> Would they benefit from a pyc that can contain _both_ timestamp+length,
> and
> the source_hash?  if both were present, I assume that only one would be
> checked at startup.  i'm not sure what would make the decision of what to
> check.  one fails, check the other?  i personally do not have a use for
> this case so i'd omit the complexity without a demonstrated need.

Yeah, it could act as a multi-tiered cache key. I agree with your
conclusion to pass for now.

> 
> Something to also state in the PEP:
> 
> This is intentionally not a "secure" hash.  Security is explicitly a
> non-goal.

Added a sentence.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 552: deterministic pycs

[Benjamin Peterson]

Thank you all for the feedback. I've now updated the PEP to specify a
4-word pyc header with a bit field in every case.

On Fri, Sep 8, 2017, at 09:43, Nick Coghlan wrote:
> On 8 September 2017 at 07:55, Antoine Pitrou  wrote:
> > On Fri, 8 Sep 2017 07:49:46 -0700
> > Nick Coghlan  wrote:
> >> > I'd rather a single magic number and a separate bitfield that tells
> >> > what the header encodes exactly.  We don't *have* to fight for a tiny
> >> > size reduction of pyc files.
> >>
> >> One of Benjamin's goals was for the existing timestamp-based pyc
> >> format to remain completely unchanged, so we need some kind of marker
> >> in the magic number to indicate whether the file is using the new
> >> format or nor.
> >
> > I don't think that's a useful goal, as long as we bump the magic number.
> 
> Yeah, we (me, Benjamin, Greg) discussed that here, and we agree -
> there isn't actually any benefit to keeping the timestamp based pyc's
> using the same layout, since the magic number is already going to
> change anyway.
> 
> Given that, I think your suggested 16 byte header layout would be a
> good one: 4 byte magic number, 4 bytes reserved for format flags, 8
> bytes with an interpretation that depends on the format flags.
> 
> Cheers,
> Nick.
> 
> -- 
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/benjamin%40python.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 556: Threaded garbage collection

[Benjamin Peterson]

I like it overall.

- I was wondering what happens during interpreter shutdown. I see you
have that listed as a open issue. How about simply shutting down the
finalization thread and not guaranteeing that finalizers are actually
ever run à la Java?

- Why not run all (Python) finalizers on the thread and not just ones
from cycles?

On Fri, Sep 8, 2017, at 08:05, Antoine Pitrou wrote:
> 
> Hello,
> 
> I've written a PEP by which you can tell the GC to run in a dedicated
> thread.  The goal is to solve reentrancy issues with finalizers:
> https://www.python.org/dev/peps/pep-0556/
> 
> Regards
> 
> Antoine.
> 
> PS: I did not come up with the idea for this PEP while other people
> were having nightmares.  Any nightmares involved in this PEP are
> fictional, and any resemblance to actual nightmares is purely
> coincidental.  No nightmares were harmed while writing this PEP.
> 
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/benjamin%40python.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 556: Threaded garbage collection

[Benjamin Peterson]

On Fri, Sep 8, 2017, at 12:24, Larry Hastings wrote:
> 
> 
> On 09/08/2017 12:04 PM, Benjamin Peterson wrote:
> > - Why not run all (Python) finalizers on the thread and not just ones
> > from cycles?
> 
> Two reasons:
> 
>  1. Because some code relies on the finalizer being called on the thread
> where the last reference is dropped.  This is usually the same
> thread where the object was created.  Some irritating third-party
> libraries make demands on callers, e.g. "you can only interact with
> / destroy X objects on your 'main thread'". This is often true of
> windowing / GUI libraries.  (For example, I believe this was true of
> Direct3D, at least as of D3D8; it was also often true of Win32 USER
> objects.)

Is the requirement that the construction and destruction be literally on
the same thread or merely non-concurrent? The GIL would provide the
latter.

>  2. Because there's so much work there.  In the Gilectomy prototype, I
> originally called all finalizers on the "reference count manager
> commit thread", the thread that also committed increfs and decrefs. 
> The thread immediately fell behind on its queue and never caught
> up.  I changed the Gilectomy so objects needing finalization are
> passed back to the thread where the last decref happened, for
> finalization on that thread; this was pleasingly self-balancing.

I'm only suggesting Python-level __del__ methods be run on the separate
thread not general deallocation work. I would those would be few and far
between.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 556: Threaded garbage collection

[Benjamin Peterson]

On Fri, Sep 8, 2017, at 12:13, Antoine Pitrou wrote:
> On Fri, 08 Sep 2017 12:04:10 -0700
> Benjamin Peterson  wrote:
> > I like it overall.
> > 
> > - I was wondering what happens during interpreter shutdown. I see you
> > have that listed as a open issue. How about simply shutting down the
> > finalization thread and not guaranteeing that finalizers are actually
> > ever run à la Java?
> 
> I don't know.  People generally have expectations towards stuff being
> finalized properly (especially when talking about files etc.).
> Once the first implementation is devised, we will know more about
> what's workable (perhaps we'll have to move _PyGC_Fini earlier in the
> shutdown sequence?  perhaps we'll want to switch back to serial mode
> when shutting down?).

Okay, I'm curious to know what ends up happening here then.

> 
> > - Why not run all (Python) finalizers on the thread and not just ones
> > from cycles?
> 
> Because a lot of code probably expects them to be run as soon as the
> last visible ref disappears.

But this assumption is broken on PyPy and sometimes already by CPython,
so I don't feel very bad moving away from it.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] [RELEASE] Python 2.7.14

[Benjamin Peterson]

I'm happy to announce to the immediate availability of Python 2.7.14,
yet another bug fix release in the Python 2.7 series. 2.7.14 includes 9
months of conservative bug fixes from the 3.x branch.

Downloads of source code and binaries are at:
https://www.python.org/downloads/release/python-2714/

Bugs may be reported at
https://bugs.python.org/

Warmly,
Benjamin
2.7 release manager
(on behalf of all of 2.7's contributors)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Intention to accept PEP 552 soon (deterministic pyc files)

[Benjamin Peterson]

Thanks, Guido and everyone who submitted feedback!

I guess I know what I'll be doing this weekend. 


On Fri, Sep 29, 2017, at 08:18, Guido van Rossum wrote:
> Let me try that again.
>
> There have been no further comments. PEP 552 is now accepted.
>
> Congrats, Benjamin! Go ahead and send your implementation for
> review.Oops.
> Let me try that again.
>
> PS. PEP 550 is still unaccepted, awaiting a new revision from Yury and> Elvis.
>
> --
> --Guido van Rossum (python.org/~guido <http://python.org/%7Eguido>)
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/benjamin%40python.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Intention to accept PEP 552 soon (deterministic pyc files)

[Benjamin Peterson]

What do you mean by bytecode-specific APIs? The internal importlib ones?

On Fri, Sep 29, 2017, at 09:38, Brett Cannon wrote:
> BTW, if you find the bytecode-specific APIs are sub-par while trying to
> update them, let me know as I have been toying with cleaning them up and
> centralizing them under importlib for a while and just never gotten
> around
> to sitting down and coming up with a better design that warranted putting
> the time into it. :)
> 
> On Fri, 29 Sep 2017 at 09:17 Benjamin Peterson 
> wrote:
> 
> > Thanks, Guido and everyone who submitted feedback!
> >
> > I guess I know what I'll be doing this weekend.
> >
> >
> > On Fri, Sep 29, 2017, at 08:18, Guido van Rossum wrote:
> > > Let me try that again.
> > >
> > > There have been no further comments. PEP 552 is now accepted.
> > >
> > > Congrats, Benjamin! Go ahead and send your implementation for
> > > review.Oops.
> > > Let me try that again.
> > >
> > > PS. PEP 550 is still unaccepted, awaiting a new revision from Yury and
> > > Elvis.
> > >
> > > --
> > > --Guido van Rossum (python.org/~guido <http://python.org/%7Eguido>)
> > > ___
> > > Python-Dev mailing list
> > > Python-Dev@python.org
> > > https://mail.python.org/mailman/listinfo/python-dev
> > > Unsubscribe:
> > > https://mail.python.org/mailman/options/python-dev/benjamin%40python.org
> >
> > ___
> > Python-Dev mailing list
> > Python-Dev@python.org
> > https://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> > https://mail.python.org/mailman/options/python-dev/brett%40python.org
> >
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Intention to accept PEP 552 soon (deterministic pyc files)

[Benjamin Peterson]

On Tue, Oct 3, 2017, at 08:03, Barry Warsaw wrote:
> Guido van Rossum wrote:
> > There have been no further comments. PEP 552 is now accepted.
> > 
> > Congrats, Benjamin! Go ahead and send your implementation for review.Oops.
> > Let me try that again.
> 
> While I'm very glad PEP 552 has been accepted, it occurs to me that it
> will now be more difficult to parse the various pyc file formats from
> Python.  E.g. I used to be able to just open the pyc in binary mode,
> read all the bytes, and then lop off the first 8 bytes to get to the
> code object.  With the addition of the source file size, I now have to
> (maybe, if I have to also read old-style pyc files) lop off the front 12
> bytes, but okay.
> 
> With PEP 552, I have to do a lot more work to just get at the code
> object.  How many bytes at the front of the file do I need to skip past?
>  What about all the metadata at the front of the pyc, how do I interpret
> that if I want to get at it from Python code?

As Guido points out, the header is just now always 4 32-bit words rather
than 3. Not long ago we underwent the transition from 2-3 words without
widespread disaster.

> 
> Should the PEP 552 implementation add an API, probably to
> importlib.util, that would understand all current and future formats?
> Something like this perhaps?
> 
> class PycFileSpec:
> magic_number: bytes
> timestamp: Optional[bytes] # maybe an int? datetime?
> source_size: Optional[bytes]
> bit_field: Optional[bytes]
> code_object: bytes
> 
> def parse_pyc(path: str) -> PycFileSpec:

I'm not sure turning the implementation details of our internal formats
into APIs is the way to go.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Intention to accept PEP 552 soon (deterministic pyc files)

[Benjamin Peterson]

On Wed, Oct 4, 2017, at 07:14, Barry Warsaw wrote:
> On Oct 3, 2017, at 13:29, Benjamin Peterson  wrote:
> 
> > I'm not sure turning the implementation details of our internal formats
> > into APIs is the way to go.
> 
> I still think an API in the stdlib would be useful and appropriate, but
> it’s not like this couldn’t be done as a 3rd party module.

It might be helpful to enumerate the usecases for such an API. Perhaps a
narrow, specialized API could satisfy most needs in a supportable way.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Broken svn lookup?

[Benjamin Peterson]

I turned viewvc off a few months ago because subversion is highly
deprecated at this point. In fact, now that Windows build dependencies
have moved off, I’m probably going to shut it off for good soon.

On Mon, Dec 18, 2017, at 06:55, Victor Stinner wrote:
> Hi,
>
> I was looking at old issues. In
> https://bugs.python.org/issue8610#msg105805 I found the link:
>
>   http://svn.python.org/view?view=revision&revision=81190
>
> Sadly, the link fails with HTTP 404 Not Found :-(
>
> Is the service down? It's no more possible to browse the old
> Subversion repository?
>
> Victor

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Heap allocate type structs in native extension modules?

[Benjamin Peterson]

I imagine Cython already takes care of this?


On Tue, Dec 26, 2017, at 02:16, Hugh Fisher wrote:
> I have a Python program which generates the boilerplate code for
> native extension modules from a Python source definition.
> (http://bitbucket.org/hugh_fisher/fullofeels if interested.)
>
> The examples in the Python doco and the "Python Essential Reference"
> book all use a statically declared PyTypeObject struct and
> PyType_Ready in the module init func, so I'm doing the same. Then
> Python 3.5 added a check for statically allocated types inheriting
> from heap types, which broke a couple of my classes. And now I'm
> trying to add a __dict__ to native classes so end users can add their> own 
> attributes, and this is turning out to be painful with static
> PyTypeObject structs
>
> Would it be better to use dynamically allocated type structs in native
> modules?>
> --
>
> cheers,
> Hugh Fisher
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/benjamin%40python.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Heap allocate type structs in native extension modules?

[Benjamin Peterson]

On Thu, Dec 28, 2017, at 03:29, Erik Bray wrote:
> On Tue, Dec 26, 2017 at 3:00 PM, Benjamin Peterson  
> wrote:
> > I imagine Cython already takes care of this?
> 
> This appears to have a distinct purpose, albeit not unrelated to
> Cython.  The OP's program would generate boilerplate C code for
> extension types the rest of which would perhaps be implemented by hand
> in C.  Cython does this as well to an extent, but the generated code
> contains quite a bit of Cython-specific cruft and is not really meant
> to be edited by hand or read by humans in most cases.

It still seems the OP is likely to reinvent a lot of Cython. One option is to 
write a bunch of "pure" .c and then only have the Python bindings in Cython.

> 
> Anyways I don't think this answers the OP's question.

I think this belongs on python-list anyway.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Whatever happened to 'nonlocal x = y'?

[Benjamin Peterson]

On Fri, Jan 5, 2018, at 01:57, Nathaniel Smith wrote: 
> Was this just an oversight, or did it get rejected at some point and
> no-one remembered to update that PEP?

There was an implementation https://bugs.python.org/issue4199. But several 
years ago, we again reached the conclusion that the feature shouldn't be added.

Normally, I think final PEPs shouldn't be updated. But maybe in this case, it's 
worth deleting those lines to avoid future confusion.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Whatever happened to 'nonlocal x = y'?

[Benjamin Peterson]

https://github.com/python/peps/commit/2d2ac2d2b66d4e37e8b930f5963735616bddbbe8

On Sat, Jan 6, 2018, at 08:56, Barry Warsaw wrote:
> On Jan 6, 2018, at 11:43, Guido van Rossum  wrote:
> > 
> > Maybe we should not delete them outright but add something like "(UPDATE: 
> > during later discussions it was decided that this feature shouldn't be 
> > added.)"
> 
> +1
> 
> -Barry
> 
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/benjamin%40python.org
> Email had 1 attachment:
> + signature.asc
>   1k (application/pgp-signature)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Drop support for old unsupported FreeBSD and Linux kernels?

[Benjamin Peterson]

+1 to both of your specific proposals.

More generally, I think it makes good sense to allow dropping support for a 
platform in the next major Python release after vendor support for the platform 
stops. Even we say we support something, it will break quickly without buildbot 
validation.

On Thu, Jan 18, 2018, at 12:27, Victor Stinner wrote:
> Hi,
> 
> I'm working on a exhaustive list of platforms supported by Python:
> 
>   http://vstinner.readthedocs.io/cpython.html#supported-platforms
> 
> 
> I noticed that the extended support phase of Windows Vista is expired,
> so I proposed to drop Vista support:
> 
>   "Drop support of Windows Vista in Python 3.7"
>   https://bugs.python.org/issue32592
>   https://github.com/python/cpython/pull/5231
> 
> Python has an explicit policy for Windows support, extract of the PEP 11:
> 
> "CPython’s Windows support now follows [Microsoft product support
> lifecycle]. A new feature release X.Y.0 will support all Windows
> releases whose extended support phase is not yet expired. Subsequent
> bug fix releases will support the same Windows releases as the
> original feature release (even if the extended support phase has
> ended)."
> 
> 
> For Linux and FreeBSD, we have no explicit rule. CPython code base
> still contains code for FreeBSD 4... but FreeBSD 4 support ended
> longer than 10 years ago (January 31, 2007). Maybe it's time to drop
> support of these old platforms to cleanup the CPython code base to
> ease its maintainance.
> 
> I proposed: "Drop FreeBSD 9 and older support:"
> 
>   https://bugs.python.org/issue32593
>   https://github.com/python/cpython/pull/5232
> 
> FreeBSD 9 supported ended 1 year ago (December 2016).
> 
> FreeBSD support:
> 
>   https://www.freebsd.org/security/
>   https://www.freebsd.org/security/unsupported.html
> 
> 
> CPython still has compatibility code for Linux 2.6, whereas the
> support of Linux 2.6.x ended in August 2011, longer than 6 years ago.
> Should we also drop support for old Linux kernels? If yes, which ones?
> The Linux kernel has LTS version, the oldest is Linux 3.2 (support
> will end in May, 2018).
> 
> Linux kernel support:
> 
>   https://www.kernel.org/category/releases.html
> 
> 
> Note: I'm only talking about changing the future Python 3.7. We should
> have the same support policy than for Windows. If Python 3.x.0
> supports a platform, this support should be kept in the whole lifetime
> of the 3.x cycle (until it's end-of-line).
> 
> Victor
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/benjamin%40python.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 2.7, long double vs allocator alignment, GCC 8 on x86-64

[Benjamin Peterson]

Yes, changing obmalloc.c's alignment guarantees would definitely be the easiest 
solution. I think someone just needs to investigate whether it wastes a lot of 
memory.

On Tue, Jan 30, 2018, at 13:22, Gregory P. Smith wrote:
> I'm curious if changing the obmalloc.c ALIGNMENT and ALIGNMENT_SHIFT
> defines is sufficient to avoid ABI breakage.
> 
> -gps
> 
> On Tue, Jan 30, 2018 at 1:20 PM Gregory P. Smith  wrote:
> 
> > The proper fix for this in the code would likely break ABI compatibility
> > (ie: not possible in python 2.7 or any other stable release).
> >
> > Clang's UBSAN (undefined behavior sanitizer) has been flagging this one
> > for a long time.
> >
> > In Python 3 a double is used instead of long double since 2012 as I did
> > some digging at the time:
> > https://github.com/python/cpython/commit/e348c8d154cf6342c79d627ebfe89dfe9de23817
> >
> > -gps
> >
> > On Tue, Jan 30, 2018 at 10:59 AM Florian Weimer  wrote:
> >
> >> I hope this is the right list for this kind of question.  We recently
> >> tried to build Python 2.6 with GCC 8, and ran into this issue:
> >>
> >>   <https://bugzilla.redhat.com/show_bug.cgi?id=1540316>
> >>
> >> Also quoting for context:
> >>
> >> | PyInstance_NewRaw contains this code:
> >> |
> >> | inst = PyObject_GC_New(PyInstanceObject, &PyInstance_Type);
> >> | if (inst == NULL) {
> >> | Py_DECREF(dict);
> >> | return NULL;
> >> | }
> >> | inst->in_weakreflist = NULL;
> >> | Py_INCREF(klass);
> >> | inst->in_class = (PyClassObject *)klass;
> >> | inst->in_dict = dict;
> >> | _PyObject_GC_TRACK(inst);
> >> |
> >> | _PyObject_GC_TRACK expands to:
> >> |
> >> | #define _PyObject_GC_TRACK(o) do { \
> >> | PyGC_Head *g = _Py_AS_GC(o); \
> >> | if (g->gc.gc_refs != _PyGC_REFS_UNTRACKED) \
> >> | Py_FatalError("GC object already tracked"); \
> >> | …
> >> |
> >> | Via:
> >> |
> >> | #define _Py_AS_GC(o) ((PyGC_Head *)(o)-1)
> >> |
> >> | We get to this:
> >> |
> >> | /* GC information is stored BEFORE the object structure. */
> >> | typedef union _gc_head {
> >> | struct {
> >> | union _gc_head *gc_next;
> >> | union _gc_head *gc_prev;
> >> | Py_ssize_t gc_refs;
> >> | } gc;
> >> | long double dummy;  /* force worst-case alignment */
> >> | } PyGC_Head;
> >> |
> >> | PyGC_Head has 16-byte alignment.  The net result is that
> >> |
> >> | _PyObject_GC_TRACK(inst);
> >> |
> >> | promises to the compiler that inst is properly aligned for the
> >> | PyGC_Head type, but it is not: PyObject_GC_New returns a pointer which
> >> | is only 8-byte-aligned.
> >> |
> >> | Objects/obmalloc.c contains this:
> >> |
> >> | /*
> >> |  * Alignment of addresses returned to the user. 8-bytes alignment works
> >> |  * on most current architectures (with 32-bit or 64-bit address busses).
> >> |  * The alignment value is also used for grouping small requests in size
> >> |  * classes spaced ALIGNMENT bytes apart.
> >> |  *
> >> |  * You shouldn't change this unless you know what you are doing.
> >> |  */
> >> | #define ALIGNMENT   8   /* must be 2^N */
> >> | #define ALIGNMENT_SHIFT 3
> >> | #define ALIGNMENT_MASK  (ALIGNMENT - 1)
> >> |
> >> | So either the allocator alignment needs to be increased, or the
> >> | PyGC_Head alignment needs to be decreased.
> >>
> >> Is this a known issue?  As far as I can see, it has not been fixed on
> >> the 2.7 branch.
> >>
> >> (Store merging is a relatively new GCC feature.  Among other things,
> >> this means that on x86-64, for sufficiently aligned pointers, vector
> >> instructions are used to update multiple struct fields at once.  These
> >> vector instructions can trigger alignment traps, similar to what
> >> happens on some other architectures for scalars.)
> >> ___
> >> Python-Dev mailing list
> >> Python-Dev@python.org
> >> https://mail.python.org/mailman/listinfo/python-dev
> >> Unsubscribe:
> >> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
> >>
> >
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/benjamin%40python.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com