Re: [Python-Dev] Python 2.7 patch levels turning two digit

[Donald Stufft]

their extensions with a
> 2008 compiler, when the rest of the world has long moved on.
> 
> -- 
> Marc-Andre Lemburg
> eGenix.com
> 
> Professional Python Services directly from the Source  (#1, Jun 21 2014)
> >>> Python Projects, Consulting and Support ...   http://www.egenix.com/
> >>> mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
> >>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/
> 
> 2014-06-17: Released eGenix PyRun 2.0.0 ...   http://egenix.com/go58
> 2014-06-09: Released eGenix pyOpenSSL 0.13.3 ...  http://egenix.com/go57
> 2014-07-02: Python Meeting Duesseldorf ... 11 days to go
> 
>eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
> D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
>Registered at Amtsgericht Duesseldorf: HRB 46611
>http://www.egenix.com/company/contact/
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/donald%40stufft.io


-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 2.7 patch levels turning two digit

[Donald Stufft]

On Jun 23, 2014, at 2:09 AM, Martin v. Löwis  wrote:

>> 
>> * Should we make use of the potential breakage with 2.7.10
>>   to introduce a new Windows compiler version for Python 2.7 ?
> 
> Assuming it is a good idea to continue producing Windows binaries
> for 2.7, I think it would be a bad idea to switch compilers. It will
> cause severe breakage of 2.7 installations, much more problematic
> than switching to two-digit version numbers.

I agree with this, we’ve just finally started getting things to the point where
it makes a lot of sense for binary distributions for Windows. Breaking all
of them on 2.7 would be very bad.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 2.7 patch levels turning two digit

[Donald Stufft]

On Jun 23, 2014, at 3:27 PM, M.-A. Lemburg  wrote:

> On 23.06.2014 18:09, Donald Stufft wrote:
>> 
>> On Jun 23, 2014, at 2:09 AM, Martin v. Löwis  wrote:
>> 
>>>> 
>>>> * Should we make use of the potential breakage with 2.7.10
>>>>  to introduce a new Windows compiler version for Python 2.7 ?
>>> 
>>> Assuming it is a good idea to continue producing Windows binaries
>>> for 2.7, I think it would be a bad idea to switch compilers. It will
>>> cause severe breakage of 2.7 installations, much more problematic
>>> than switching to two-digit version numbers.
>> 
>> I agree with this, we’ve just finally started getting things to the point 
>> where
>> it makes a lot of sense for binary distributions for Windows. Breaking all
>> of them on 2.7 would be very bad.

Err, sorry that “We” was with my pip hat on.

> 
> Not sure what you mean. We've had binary wininst distributions
> for Windows for more than a decade, and egg and msi distributions
> for 8 years :-)

Nonetheless, changing the compiler will not only break pip, but every
automated installer tool (easy_install, buildout) that i’m aware of. The
blow back for binary installation is going to be huge I think.

> 
> But without access to the VS 2008 compiler that is needed to
> compile those extensions, it will become increasingly difficult
> for package authors to provide such binary packages, so we have to
> ask ourselves:
> 
> What's worse: breaking old Windows binaries for Python 2.7
> or not having updated and new Windows binaries for Python 2.7
> at all in a few years ?

At the risk of getting Guido to post his slide again, I still think the
solution to the old compiler is to just roll a 2.8 with minimal changes.
It could even be a good place to move to the ssl backport changes
too since they were the riskier set of changes in PEP466.

But either way, if a compiler does change in a 2.7 release we’ll need
to update a lot of tooling to cope with that, so any plan to do that should
include that and a timeline for adoption of that.

> 
> Switching to a newer compiler will make things easier for everyone
> and we'd see more binary packages for Windows again.
> 
> Given that Python 2.7 support was extended for another 5 years at the
> recent Python Language Summit to 2020, we have to face this
> breakage sooner or later anyway. Extended support for VS 2008
> will end in 2018 (but then: Python developers usually don't have
> extended support contracts with MS). Service pack support has already
> ended in 2009.
> 
> Depending on how you see it, using such an old compiler also
> poses security risks. The last security update for VS 2008 dates
> back to 2011 (http://support.microsoft.com/kb/2538243).
> 
> -- 
> Marc-Andre Lemburg
> eGenix.com
> 
> Professional Python Services directly from the Source  (#1, Jun 23 2014)
>>>> Python Projects, Consulting and Support ...   http://www.egenix.com/
>>>> mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
>>>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/
> 
> 2014-06-17: Released eGenix PyRun 2.0.0 ...   http://egenix.com/go58
> 2014-07-02: Python Meeting Duesseldorf ...  9 days to go
> 2014-07-21: EuroPython 2014, Berlin, Germany ...   28 days to go
> 
>   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
>D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
>   Registered at Amtsgericht Duesseldorf: HRB 46611
>   http://www.egenix.com/company/contact/


-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 2.7 patch levels turning two digit

[Donald Stufft]

On Jun 23, 2014, at 4:31 PM, Barry Warsaw  wrote:

> On Jun 23, 2014, at 04:20 PM, Donald Stufft wrote:
> 
>> At the risk of getting Guido to post his slide again, I still think the
>> solution to the old compiler is to just roll a 2.8 with minimal changes.
> 
> No.  It's not going to happen, for all the reasons discussed previously.
> Python 2.8 is not a solution to anything.
> 
> If a new, incompatible compiler suite is required, why can't there just be
> multiple Windows downloads on https://www.python.org/download/releases/2.7.7/
> ?  Well, on reason is that you'd have to convince MvL or someone else to take
> over the work that would require, but that's gotta be *much* lighter weight
> than releasing a Python 2.8.

As far as I am aware, a 2.7 with a different compiler, even if it’s just an 
option
is an attractive nuisance. None of the tooling right now differentiates between
binary compatibility by anything other than “CPython 2.7”.

The end result of having a 2.7 which is built with the old compiler, and a 2.7 
built
with the new compiler is that you’ll end up with binary distributions which work
sometimes if you’re lucky and the creator of the binary distribution and you
happened to pick the same “variant” of 2.7. Most likely result is all the binary
distributions will *mostly* still depend on using the old compiler because of 
the
corpus of existing binary packages that depend on that. Which means that the
2.7 with new compiler will exist entirely to act as a footgun to anyone who 
picks
it and also wants to use binary packages.

> 
> -Barry
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/donald%40stufft.io


-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 2.7 patch levels turning two digit

[Donald Stufft]

On Jun 23, 2014, at 4:31 PM, Martin v. Löwis  wrote:

>> 
>> Would that mitigate the pain, assuming that 
>> Steve (or someone else) would be willing to build the additional 
>> installers for the transition period?  I've done something similar on a 
>> smaller scale with the OS X 32-bit installer for 2.7.x but that impact 
>> is much less as the audience for that installer is much smaller.
> 
> Well, the question really is whether precompiled extension modules
> available from PyPI would work on both compilers. I understand that
> for OSX, you typically don't have precompiled binaries for the extension
> modules, so installation compiles the modules from scratch. This is
> easier, as it can use the ABI of the Python which will be installed
> to.
> 
> If you go the "parallel ABIs" route, extension authors have to provide
> two parallel sets of packages as well. Given 32-bit and 64-bit packages,
> this will make actually two additional packages - just as if they had
> to support another Python version.

As far as I know, stuff on OSX is generally built for “X compiler or later”
so binary compatibility is kept as long as you’re using an “or later” but
I could be wrong about that. Using binary packages on OSX is a much
less frequent thing I think though since getting a working compiler toolchain
is easier there.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 2.7 patch levels turning two digit

[Donald Stufft]

On Jun 23, 2014, at 5:07 PM, M.-A. Lemburg  wrote:

> On 23.06.2014 22:20, Donald Stufft wrote:
>> 
>> On Jun 23, 2014, at 3:27 PM, M.-A. Lemburg  wrote:
>> 
>>> On 23.06.2014 18:09, Donald Stufft wrote:
>>>> 
>>>> On Jun 23, 2014, at 2:09 AM, Martin v. Löwis  wrote:
>>>> 
>>>>>> 
>>>>>> * Should we make use of the potential breakage with 2.7.10
>>>>>> to introduce a new Windows compiler version for Python 2.7 ?
>>>>> 
>>>>> Assuming it is a good idea to continue producing Windows binaries
>>>>> for 2.7, I think it would be a bad idea to switch compilers. It will
>>>>> cause severe breakage of 2.7 installations, much more problematic
>>>>> than switching to two-digit version numbers.
>>>> 
>>>> I agree with this, we’ve just finally started getting things to the point 
>>>> where
>>>> it makes a lot of sense for binary distributions for Windows. Breaking all
>>>> of them on 2.7 would be very bad.
>> 
>> Err, sorry that “We” was with my pip hat on.
>> 
>>> 
>>> Not sure what you mean. We've had binary wininst distributions
>>> for Windows for more than a decade, and egg and msi distributions
>>> for 8 years :-)
>> 
>> Nonetheless, changing the compiler will not only break pip, but every
>> automated installer tool (easy_install, buildout) that i’m aware of. The
>> blow back for binary installation is going to be huge I think.
>> 
>>> But without access to the VS 2008 compiler that is needed to
>>> compile those extensions, it will become increasingly difficult
>>> for package authors to provide such binary packages, so we have to
>>> ask ourselves:
>>> 
>>> What's worse: breaking old Windows binaries for Python 2.7
>>> or not having updated and new Windows binaries for Python 2.7
>>> at all in a few years ?
>> 
>> At the risk of getting Guido to post his slide again, I still think the
>> solution to the old compiler is to just roll a 2.8 with minimal changes.
>> It could even be a good place to move to the ssl backport changes
>> too since they were the riskier set of changes in PEP466.
>> 
>> But either way, if a compiler does change in a 2.7 release we’ll need
>> to update a lot of tooling to cope with that, so any plan to do that should
>> include that and a timeline for adoption of that.
> 
> Sure, and we'd need to hash out possible solutions to minimize
> breakage, but first we'll have to see whether we want to consider
> this step or not.
> 
> 
> BTW: It's strange that I'm arguing for breaking things. I'm usually
> on the other side of such arguments :-)

Ok. I’m just making sure that any proposal to do this includes how
it plans to work around/minimize that. I agree with Martin (I think)
that trying to fix the entire ecosystem to cope with that change is
going to be far more work than folks realize and that it needs to
be an explicit part of the discussion when deciding how to solve
the problem.

Normally when I see someone suggest that switching compilers
in 2.7.x is likely to be less work than releasing a 2.8 It normally
appears to me they haven’t looked at the impact on the packaging
tooling.

> 
> -- 
> Marc-Andre Lemburg
> eGenix.com
> 
> Professional Python Services directly from the Source
>>>> Python/Zope Consulting and Support ...http://www.egenix.com/
>>>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
>>>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/
> 
> 
> ::: Try our new mxODBC.Connect Python Database Interface for free ! 
> 
> 
>   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
>D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
>   Registered at Amtsgericht Duesseldorf: HRB 46611
>   http://www.egenix.com/company/contact/


-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 2.7 patch levels turning two digit

[Donald Stufft]

On Jun 23, 2014, at 5:22 PM, Barry Warsaw  wrote:

> On Jun 23, 2014, at 05:15 PM, Donald Stufft wrote:
> 
>> Normally when I see someone suggest that switching compilers
>> in 2.7.x is likely to be less work than releasing a 2.8 It normally
>> appears to me they haven’t looked at the impact on the packaging
>> tooling.
> 
> Just to be clear, releasing a Python 2.8 has enormous impact outside of just
> the amount of work to do it.  It's an exceedingly bad idea.

Can you clarify?

Also FWIW I’m not really married to the 2.8 thing, it’s mostly that, on 
Windows, the X.Y release
prior to the ABI thing in 3.x _was_ the ABI so all the tooling builds on that. 
So you need to
either

1) Stick with the old Compiler
2) Release 2.8
3) Do all the work to fix all the tooling to cope with the fact that X.Y isn’t 
the ABI on 2.x anymore

I don’t think a reasonable option is:

4) Just switch compilers and leave it on someone else’s doorsteps to fix the 
entire packaging
tool chain to cope.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 2.7 patch levels turning two digit

[Donald Stufft]

On Jun 23, 2014, at 5:48 PM, Chris Angelico  wrote:

> On Tue, Jun 24, 2014 at 6:42 AM, "Martin v. Löwis"  wrote:
>> See my other message. It's actually heavier, since it requires changes
>> to distutils, PyPI, pip, buildout etc., all which know how to deal with
>> Python minor version numbers, but are unaware of the notion of competing
>> ABIs on Windows (except that they know how to deal with 32-bit vs. 64-bit).
> 
> Is it possible to hijack the "deal with 32-bit vs 64-bit"ness of
> things to handle the different compilers? So, for instance, there
> might be a "32-bit-NewCompiler" and a "64-bit-NewCompiler", two new
> architectures, just as if someone came out with a 128-bit Windows and
> built Python 2.7 for it. Would packaging be able to handle that more
> easily than a compiler change within the same architecture?
> 
> ChrisA
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/donald%40stufft.io

I’m not sure about this FWIW. I’d have to look at the implementations of
stuff to see how they’d cope with a new thing like that.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] buildbot.python.org down again?

[Donald Stufft]

On Jul 8, 2014, at 12:58 AM, Nick Coghlan  wrote:

> 
> On 7 Jul 2014 10:47, "Guido van Rossum"  wrote:
> >
> > It would still be nice to know who "the appropriate persons" are. Too much 
> > of our infrastructure seems to be maintained by house elves or the ITA.
> 
> I volunteered to be the board's liaison to the infrastructure team, and 
> getting more visibility around what the infrastructure *is* and how it's 
> monitored and supported is going to be part of that. That will serve a couple 
> of key purposes:
> 
> - making the points of escalation clearer if anything breaks or needs 
> improvement (although "infrastruct...@python.org" is a good default choice)
> - making the current "todo" list of the infrastructure team more visible 
> (both to calibrate resolution time expectations and to provide potential 
> contributors an idea of what's involved)
> 
> Noah has already set up http://status.python.org/ to track service status, I 
> can see about getting buildbot.python.org added to the list.
> 
> Cheers,
> Nick.
> 
> 

We (the infrastructure team) were actually looking earlier about
buildbot.python.org and we’re not entirely sure who "owns" buildbot.python.org.
Unfortunately a lot of the *.python.org services are in a similar state where
there is no clear owner. Generally we've not wanted to just step in and take
over for fear of stepping on someones toes but it appears that perhaps
buildbot.p.o has no owner?

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 467: Minor API improvements for bytes & bytearray

[Donald Stufft]

> On Aug 17, 2014, at 1:07 PM, Raymond Hettinger  
> wrote:
> 
> 
> On Aug 17, 2014, at 1:41 AM, Nick Coghlan  <mailto:ncogh...@gmail.com>> wrote:
> 
>> If I see "bytearray(10)" there is nothing there that suggests "this
>> creates an array of length 10 and initialises it to zero" to me. I'd
>> be more inclined to guess it would be equivalent to "bytearray([10])".
>> 
>> "bytearray.zeros(10)", on the other hand, is relatively clear,
>> independently of user expectations.
> 
> Zeros would have been great but that should have been done originally.
> The time to get API design right is at inception.
> Now, you're just breaking code and invalidating any published examples.
> 
>>> 
>>> Another thought is that the core devs should be very reluctant to deprecate
>>> anything we don't have to while the 2 to 3 transition is still in progress.
>>> Every new deprecation of APIs that existed in Python 2.7 just adds another
>>> obstacle to converting code.  Individually, the differences are trivial.
>>> Collectively, they present a good reason to never migrate code to Python 3.
>> 
>> This is actually one of the inconsistencies between the Python 2 and 3
>> binary APIs:
> 
> However, bytearray(n) is the same in both Python 2 and Python 3.
> Changing it in Python 3 increases the gulf between the two.
> 
> The further we let Python 3 diverge from Python 2, the less likely that
> people will convert their code and the harder you make it to write code
> that runs under both.
> 
> FWIW, I've been teaching Python full time for three years.  I cover the
> use of bytearray(n) in my classes and not a single person out of 3000+
> engineers have had a problem with it.   I seriously question the PEP's
> assertion that there is a real problem to be solved (i.e. that people
> are baffled by bytearray(bufsiz)) and that the problem is sufficiently
> painful to warrant the headaches that go along with API changes.
> 
> The other proposal to add bytearray.byte(3) should probably be named
> bytearray.from_byte(3) for clarity.  That said, I question whether there is
> actually a use case for this.   I have never seen seen code that has a
> need to create a byte array of length one from a single integer.
> For the most part, the API will be easiest to learn if it matches what
> we do for lists and for array.array.
> 
> Sorry Nick, but I think you're making the API worse instead of better.
> This API isn't perfect but it isn't flat-out broken either.   There is some
> unfortunate asymmetry between bytes() and bytearray() in Python 2,
> but that ship has sailed.  The current API for Python 3 is pretty good
> (though there is still a tension between wanting to be like lists and like
> strings both at the same time).
> 
> 
> Raymond
> 
> 
> P.S.  The most important problem in the Python world now is getting
> Python 2 users to adopt Python 3.  The core devs need to develop
> a strong distaste for anything that makes that problem harder.
> 

For the record I’ve had all of the problems that Nick states and I’m
+1 on this change.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 467: Minor API improvements for bytes & bytearray

[Donald Stufft]

> On Aug 17, 2014, at 5:19 PM, Raymond Hettinger  
> wrote:
> 
> 
> On Aug 17, 2014, at 11:33 AM, Ethan Furman  <mailto:et...@stoneleaf.us>> wrote:
> 
>> I've had many of the problems Nick states and I'm also +1.
> 
> There are two code snippets below which were taken from the standard library.
> Are you saying that:
> 1) you don't understand the code (as the pep suggests)
> 2) you are willing to break that code and everything like it
> 3) and it would be more elegantly expressed as:  
> charmap = bytearray.zeros(256)
> and
> mapping = bytearray.zeros(256)
> 
> At work, I have network engineers creating IPv4 headers and other structures
> with bytearrays initialized to zeros.  Do you really want to break all their 
> code?
> No where else in Python do we create buffers that way.  Code like
> "msg, who = s.recvfrom(256)" is the norm.
> 
> Also, it is unclear if you're saying that you have an actual use case for this
> part of the proposal?
> 
>ba = bytearray.byte(65)
> 
> And than the code would be better, clearer, and faster than the currently 
> working form?
> 
>ba = bytearray([65])
> 
> Does there really need to be a special case for constructing a single byte?
> To me, that is akin to proposing "list.from_int(65)" as an important special
> case to replace "[65]".
> 
> If you must muck with the ever changing bytes() API, then please 
> leave the bytearray() API alone.  I think we should show some respect
> for code that is currently working and is cleanly expressible in both
> Python 2 and Python 3.  We aren't winning users with API churn.
> 
> FWIW, I guessing that the differing view points in the thread stem
> mainly from the proponents experiences with bytes() rather than
> from experience with bytearray() which doesn't seem to have any
> usage problems in the wild.  I've never seen a developer say they
> didn't understand what "buf = bytearray(1024)" means.   That is
> not an actual problem that needs solving (or breaking).
> 
> What may be an actual problem is code like "char = bytes(1024)"
> though I'm unclear what a user might have actually been trying
> to do with code like that.

I think this is probably correct. I generally don’t think that bytes(1024)
makes much sense at all, especially not as a default constructor. Most likely
it exists to be similar to bytearray().

I don't have a specific problem with bytearray(1024), though I do think it's
more elegantly and clearly described as bytearray.zeros(1024), but not by much.

I find bytes.byte()/bytearray to be needed as long as there isn't a simple way
to iterate over a bytes or bytearray in a way that yields bytes or bytearrays
instead of integers. To be honest I can't think of a time when I'd actually
*want* to iterate over a bytes/bytearray as integers. Although I realize there
is unlikely to be a reasonable method to change that now. If iterbytes is added
I'm not sure where i'd personally use either bytes.byte() or bytearray.byte().

In general though I think that overloading a single constructor method to do
something conceptually different based on the type of the parameter leads to
these kind of confusing scenarios and that having differently named constructors
for the different concepts is far clearer.

So given all that, I am:

* +1 for some method of iterating over both types as bytes instead of
  integers.
* +1 on adding .zeros to both types as an alternative and preferred method of
  creating a zero filled instance and deprecating the original method[1].
* -0 on adding .byte to both types as an alternative method of creating a
  single byte instance.
* -1 On changing the meaning of bytearray(1024).
* +/-0 on changing the meaning of bytes(1024), I think that bytes(1024) is
  likely to *not* be what someone wants and that what they really want is
  bytes([N]). I also think that the number one reason for someone to be doing
  bytes(N) is because they were attempting to iterate over a bytes or bytearray
  object and they got an integer. I also think that it's bad that this changes
  from 2.x to 3.x and I wish it hadn't. However I can't decide if it's worth
  reverting this at this time or not.

[1] By deprecating I mean, raise a deprecation warning, or something but my
thoughts on actually removing the other methods are listed explicitly.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 4000 to explicitly declare we won't be doing a Py3k style compatibility break again?

[Donald Stufft]

On Sun, Aug 17, 2014, at 09:02 PM, Guido van Rossum wrote:
> On Sun, Aug 17, 2014 at 6:29 AM, Barry Warsaw  wrote:
>> On Aug 16, 2014, at 07:43 PM, Guido van Rossum wrote:
>>  
>> 
>(Don't understand this to mean that we should never deprecate things.
>> 
>Deprecations will happen, they are necessary for the evolution of any
>> 
>programming language. But they won't ever hurt in the way that Python 3
>> 
>hurt.)
>>  
>> It would be useful to explore what causes the most pain in the 2->3
>> 
transition?  IMHO, it's not the deprecations or changes such as print ->
>> 
print().  It's the bytes/str split - a fundamental change to core and
common
>> 
data types.  The question then is whether you foresee any similar
looming
>> 
pervasive change? [*]
> 
> I'm unsure about what's the single biggest pain moving to Python 3. In the 
> past I would have said that it's for sure the bytes/str split (which both the 
> biggest pain and the biggest payoff).
>  
> But if I look carefully into the soul of teams that are still on 2.7 (I know 
> a few... :-), I think the real reason is that Python 3 changes so many 
> different things, you have to actually understand your code to port it 
> (unlike with minor version transitions, where the changes usually spike in 
> one specific area, and you can leave the rest to normal attrition and 
> periodic maintenance).
>  

In my experience bytes/str is the single biggest change that causes the
most problems. Most of the other changes can be mechanically transformed
and/or papered over using helpers like six. The bytes/str change is the
main one that requires understanding code and where it requires a
serious untangling of things in code bases where str/bytes are freely
used intechangingbly. Often times this requires making a decision about
what *should* be bytes or str as well which requires having some deep
knowledge about the APIs in question too.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Fwd: PEP 467: Minor API improvements for bytes & bytearray

[Donald Stufft]

from __future__ import bytesdoneright? :D

-- 
  Donald Stufft
  don...@stufft.io

On Sun, Aug 17, 2014, at 09:40 PM, Antoine Pitrou wrote:
> Le 17/08/2014 20:08, Nick Coghlan a écrit :
> >
> > On 18 Aug 2014 09:57, "Barry Warsaw"  > <mailto:ba...@python.org>> wrote:
> >  >
> >  > On Aug 18, 2014, at 09:12 AM, Nick Coghlan wrote:
> >  >
> >  > >I'm talking more generally - do you *really* want to be explaining that
> >  > >"bytes" behaves like a tuple of integers, while "bytes.bytes"
> > behaves like
> >  > >a tuple of bytes?
> >  >
> >  > I would explain it differently though, using concrete examples.
> >  >
> >  > data = bytes(...)
> >  > for i in data: # iterate over data as integers
> >  > for i in data.bytes: # iterate over data as bytes
> >  >
> >  > But whatever.  I just wish there was something better than iterbytes.
> >
> > There's actually another aspect to your idea, independent of the naming:
> > exposing a view rather than just an iterator.
> 
> So that view would actually be the bytes object done right? Funny :-)
> Will it have lazy slicing?
> 
> Regards
> 
> Antoine.
> 
> 
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/donald%40stufft.io
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 476: Enabling certificate validation by default!

[Donald Stufft]

> On Aug 29, 2014, at 4:00 PM, "M.-A. Lemburg"  wrote:
> 
> * choice of trusted certificate:
> 
>   Instead of hard wiring using the system CA roots into
>   Python it would be good to just make this default and
>   permit the user to point Python to a different set of
>   CA roots.
> 
>   This would enable using self signed certs more easily.
>   Since these are often used for tests, demos and education,
>   I think it's important to allow having more control of
>   the trusted certs.

If I recall OpenSSL already allows this to be configured via envvar and the 
python API already allows it to be configured via API. 
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 476: Enabling certificate validation by default!

[Donald Stufft]

Sorry I was on my phone and didn’t get to fully reply to this.

> On Aug 29, 2014, at 4:00 PM, M.-A. Lemburg  wrote:
> 
> On 29.08.2014 21:47, Alex Gaynor wrote:
>> Hi all,
>> 
>> I've just submitted PEP 476, on enabling certificate validation by default 
>> for
>> HTTPS clients in Python. Please have a look and let me know what you think.
>> 
>> PEP text follows.
> 
> Thanks for the PEP. I think this is generally a good idea,
> but some important parts are missing from the PEP:
> 
> * transition plan:
> 
>   I think starting with warnings in Python 3.5 and going
>   for exceptions in 3.6 would make a good transition
> 
>   Going straight for exceptions in 3.5 is not in line with
>   our normal procedures for backwards incompatible changes.

As far as a transition plan, I think that this is an important
enough thing to have an accelerated process. If we need
to provide a warning than let’s add it to the next 3.4 otherwise
it’s going to be 2.5+ years until we stop being unsafe by
default.

Another problem with this is that I don’t think it’s actually
possible to do. Python itself isn’t validating the TLS certificates,
OpenSSL is doing that. To my knowledge OpenSSL doesn’t
have a way to say “please validate these certificates and if
they don’t validate go ahead and keep going and just let me
get a warning from it”. It’s a 3 way switch, no validation, validation
if a certificate is provided, and validation always.

Now that’s strictly for the “verify the certificate chain” portion,
the hostname verification is done entirely on our end and we
could do something there… but I’m not sure it makes sense
to do so if we can’t do it for invalid certificates too.

> 
> * configuration:
> 
>   It would be good to be able to switch this on or off
>   without having to change the code, e.g. via a command
>   line switch and environment variable; perhaps even
>   controlling whether or not to raise an exception or
>   warning.

I’m on the fence about this, if someone provides a certificate
that we can validate against (which can be done without
touching the code) then the only thing that really can’t be
“fixed” without touching the code is if someone has a certificate
that is otherwise invalid (expired, not yet valid, wrong hostname,
etc). I’d say if I was voting on this particular thing I’d be -0, I’d
rather it didn’t exist but I wouldn’t cry too much if it did.

> 
> * choice of trusted certificate:
> 
>   Instead of hard wiring using the system CA roots into
>   Python it would be good to just make this default and
>   permit the user to point Python to a different set of
>   CA roots.
> 
>   This would enable using self signed certs more easily.
>   Since these are often used for tests, demos and education,
>   I think it's important to allow having more control of
>   the trusted certs.


Like my other email said, the Python API has everything needed
to easily specify your own CA roots and/or disable the validations.
The OpenSSL library also allows you to specify either a directory
or a file to change the root certificates without code changes. The
only real problems with the APIs are that the default is bad and
an unrelated thing where you can’t pass in an in memory certificate.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 476: Enabling certificate validation by default!

[Donald Stufft]

> On Aug 29, 2014, at 5:42 PM, R. David Murray  wrote:
> 
> On Fri, 29 Aug 2014 17:11:35 -0400, Donald Stufft  wrote:
>> Sorry I was on my phone and didn’t get to fully reply to this.
>>> On Aug 29, 2014, at 4:00 PM, M.-A. Lemburg  wrote:
>>> 
>>> * configuration:
>>> 
>>>  It would be good to be able to switch this on or off
>>>  without having to change the code, e.g. via a command
>>>  line switch and environment variable; perhaps even
>>>  controlling whether or not to raise an exception or
>>>  warning.
>> 
>> I’m on the fence about this, if someone provides a certificate
>> that we can validate against (which can be done without
>> touching the code) then the only thing that really can’t be
>> “fixed” without touching the code is if someone has a certificate
>> that is otherwise invalid (expired, not yet valid, wrong hostname,
>> etc). I’d say if I was voting on this particular thing I’d be -0, I’d
>> rather it didn’t exist but I wouldn’t cry too much if it did.
> 
> Especially if you want an accelerated change, there must be a way to
> *easily* get back to the previous behavior, or we are going to catch a
> lot of flack.  There may be only 7% of public certs that are problematic,
> but I'd be willing to bet you that there are more not-really-public ones
> that are critical to day to day operations *somewhere* :)
> 
> wget and curl have 'ignore validation' as a command line flag for a reason.
> 

Right, that’s why I’m on the fence :)

On one hand, it’s going to break things for some people, (arguably they are
already broken, just silently so, but we’ll leave that argument aside) and a
way to get back the old behavior is good. There are already ways within
the Python code itself, so that’s covered. From outside of the Python code
there are ways if the certificate is untrusted but otherwise valid which are
pretty easy to do. The major “gap” is when you have an actual invalid
certificate due to expiration or hostname or some other such thing.

On the other hand Python is not wget/curl and the people who are most
likely to be the target for a “I can’t change the code but I need to get the
old behavior back” are people who are likely to not be invoking Python
itself but using something written in Python which happens to be using
Python. IOW they might be executing “foobar” not “python -m foobar”.

Like I said though, I’m personally fine either way so don’t take this as
being against that particular change!

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 476: Enabling certificate validation by default!

[Donald Stufft]

> On Aug 29, 2014, at 5:58 PM, M.-A. Lemburg  wrote:
> 
> On 29.08.2014 23:11, Donald Stufft wrote:
>> 
>> Sorry I was on my phone and didn’t get to fully reply to this.
>> 
>>> On Aug 29, 2014, at 4:00 PM, M.-A. Lemburg  wrote:
>>> 
>>> On 29.08.2014 21:47, Alex Gaynor wrote:
>>>> Hi all,
>>>> 
>>>> I've just submitted PEP 476, on enabling certificate validation by default 
>>>> for
>>>> HTTPS clients in Python. Please have a look and let me know what you think.
>>>> 
>>>> PEP text follows.
>>> 
>>> Thanks for the PEP. I think this is generally a good idea,
>>> but some important parts are missing from the PEP:
>>> 
>>> * transition plan:
>>> 
>>>  I think starting with warnings in Python 3.5 and going
>>>  for exceptions in 3.6 would make a good transition
>>> 
>>>  Going straight for exceptions in 3.5 is not in line with
>>>  our normal procedures for backwards incompatible changes.
>> 
>> As far as a transition plan, I think that this is an important
>> enough thing to have an accelerated process. If we need
>> to provide a warning than let’s add it to the next 3.4 otherwise
>> it’s going to be 2.5+ years until we stop being unsafe by
>> default.
> 
> Fine with me; we're still early in the Python 3.4
> patch level releases.
> 
>> Another problem with this is that I don’t think it’s actually
>> possible to do. Python itself isn’t validating the TLS certificates,
>> OpenSSL is doing that. To my knowledge OpenSSL doesn’t
>> have a way to say “please validate these certificates and if
>> they don’t validate go ahead and keep going and just let me
>> get a warning from it”. It’s a 3 way switch, no validation, validation
>> if a certificate is provided, and validation always.
>> 
>> Now that’s strictly for the “verify the certificate chain” portion,
>> the hostname verification is done entirely on our end and we
>> could do something there… but I’m not sure it makes sense
>> to do so if we can’t do it for invalid certificates too.
> 
> OpenSSL provides a callback for certificate validation,
> so it is possible to issue a warning and continue with
> accepting the certificate.

Ah right, I forgot about that. I was thinking in terms of CERT_NONE,
CERT_OPTIONAL, CERT_REQUIRED. I think it’s fine to add a warning
if possible to Python 3.4, I just couldn’t think off the top of my head
a way of doing it.

> 
>>> * configuration:
>>> 
>>>  It would be good to be able to switch this on or off
>>>  without having to change the code, e.g. via a command
>>>  line switch and environment variable; perhaps even
>>>  controlling whether or not to raise an exception or
>>>  warning.
>> 
>> I’m on the fence about this, if someone provides a certificate
>> that we can validate against (which can be done without
>> touching the code) then the only thing that really can’t be
>> “fixed” without touching the code is if someone has a certificate
>> that is otherwise invalid (expired, not yet valid, wrong hostname,
>> etc). I’d say if I was voting on this particular thing I’d be -0, I’d
>> rather it didn’t exist but I wouldn’t cry too much if it did.
> 
> If you're testing code or trying out some new stuff, you
> don't want to get a valid cert first, but instead go ahead
> with a self signed one. That's the use case.
> 
>>> * choice of trusted certificate:
>>> 
>>>  Instead of hard wiring using the system CA roots into
>>>  Python it would be good to just make this default and
>>>  permit the user to point Python to a different set of
>>>  CA roots.
>>> 
>>>  This would enable using self signed certs more easily.
>>>  Since these are often used for tests, demos and education,
>>>  I think it's important to allow having more control of
>>>  the trusted certs.
>> 
>> 
>> Like my other email said, the Python API has everything needed
>> to easily specify your own CA roots and/or disable the validations.
>> The OpenSSL library also allows you to specify either a directory
>> or a file to change the root certificates without code changes. The
>> only real problems with the APIs are that the default is bad and
>> an unrelated thing where you can’t pass in an in memory certificate.
> 
> Are you sure that's possible ? Python doesn't load the
> openssl.cnf file and the SSL_CERT_FILE, SSL_CERT_DIR env
> vars only work for the openssl command line binary

Re: [Python-Dev] PEP 476: Enabling certificate validation by default!

[Donald Stufft]

> On Aug 31, 2014, at 2:09 AM, Nick Coghlan  wrote:
> 
> At the same time, we need to account for the fact that most existing
> organisations still trust in perimeter defence for their internal
> network security, and hence tolerate (or even actively encourage) the
> use of unsecured connections, or skipping certificate validation,
> internally. This is actually a really terrible idea, but it's still
> incredibly common due to the general failure of the technology
> industry to take usability issues seriously when we design security
> systems (at least until recently) - doing the wrong "unsafe" thing is
> genuinely easier than doing things right.
> 


Just a quick clarification in order to be a little clearer, this change will
(obviously) only effect those who trust perimeter security *and* decided to
install an invalid certificate instead of just using HTTP. I'm not saying that
this doesn't happen, just being specific (I'm not actually sure why they would
install a TLS certificate at all if they are trusting perimeter security, but
I'm sure folks do).

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 476: Enabling certificate validation by default!

[Donald Stufft]

> On Aug 31, 2014, at 5:43 PM, Christian Heimes  wrote:
> 
> On 31.08.2014 08:09, Nick Coghlan wrote:
>> As Antoine says here, I'm also opposed to adding more Python specific
>> configuration options. However, I think there may be something
>> worthwhile we can do that's closer to the way browsers work, and has
>> the significant benefit of being implementable as a PyPI module first
>> (more on that in a separate reply).
> 
> I'm on your and Antoine's side and strictly against any additional
> environment variables or command line arguments. That would make the
> whole validation process even more complex and harder to understand.
> 
> There might be a better option to give people and companies the option
> to tune the SSL module to their needs. Python already have a
> customization hook for the site module called sitecustomize. How about
> another module named sslcustomize? Such a module could be used to tune
> the ssl module to the needs of users, e.g. configure a different default
> context, add certificates to a default context etc.
> 
> Companies could install them in a system global directory on their
> servers. Users could put them in their own user site directory and even
> each virtual env can have one sslcustomize of its own. It's fully
> backward compatible, doesn't add any flags and developers have the full
> power of Python for configuration and customization.
> 
> 

This may be a dumb question, but why can’t sitecustomize do this already?

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 477: selected ensurepip backports for Python 2.7

[Donald Stufft]

> On Sep 1, 2014, at 2:22 AM, Ned Deily  wrote:
> 
> In article 
>  <mailto:cadisq7e4zachw4b_nmwewpwtxg6davndc-zwhaoajkq4sa3...@mail.gmail.com>>,
> Nick Coghlan mailto:ncogh...@gmail.com>> wrote:
>> On 1 Sep 2014 09:23, "Benjamin Peterson"  wrote:
>>> On Sun, Aug 31, 2014, at 16:17, Antoine Pitrou wrote:
>>>> On Mon, 1 Sep 2014 08:00:14 +1000
>>>> Nick Coghlan  wrote:
>>>>> 
>>>>> That part of the proposal proved to be controversial, so we dropped
>> it from
>>>>> the original PEP in order to focus on meeting the Python 3.4 specific
>>>>> release deadlines. This also had the benefit of working out the kinks
>> in
>>>>> the bootstrapping processing as part of the Python 3.4 release cycle.
>>>>> 
>>>>> However, we still think we should start providing pip by default to
>> Python
>>>>> 2.7 users as well, at least as part of the Windows and Mac OS X
>> installers.
> 
> A much bigger than average +1
> 
>>>> I don't agree with this. pip is simply not part of the 2.7 feature set.
>>>> If you add pip to a bugfix version, then you have bugfix versions which
>>>> are more featureful than others, which makes things more complicated to
>>>> explain.
>>> 2.7.x has been and will be alive for so long that will already have to
>>> explain that sort thing; i.e. PEP 466 and why different bugfix releases
>>> support different versions of dependency libraries.
> 
> And that is a minor complication compared with the confusion and 
> difficulty of trying to explain to users (stuck with 2.7 for the time 
> being) of how to install third-party packages on each platform 
> (especially Windows) versus the simplicity of the 3.4.x story, thanks to 
> ensurepip.  Having pip available as a documented, batteries-included 
> tool for all current releases would be a huge usability improvement.

Yes this is a major driver. I mean I think I probably have an above average
knowledge of how to bootstrap pip, and if you dump me on a Windows box
I struggle to actually do it the first time around without stumbling around and
doing things in the wrong order and the like. (Getting a compiler toolchain is
worse, but yay for Wheels).

> 
>> Exactly. LTS is genuinely different from stopping maintenance after the
>> next feature release - it requires considering the "stability risk" and
>> "user experience improvement" questions separately.
>> 
>> In this case, the problem is that the Python 2 platform *is* still
>> evolving, but the centre of that evolution has moved to PyPI. For "standard
>> library only" users, Python 2 stopped evolving back in 2010. For PyPI
>> users, by contrast, it's still evolving at a rapid pace.
>> 
>> For our Python 3 transition story to be coherent, we need to ensure tools
>> like six, modernize and future are readily available, while still remaining
>> free to evolve independently of the standard library. That means providing
>> a package management utility as easily and as painlessly as possible.
>> 
>> Embracing pip upstream for Python 2 as well as Python 3 also sends a
>> powerful signal to redistributors that says "your users are going to need
>> this" and makes them do additional work to *avoid* providing it. Some of
>> them *will* choose that path, but that becomes a matter for discussion
>> between them and their user base, rather than a problem we need to worry
>> about upstream.
> 
> FTR, I'm willing to backport the pieces I did for 3.4 and I could do the 
> ensurepip backport, as well.  I'll leave the Windows installer changes 
> for someone else, though.

Awesome, I’m of course willing to back port ensure pip itself as well. 
Truthfully
it shouldn’t be a very difficult backport. It’s only ~200 SLOC or so and the 
only
real things would be removing a Python3ism here or there.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 476: Enabling certificate validation by default!

[Donald Stufft]

> On Sep 1, 2014, at 11:35 AM, Nick Coghlan  wrote:
> 
> 
> On 2 Sep 2014 00:59, "Antoine Pitrou"  <mailto:solip...@pitrou.net>> wrote:
> >
> > On Tue, 2 Sep 2014 00:53:11 +1000
> > Nick Coghlan mailto:ncogh...@gmail.com>> wrote:
> > > >
> > > > To be frank I don't understand what you're arguing about.
> > >
> > > When I said "shadowing ssl can be tricky to arrange", Chris correctly
> > > interpreted it as referring to the filesystem based privilege escalation
> > > scenario that isolated mode handles, not to normal in-process
> > > monkeypatching or module injection.
> >
> > There's no actual difference. You can have a sitecustomize.py that does
> > the monkeypatching or the shadowing. There doesn't seem to be anything
> > "tricky" about that.
> 
> Oh, now I get what you mean - yes, sitecustomize already poses the same kind 
> of problem as the proposed sslcustomize (hence the existence of the related 
> command line options).
> 
> I missed that you had switched to talking about using that attack vector, 
> rather than trying to shadow stdlib modules directly through the filesystem 
> (which is the only tricky thing I was referring to).
> 
> Cheers,
> Nick.
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/donald%40stufft.io


Or you can just install something with easy_install, or you can drop a .pth 
file and monkey patch there. You can’t stop people from overriding modules, 
it’s trivial to do. The sys.path ordering just makes it slightly less trivial.

—
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 476: Enabling certificate validation by default!

[Donald Stufft]

> On Sep 1, 2014, at 1:01 PM, Christian Heimes  wrote:
> 
> On 01.09.2014 17:35, Nick Coghlan wrote:
>> Oh, now I get what you mean - yes, sitecustomize already poses the same
>> kind of problem as the proposed sslcustomize (hence the existence of the
>> related command line options).
> 
> If an attacker is able to place a module like sitecustomize.py in an
> import directory or any .pth file in a site-packages directory than this
> Python installation is compromised. .pth files are insidious because
> they are always loaded and their code is always executed. I don't see
> how sslcustomize is going to make a difference here.
> 

Right, this is the point I was trying to make. If you’ve installed a malicious
package it’s game over. There’s nothing Python can do to help you.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 476: Enabling certificate validation by default!

[Donald Stufft]

> On Sep 2, 2014, at 7:47 PM, Glyph Lefkowitz  wrote:
> 
> 
> On Sep 2, 2014, at 4:28 PM, Nick Coghlan  <mailto:ncogh...@gmail.com>> wrote:
> 
>> On 3 Sep 2014 09:08, "David Reid" mailto:dr...@dreid.org>> 
>> wrote:
>> >
>> > Nick Coghlan  gmail.com <http://gmail.com/>> writes:
>> >
>> > > Creating *new* incompatibilities between Python 2 & Python 3 is a major 
>> > > point
>> > > of concern.
>> >
>> > Clearly this change should be backported to Python2.
>> 
>> Proposing to break backwards compatibility in a maintenance release (...)
>> 
> 
> As we keep saying, this is not a break in backwards compatibility, it's a bug 
> fix.  Yes, systems might break, but that breakage represents an increase in 
> security which may well be operationally important.  Not everyone with a 
> "working" application has the relevant understanding an expertise to know 
> that Python's HTTP client is exposing them to surveillance.  These 
> applications should break. That is the very nature of the fix.  It is not a 
> "compatibility break" that the system starts correctly rejecting invalid 
> connections.
> 
> By way of analogy, here's another kind of breach in security: an arbitrary 
> remote code execution vulnerability in XML-RPC.  I think we all agree that 
> any 0day RCE vulnerabilities in Python really ought to be fixed and could be 
> legitimately included without worrying about backwards compatibility breaks.  
> (At least... gosh, I hope so.)
> 
> Perhaps this arbitrary remote execution looks harmless; the use of an eval() 
> instead of an int() someplace.  Perhaps someone discovered that they can do 
> "3 + 4" in their XML-RPC and the server does the computation for them.  
> Great!  They start relying on this in their applications to use symbolic 
> values in their requests instead of having explicit enumerations.  This can 
> save you quite a bit of code!
> 
> When the RCE is fixed, this application will break, and that's fine.  In fact 
> that's the whole point of issuing the fix, that people will no longer be able 
> to make arbitrary computation requests of your server any more.  If that 
> server's maintainer has the relevant context and actually wants the XML-RPC 
> endpoint to enable arbitrary RCE, they can easily modify their application to 
> start doing eval() on the data that they received, just as someone can easily 
> modify their application to intentionally disable all connection security.  
> (Let's stop calling it "certificate verification" because that sounds like 
> some kind of clerical detail: if you disable certificate verification, TLS 
> connections are unauthenticated and unidentified and therefore insecure.)
> 
> For what it's worth, on the equivalent Twisted change, I originally had just 
> these concerns, but my mind was changed when I considered what exactly the 
> user-interface ramifications were for people typing that 's' for 'secure' in 
> URLs.  I was convinced, and we made the change, and there have been no ill 
> effects that I'm aware of as a result.  In fact, there has been a renewed 
> interest in Twisted for HTTP client work, because we finally made security 
> work more or less like it's supposed to, and the standard library is so 
> broken.
> 
> I care about the health of the broader Python community, so I will 
> passionately argue that this change should be made, but for me personally 
> it's a lot easier to justify that everyone should use Twisted (at least since 
> 14+) because transport security in the stdlib is such a wreck and even if it 
> gets fixed it's going to have easy options to turn it off unilaterally so 
> your application can never really be sure if it's getting transport security 
> when it's requesting transport security.
> 

I completely agree with everything Glyph has said in this post. (To the shock 
of everyone involved I’m sure!).

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 476: Enabling certificate validation by default!

[Donald Stufft]

> On Sep 3, 2014, at 1:54 PM, Guido van Rossum  wrote:
> 
> On Wed, Sep 3, 2014 at 8:58 AM, R. David Murray  <mailto:rdmur...@bitdance.com>> wrote:
> I'm OK with letting go of this invalid-cert issue myself, given the lack
> of negative feedback Twisted got.  I'll just keep my fingers crossed.
> 
> I'm with this sentiment (cautiously +1) -- and not just because of Twisted's 
> experience or Glyph's passion.
> 
> Network security is much more important now than it was five years ago -- and 
> yet Python 2.7 is at least that old. My own experience has changed a lot: 
> five years ago (when I worked at Google!) it was common to find internal 
> services that required SSL but had a misconfigured certificate, and the only 
> way to access those services was to override the browser complaints. Today 
> (working at Dropbox, a much smaller company!) I don't even remember the last 
> time I had to deal with such a browser complaint -- internal services here 
> all redirect to SSL, and not a browser that can find fault with their certs. 
> If I did get a complaint about a certificate I would fire off an email to a 
> sysadmin alerting them to the issue.
> 
> Let's take the plunge on this issue for the next 2.7 release (3.5 being a 
> done deal). Yes, some people will find that they have an old script accessing 
> an old service which breaks. Surely some of the other changes in the same 2.7 
> bugfix release will also break some other scripts. People deal with it. 
> Probably 90% of the time it's an annoyance (but no worse than any other 
> minor-release upgrade -- you should test upgrades before committing to them, 
> and if all else fails, roll it back). But at least some of the time it will 
> be a wake-up call and an expired certificate will be replaced, resulting in 
> more security for all.

+1, this makes me unreasonably happy.

> 
> I don't want to start preaching security doom and gloom (the experts are 
> doing enough of that :-), but the scale and sophistication of attacks 
> (whether publicized or not) is constantly increasing, and routine maintenance 
> checks on old software are just one of the small ways that we can help the 
> internet become more secure. (And please let the PSF sysadmin team beef up 
> *.python.org <http://python.org/> -- sooner or later some forgotten part of 
> our infrastructure *will* come under attack.)

This is an ongoing effort amongst the Infra team, part of the process is moving 
infrastructure away from hand crafted servers towards servers managed by config 
management as well as making sure all our services are behind TLS as well.

> 
> -- 
> --Guido van Rossum (python.org/~guido <http://python.org/~guido>)
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/donald%40stufft.io

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposed schedule for 3.4.2

[Donald Stufft]

> On Sep 8, 2014, at 6:20 PM, Nick Coghlan  wrote:
> 
> 
> On 9 Sep 2014 04:00, "Barry Warsaw"  <mailto:ba...@python.org>> wrote:
> > >
> > >This would need to be updated first, once it *did* take such an argument,
> > >this would be accomplished by:
> > >
> > >context = ssl.create_default_context()
> > >context.verify_mode = CERT_OPTIONACERT_NONE
> > >context.verify_hostname = False
> > >urllib.request.urlopen("https://something-i-apparently-dont-care-much-about
> > > <https://something-i-apparently-dont-care-much-about/>",
> > >context=context)
> >
> > There's probably an ugly hack possibility that uses unittest.mock.patch. ;)
> 
> We could actually make it an "official" hack:
> 
> import urllib.request
> urllib.request.urlopen = urllib.request._unverified_urlopen
> 
> Or else the user can just change the code to call the unverified one directly.
> 
> All we'd have to do is keep the existing version that doesn't validate certs 
> properly around under the name "_unverified_urlopen".
> 
> I like this for a few reasons:
> 
> 1. It doesn't get much easier than calling function A instead of function B
> 2. Monkeypatching lets you do a process global hack 
> 3. The name tells you exactly why this is a bad idea
> 4. It's easy to grep for later after you fix your certs
> 5. The leading underscore acts as a strong "keep away" signal
> 6. The leading underscore makes it clear this function may not always be 
> available (e.g. Jython, older versions of Python)
> 
> 

If someone wants to do this, can’t they write their own 6 line function? 

import ssl
import urllib.request
_real_urlopen = urllib.request.urlopen
def _unverified(*args, **kwargs):
    if not kwargs.keys() & {“context”, “cafile”, “capath”, “cadefault”}:
ctx = ssl.create_default_context()
ctx.verify_mode = CERT_NONE
ctx.verify_hostname = False
kwargs[“context”] = ctx
return _real_urlopen(*args, **kwargs)

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposed schedule for 3.4.2

[Donald Stufft]

> On Sep 8, 2014, at 6:43 PM, Nick Coghlan  wrote:
> 
> 
> On 9 Sep 2014 08:30, "Donald Stufft"  <mailto:don...@stufft.io>> wrote:
> >
> > If someone wants to do this, can’t they write their own 6 line function?
> 
> Unfortunately not, as the domain knowledge required to know what those six 
> lines should look like is significant.
> 
> Keeping the old unsafe behaviour around with a more obviously dangerous name 
> is much simpler than explaining to people "Here, copy this chunk of code you 
> don't understand".
> 
> If we were starting with a blank slate there's no way we'd offer such a 
> thing, but as Jim pointed out, we do want to make it relatively easy for 
> Standard Operating Environment maintainers to hack around it if necessary.
> 
> Cheers,
> Nick.
> 
> >
> > import ssl
> > import urllib.request
> > _real_urlopen = urllib.request.urlopen
> > def _unverified(*args, **kwargs):
> > if not kwargs.keys() & {“context”, “cafile”, “capath”, “cadefault”}:
> > ctx = ssl.create_default_context()
> > ctx.verify_mode = CERT_NONE
> > ctx.verify_hostname = False
> > kwargs[“context”] = ctx
> > return _real_urlopen(*args, **kwargs)
> >
> > ---
> > Donald Stufft
> > PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
> >


Why isn’t documentation with appropriate red warnings a suitable place if we 
really must have it? That sounds like a much better solution that some weird 
function people monkeypatch. It gives them more control over things (maybe they 
have a valid certificate chain, but an invalid host name!), it’ll work across 
all Python implementations, and most importantly, it gives us a place where 
there is some long form location to be like “yea you really probably don’t want 
to be doing this” in big red letters.

Overall I’m -1 on either offering the function or documenting it at all, but if 
we must do something then I think documentation is more than enough.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Backwards compatibility after certificate autovalidation

[Donald Stufft]

It’s create_default_context() -> 
https://docs.python.org/3.4/library/ssl.html#ssl.create_default_context


> On Sep 8, 2014, at 11:40 PM, Guido van Rossum  wrote:
> 
> The multiple threads going on are confusing (or maybe GMail makes them more 
> confusing), but the architecture you are sketching here sounds good. I can't 
> find get_default_context() in the repo, but perhaps I need to refresh, or 
> perhaps you're talking about a design in a PEP.
> 
> On Mon, Sep 8, 2014 at 8:03 PM, Nick Coghlan  <mailto:ncogh...@gmail.com>> wrote:
> 
> On 9 Sep 2014 10:48, "Jim J. Jewett"  <mailto:jimjjew...@gmail.com>> wrote:
> > I assume that adding _unverified_urlopen or urlopen(context=...) do
> > provide incremental improvements compatible with the eventual full
> > opt-in.  If so, adding them is probably reasonable, but I think the
> > PEP should explicitly list all such approved half-measures as a guard
> > against API feature creep.
> 
> From Guido's and your feedback, I think we may need two things to approve 
> this for 3.4.2 (putting 2.7 aside for now):
> 
> 1. "context" parameter support in urllib.request (to opt out on a per-call 
> basis)
> 2. a documented way to restore the old behaviour via sitecustomize (which may 
> involve monkeypatching)
> 
> The former change seems non-controversial.
> 
> I think the more fine-grained solution for the latter can wait until 3.5 (and 
> will be an independent PEP), we just need an interim workaround for 3.4 that 
> could conceivably be backported to 2.7.
> 
> On that front, ssl currently has two context factories: get_default_context() 
> and _get_stdlib_context. One possible option would be to:
> 
> 1. Rename "_get_stdlib_context" to "_get_unverified_context"
> 2. Add "_get_https_context" as an alias for "get_default_context"
> 
> Opting out on a per-call basis means passing an unverified context.
> 
> Opting out globally would mean monkeypatching _get_https_context to refer to 
> _get_unverified_context instead.
> 
> These would both be documented as part of transition, but with clear security 
> warnings. The use of the leading underscores in the names is intended to 
> emphasise "you probably don't want to be using this".
> 
> Regards,
> Nick.
> 
> 
> 
> >
> > -jJ
> > ___
> > Python-Dev mailing list
> > Python-Dev@python.org <mailto:Python-Dev@python.org>
> > https://mail.python.org/mailman/listinfo/python-dev 
> > <https://mail.python.org/mailman/listinfo/python-dev>
> > Unsubscribe: 
> > https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com 
> > <https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com>
> 
> ___
> Python-Dev mailing list
> Python-Dev@python.org <mailto:Python-Dev@python.org>
> https://mail.python.org/mailman/listinfo/python-dev 
> <https://mail.python.org/mailman/listinfo/python-dev>
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/guido%40python.org 
> <https://mail.python.org/mailman/options/python-dev/guido%40python.org>
> 
> 
> 
> 
> -- 
> --Guido van Rossum (python.org/~guido <http://python.org/~guido>)
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/donald%40stufft.io

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 394 - Clarification of what "python" command should invoke

[Donald Stufft]

> On Sep 19, 2014, at 3:31 AM, Bohuslav Kabrda  wrote:
> 
> Hi,
> as Fedora is getting closer to having python3 as a default, I'm being more 
> and more asked by Fedora users/contributors what'll "/usr/bin/python" invoke 
> when we achieve this (Fedora 22 hopefully). So I was rereading PEP 394 and I 
> think I need a small clarification regarding two points in the PEP:
> - "for the time being, all distributions should ensure that python refers to 
> the same target as python2."
> - "Similarly, the more general python command should be installed whenever 
> any version of Python is installed and should invoke the same version of 
> Python as either python2 or python3."
> 
> The important word in the second point is, I think, *whenever*. Trying to 
> apply these two points to Fedora 22 situation, I can think of several 
> approaches:
> - /usr/bin/python will always point to python3 (seems to go against the first 
> mentioned PEP recommendation)
> - /usr/bin/python will always point to python2 (seems to go against the 
> second mentioned PEP recommendation, there is no /usr/bin/python if python2 
> is not installed)
> - /usr/bin/python will point to python3 if python2 is not installed, else it 
> will point to python2 (inconsistent; also the user doesn't know he's running 
> and what libraries he'll be able to import - the system can have different 
> sets of python2-* and python3-* extension modules installed)
> - there will be no /usr/bin/python (goes against PEP and seems just wrong)
> 
> I'd really appreciate upstream guidance and perhaps a PEP clarification for 
> distributions that ship both python2 and python3, but can live without 
> python2 (and are not Arch :)).
> 
> Thanks a lot!


I don’t know for a fact, but I assume that as long as Python 2.x is installed 
by default than ``python`` should point to ``python2``. If Python 3.x is the 
default version and Python 2.x is the “optional” version than I think 
personally it makes sense to switch eventually. Maybe not immediately to give 
people time to update though?

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 394 - Clarification of what "python" command should invoke

[Donald Stufft]

> On Sep 19, 2014, at 10:16 AM, Barry Warsaw  wrote:
> 
> If the user wants to invoke Python 3, it's not hard to type 'python3' and I
> think that's the message we should be spreading.  That already seems pretty
> ingrained in user habits afaict.


My biggest problem with ``python3``, is what happens after 3.9. I know Guido
doesn’t particularly like two digit version numbers and it’s been suggested on
this list that instead of 3.10 we’re likely to move directly into 4.0 
regardless of
if it’s a “big” change or not.

If that is the case, then all of the user education, ui, etc around ``python3`` 
would
then need to be again updated to ``python4`` *OR* we’d need a ``python3`` bin
which points to ``python4``. If there’s a call to action for at some point 
moving
``python`` to invoke Python 3.x at some point then hopefully at that point 
Python 4.x
could just be ``python``.

All of this assuming of course that 4.0 isn’t a major break like 3.0 and that we
do 4.0 instead of 3.10 as has been suggested.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 394 - Clarification of what "python" command should invoke

[Donald Stufft]

> 
> On Sep 19, 2014, at 8:02 PM, Greg Ewing  wrote:
> 
> Donald Stufft wrote:
> 
>> My biggest problem with ``python3``, is what happens after 3.9.
> 
> Python2 technically includes 1.x versions as well, so it
> wouldn't be unprecedented for python3 to imply versions
> beyond 3.x. It would be a bit confusing, though.

My problem isn’t that it *includes* it, it’s that either people have to go
through the mess to update all of their things to ``python4`` at some point
in the future, or Python 3.x will need to eventually mean ``python``.

Well that or Python 4.x has a ``python3`` binary.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 3.5 release schedule PEP

[Donald Stufft]

> On Sep 22, 2014, at 8:20 PM, Larry Hastings  wrote:
> 
> 
> On 09/19/2014 03:31 PM, Barry Warsaw wrote:
>> I think we need a Python 3.5 Release Schedule PEP.
> 
> Just checked it in as PEP 478.  It should show up here in a few minutes:
> http://legacy.python.org/dev/peps/pep-0478/ 
> <http://legacy.python.org/dev/peps/pep-0478/>
> 
> Key facts:
> Beta 1 is May 24th 2015, about a month after the end of the PyCon US 2015 
> sprints.
> Final release is September 13, 2015, just over a year from now.
> 
> Comments?

It says 3.4.0 all through it.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 3.5 release schedule PEP

[Donald Stufft]

> On Sep 23, 2014, at 9:48 PM, Nick Coghlan  wrote:
> 
> On 24 September 2014 03:05, Steve Dower  <mailto:steve.do...@microsoft.com>> wrote:
>> Larry Hastings wrote:
>>> 
>>> On 09/19/2014 03:31 PM, Barry Warsaw wrote:
>>> I think we need a Python 3.5 Release Schedule PEP.
>>> 
>>> Just checked it in as PEP 478.  It should show up here in a few minutes:
>>> http://legacy.python.org/dev/peps/pep-0478/
>>> 
>>> Key facts:
>>> . Beta 1 is May 24th 2015, about a month after the end of the PyCon US 2015 
>>> sprints.
>>> . Final release is September 13, 2015, just over a year from now.
>>> 
>>> Comments?
>> 
>> Martin is no longer producing the Windows installers - that task has been 
>> handed to me. I'm planning to have a rewritten installer (also in the same 
>> repo) that should be easier to modify and maintain, as well as being able to 
>> produce alternative packages (such as a Python 3.5 or stdlib merge module, 
>> for example), though that doesn't necessarily need to go into the PEP.
>> 
>> I'm also considering/experimenting with installing into "Program Files" by 
>> default, but I suspect that isn't going to work out yet.
>> 
>> I'd like to move the Windows versions onto the next release of VC (currently 
>> "VC 14" until the branding team figures out what to call it). There isn't a 
>> promised RTM date for VC 14 yet, so it looks like the best available 
>> compiler by Beta 1 will be a "Go Live" RC. (The "Go Live" marking basically 
>> means "we think this is ready for use, but expect a round of minor 
>> updates/fixes soon - the compiler is least likely to be updated, my guess is 
>> that it'll be Visual Studio UI mostly).
>> 
>> I personally don't have any qualms about using the RC compiler for Beta 1, 
>> and Beta 2 will almost certainly use VC 14 RTM, but I know when I proposed 
>> this topic that some people were concerned about having the final version 
>> available for Python 3.5 Beta.
>> 
>> So far I've been building regularly with internal versions of VC and haven't 
>> been hitting any major issues with Python (OpenSSL has some issues, but I've 
>> been filing bugs on both sides so they should be worked out soon enough). My 
>> work is at http://hg.python.org/sandbox/steve.dower (branch: VC14) for 
>> anyone interested.
>> 
>> For the alphas, I'm contemplating producing two builds (VC 10 and VC 14), 
>> but I obviously want to settle on one or the other by Beta. Last time we 
>> discussed it, there was strong support for changing compiler, but I have a 
>> better idea of the timeline now and it's tighter than I thought...
>> 
>> Thoughts, anyone?
> 
> It's ultimately up to Larry as RM, but I'd personally favour targeting
> the newer compiler and runtime, even with the slight risk of
> potentially needing to slip our schedule. There's also a fair amount
> of wiggle room between the first beta and the first release candidate.
> 
> Regards,
> Nick.
> 
> -- 
> Nick Coghlan   |   ncogh...@gmail.com <mailto:ncogh...@gmail.com>   |   
> Brisbane, Australia
> ___
> Python-Dev mailing list
> Python-Dev@python.org <mailto:Python-Dev@python.org>
> https://mail.python.org/mailman/listinfo/python-dev 
> <https://mail.python.org/mailman/listinfo/python-dev>
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/donald%40stufft.io 
> <https://mail.python.org/mailman/options/python-dev/donald%40stufft.io>

This new compiler has the incredibly awesome feature of being forwards 
compatible
right? Like in 10 years stuff compiled with a newer compiler will still work?

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 3.5 release schedule PEP

[Donald Stufft]

> On Sep 23, 2014, at 10:14 PM, Steve Dower  wrote:
> 
> "This new compiler has the incredibly awesome feature of being forwards 
> compatible
> right? Like in 10 years stuff compiled with a newer compiler will still work?"
> 
> That's the promise at least :)
> 
> All the macros that leaked implementation details (like file descriptors) are 
> now isolated so if they change it won't break existing applications. It'll 
> still be possible for newer compiler versions to break them, but the design 
> now obviously discourages it. There's also an official guarantee, so if it is 
> broken in future it'll be treated as a bug. As much as I'd love to make solid 
> promises, I can only pass on the promises that were made to me.
> 
> But yes, we should have forward compatibility with later MSVC versions, which 
> will help avoid the situation where it's hard to get hold of the right 
> compiler...
> 
> Top-posted from my Windows Phone

Yea I think it makes incredible sense to aim for it then, even if it makes 
things slip.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 3.5 release schedule PEP

[Donald Stufft]

> On Sep 24, 2014, at 4:24 PM, Steve Dower  wrote:
> 
>> Paul Moore wrote:
>> On 24 September 2014 14:16, Mike Miller  wrote:
>>> It has been a supported option for just shy of 15 years on 2.X...
>>> most if not all the bugs (setuptools) were fixed a decade ago, and
>>> right now thousands, if not millions of people are running it under
>>> Program Files right now. I can vouch for several thousand because a
>>> company I work for distributes Python and pip there for its customers
>>> all around the world w/o issue.
>> 
>> One thing that I presume would be an issue. Isn't Program Files protected in
>> newer versions of Windows? I haven't tested this myself, so I may be wrong 
>> about
>> this. So take the following with a pinch of salt.
> 
> It's protected very well in newer versions. You typically need to be an 
> administrator AND have opted in to being able to modify system files without 
> warning.
> 
>> Assuming so, that means that if Python is installed there, the standard "pip
>> install XXX" would not work unless run in an elevated shell. We are currently
>> trying to focus on a unified message for how users should install 
>> distributions
>> from PyPI, by using pip install.
>> I'm not sure it's a good idea to complicate that message yet by adding
>> provisos about managing the system Python (which is the only one most Windows
>> users will use).
> 
> This is my main concern. Until pip install --user is the default (or the 
> fallback if there are no write permissions on the destination), a default 
> that locks users out of the simplest PyPI experience is a genuine problem. 
> Yes, users can elevate to run pip, but I'd prefer pip to use elevation if it 
> has it and to use per-user if not.
> 
> There also isn't a great story for per-user Python installs on Windows, but 
> that becomes fairly cheap with the installer rewrite.
> 
>> I know this is only the same situation as Unix users have, but Windows users
>> don't have a distro offering packaged versions of PyPI modules.
>> I also know we should be moving towards promoting --user, but I don't think
>> we're quite ready for that yet. And my speculation doesn't compete with your
>> real-life experience, certainly. But I would suggest carefully checking 
>> before
>> making the switch.
> 
> A good reason to decide early on a change like this, or at least to promote 
> it as an option in 3.5 and make it the default in 3.6.


One thing about *nix is even though you can’t write to your normal Python
install location without root, invoking pip with permissions (assuming you have
them) is as easy as prefacing it with ``sudo`` in most cases. Does Windows have
an equivalent or do you need to launch a whole new shell?

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 3.5 release schedule PEP

[Donald Stufft]

> On Sep 25, 2014, at 4:54 AM, Antoine Pitrou  wrote:
> 
> On Thu, 25 Sep 2014 07:34:31 +0100
> Paul Moore  wrote:
>> On 25 September 2014 02:08, Antoine Pitrou  wrote:
>>>> Indeed. Moving towards having --user as the norm is definitely
>>>> something we want to look at for pip. One of the biggest concerns is
>>>> how well-exercised the whole user site directory area is in practice.
>>> 
>>> What do you mean by well-exercised?
>> 
>> Basically, although --user is available in pip (and the underlying
>> facilities in Python have been around for some time), it's difficult
>> to gauge how many people are using them, and as a result what level of
>> testing has happened in real-life situations.
> 
> I'm using it often. I'm also unsure how broken it could be. The user
> site-packages is just another site-packages directory.
> 

Broken like the prefix problem :)

Basically people have Python in a ton of different configurations and it’s
hard to figure out if —user will work out of the box in all of them or not.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 3.5 release schedule PEP

[Donald Stufft]

> On Sep 25, 2014, at 11:54 AM, Paul Moore  wrote:
> 
> On 25 September 2014 16:43, Donald Stufft  wrote:
>> Basically people have Python in a ton of different configurations and it’s
>> hard to figure out if —user will work out of the box in all of them or not.
> 
> I guess that "Using the python.org Python installer on Windows" is a
> limited enough subset that we probably could check that --user worked
> in that situation.
> 
> The problem is, how do we implement it? A special case so that pip
> defaults to --user sometimes, but not others? (I'm strongly against
> that) Leave the default as not --user and document that Windows users
> with Python in "Program Files" should always specify --user? (I'm
> against that because it makes the documentation highly confusing, and
> we've just done a lot of work to simplify it).
> 
> Basically, I'd like to hold off moving to "Program Files" as a default
> until *after* we have enough confidence in user installs that we are
> willing to switch pip to --user as the default behaviour everywhere.
> And yes, I'm aware that the first "we" in that was "python-dev" and
> the second was "PyPA". And that expecting python-dev to wait for PyPA
> to approve the change may well be unacceptable.
> 
> Paul


My thoughts on the pip side has basically always been that pip should
either:

1) Just always default to —user and add a —system or similar flag, this
is super easy to change but is a backwards incompatible change and
would need to go through a deprecation window.
2) Switch to —user based on if the user has permission to write to the
site-packages or not.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 3.5 release schedule PEP

[Donald Stufft]

> On Sep 25, 2014, at 6:44 PM, Chris Barker  wrote:
> 
> On Thu, Sep 25, 2014 at 9:00 AM, Donald Stufft  <mailto:don...@stufft.io>> wrote:
> 1) Just always default to —user and add a —system or similar flag, this
> is super easy to change but is a backwards incompatible change and
> would need to go through a deprecation window.
> 
> Maybe would have been the way to go to begin with, but I think backwards 
> compatibility should probably trump "better" -- even with a deprecation 
> window.
>  
> 2) Switch to —user based on if the user has permission to write to the
> site-packages or not.
> 
> ouch -- no. Why not a clear error message if pip can't write to site-packages 
> -- something like:
> 
> """
> pip doesn't have permissions to write to the system location. If you want to 
> install this package system-wide, you need to run pip with admin priviledges 
> (and example here if it's easy), if you want to install for this user only, 
> pass the "--user" flag to pip install
> """ 
> 
> "In the face of ambiguity, refuse the temptation to guess.”

I fairly strongly believe that the current default is doing a great disservice
to users. I believe that for *most* people --user is the correct option for
them to be using and the fact that it's not the default and requires opt in
is a historical artifact more than anything else.

Immediately switching to --user as a default (outside of a virtual environment)
would break two important use cases:

- A root/admin user attempting to install into a global site-packages
- A user who uses complete Python installations to isolate themselves such as
  those created by pyenv.

It would techincally break anyone relying on the fact that pip will currently
throw an error if you attempt to ``pip install`` something without sudo,
however I do not consider that to be an actual use case that is going to have
wide signifcance.

So we get down to how do we not break the two important use cases above. At
first I thought about detecting uid 0 (and something similar on Windows?) and
turning off the --user default in those cases. However that still left us
dealing with something for the second use case.

After thinking about it I realized there isn't a good way to determine if
something is a user controlled Python installation instead of a system
controlled one. This lead me to the realization that something that could be
used to "detect" this is whether or not the current user has the ability to
write to the site-packages as a means of determining "is my current user
allowed to manage that particular Python".

Either way I'm fairly commited to making --user the default, the only question
on my mind is what exactly does that look like (e.g. does root get --user by
default?) and how we get from where we are now to that point. I think that
raising an error here is fairly unfriendly when we can know with pretty good
certainity what the user wanted (and they can explicitly declare that if they
want too).

However this particular thing is somewhat offtopic for this list and the
particulars of pip moving to --user by default is best discussed on our issue
tracker here: https://github.com/pypa/pip/issues/1668.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 3.5 release schedule PEP

[Donald Stufft]

> On Sep 26, 2014, at 3:09 AM, Paul Moore  wrote:
> 
> On 26 September 2014 01:38, Donald Stufft  wrote:
>> Either way I'm fairly commited to making --user the default, the only
>> question
>> on my mind is what exactly does that look like (e.g. does root get --user by
>> default?) and how we get from where we are now to that point. I think that
>> raising an error here is fairly unfriendly when we can know with pretty good
>> certainity what the user wanted (and they can explicitly declare that if
>> they want too).
> 
> A couple of points come to mind (although they may be more suitable
> for distutils-sig).
> 
> 1. Do user installs "leak" into virtualenvs? If so, then in effect
> --use-system-packages is switched back on again if --user installs
> become the norm. Which is almost certainly not what is wanted.
> 2. pip install should default to not being --user when run from within
> a virtualenv (same logic as the isolated Python case, but much more
> important that behaviour remains as now, because the whole *point* of
> virtualenvs is to isolate).
> 
> Note that both of these points apply both to venv and virtualenvs.


1. No they don’t leak as far as I’m aware.
2. Yea, I think we throw an error when you use —user inside a virtual
environment.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 3.5 release schedule PEP

[Donald Stufft]

> On Sep 26, 2014, at 9:53 AM, Paul Moore  wrote:
> 
> On 26 September 2014 14:31, Donald Stufft  wrote:
>> Yea, I think we throw an error when you use —user inside a virtual
>>environment.
> 
> So if --user became the default, what would happen? I'd like pip
> inside a virtualenv to install into the environment without needing a
> --system flag. Is that the intention?
> 
> Paul


default = “—user” if not running_inside_virtualenv() else “—system"

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Microsoft Visual C++ Compiler for Python 2.7

[Donald Stufft]

Awesome!


> On Sep 26, 2014, at 2:01 PM, Steve Dower  wrote:
> 
> Hi all,
> 
> (This is advance notice since people on this list will be interested. 
> Official announcements are coming when setuptools makes their next release.)
> 
> Microsoft has released a compiler package targeting Python 2.7 (i.e. VC9). 
> We've produced this package to help library developers build wheels for 
> Windows, but also to help users unblock themselves when they need to build C 
> extensions themselves.
> 
> The Microsoft Visual C++ Compiler for Python 2.7 is available from: 
> http://aka.ms/vcpython27 
> 
> This package contains all the tools and headers required to build C extension 
> modules for Python 2.7 32-bit and 64-bit (note that some extension modules 
> require 3rd party dependencies such as OpenSSL or libxml2 that are not 
> included). Other versions of Python built with Visual C++ 2008 are also 
> supported.
> 
> You can install the package without requiring administrative privileges and, 
> with the latest version of setuptools (when it releases), use tools such as 
> pip, wheel, or a setup.py file to produce binaries on Windows.
> 
> Unfortunately, this package isn't necessarily going to help with building 
> CPython 2.7 itself, as the build process is complicated and Visual Studio 
> 2008 is practically required. However, as most people aren't building CPython 
> on Windows, this isn't a huge issue. This compiler package should be 
> sufficient for most extension modules.
> 
> I should also point out that VC9 is no longer supported by Microsoft. This 
> means there won't be any improvements or bug fixes coming, and there's no 
> official support offered. Feel free to contact me directly 
>  if there are issues with the package.
> 
> Cheers,
> Steve
> 
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/donald%40stufft.io
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Backporting ensurepip to 2.7, Which commands to install?

[Donald Stufft]

I'm working on the backport of ensurepip to Python 2.7, and I realized that
I'm not sure which commands to install. Right now by default pip (outside of
the context of ensurepip) will install pip, pip2, and pip2.7 if installed in
Python 2.7. In Python 3's ensurepip we modified it so that it would install
pip3, and pip3.4, but *not* pip if it was an "install", and only pip3.4 if it
was an "alt install".

My question is, does this behavior make sense for ensurepip in 2.7? Or should
it also install the "pip" command if it is an "install"?

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Backporting ensurepip to 2.7, Which commands to install?

[Donald Stufft]

Ok, so neither Python 2.7 nor Python 3.x’s ensure pip command will install a
``pip`` binary by default without a flag. That's fine with me, just wanted to
make sure it made sense for Python 2.x. Thanks!

> On Oct 3, 2014, at 8:31 PM, Guido van Rossum  wrote:
> 
> That is copying the (alt)install targets of Python's own Makefile, and I 
> think those are exactly right.
> 
> On Oct 3, 2014 3:07 PM, "Donald Stufft"  <mailto:don...@stufft.io>> wrote:
> I'm working on the backport of ensurepip to Python 2.7, and I realized that
> I'm not sure which commands to install. Right now by default pip (outside of
> the context of ensurepip) will install pip, pip2, and pip2.7 if installed in
> Python 2.7. In Python 3's ensurepip we modified it so that it would install
> pip3, and pip3.4, but *not* pip if it was an "install", and only pip3.4 if it
> was an "alt install".
> 
> My question is, does this behavior make sense for ensurepip in 2.7? Or should
> it also install the "pip" command if it is an "install"?
> 
> ---
> Donald Stufft
> PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
> 
> ___
> Python-Dev mailing list
> Python-Dev@python.org <mailto:Python-Dev@python.org>
> https://mail.python.org/mailman/listinfo/python-dev 
> <https://mail.python.org/mailman/listinfo/python-dev>
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/guido%40python.org 
> <https://mail.python.org/mailman/options/python-dev/guido%40python.org>

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Backporting ensurepip to 2.7, Which commands to install?

[Donald Stufft]

Whoops, I misred.

So to be clear, you think:

install -> pip, pip2, pip2.7
altinstall -> pip2.7

> On Oct 3, 2014, at 8:46 PM, Guido van Rossum  wrote:
> 
> That's not what I meant. Python 2.7 does install "python" unless you use 
> altinstall.
> 
> On Oct 3, 2014 5:33 PM, "Donald Stufft"  <mailto:don...@stufft.io>> wrote:
> Ok, so neither Python 2.7 nor Python 3.x’s ensure pip command will install a
> ``pip`` binary by default without a flag. That's fine with me, just wanted to
> make sure it made sense for Python 2.x. Thanks!
> 
>> On Oct 3, 2014, at 8:31 PM, Guido van Rossum > <mailto:gu...@python.org>> wrote:
>> 
>> That is copying the (alt)install targets of Python's own Makefile, and I 
>> think those are exactly right.
>> 
>> On Oct 3, 2014 3:07 PM, "Donald Stufft" > <mailto:don...@stufft.io>> wrote:
>> I'm working on the backport of ensurepip to Python 2.7, and I realized that
>> I'm not sure which commands to install. Right now by default pip (outside of
>> the context of ensurepip) will install pip, pip2, and pip2.7 if installed in
>> Python 2.7. In Python 3's ensurepip we modified it so that it would install
>> pip3, and pip3.4, but *not* pip if it was an "install", and only pip3.4 if it
>> was an "alt install".
>> 
>> My question is, does this behavior make sense for ensurepip in 2.7? Or should
>> it also install the "pip" command if it is an "install"?
>> 
>> ---
>> Donald Stufft
>> PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
>> 
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org <mailto:Python-Dev@python.org>
>> https://mail.python.org/mailman/listinfo/python-dev 
>> <https://mail.python.org/mailman/listinfo/python-dev>
>> Unsubscribe: 
>> https://mail.python.org/mailman/options/python-dev/guido%40python.org 
>> <https://mail.python.org/mailman/options/python-dev/guido%40python.org>
> 
> ---
> Donald Stufft
> PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
> 

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] shebang policy, and pip

[Donald Stufft]

> On Oct 8, 2014, at 6:16 AM, Christian Tismer  wrote:
> 
> 
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA512
> 
> Howdy,
> 
> this question is a bit about general policy which is not yet
> covered in the python recommendations:
> 
> I see projects which do check-ins like "get rid of shebang lines"
> and they remove those lines from non-script sources.
> 
> It is not always clear to me what to do, so I tend to leave those
> lines in per default, in order not to waste time thinking about it,
> but well, today I was confronted with that.
> 
> Digging a bit deeper shows the following:
> 
> python docs:
> No mention of shebang, but for Windows.
> https://docs.python.org/3/search.html?q=shebang&check_keywords=yes&area=default
> https://docs.python.org/3/using/windows.html?highlight=shebang
> 
> Google's python style guide also says when a shebang is needed, but
> does not forbid it.
> 
> Pep 394 explains how to use shebang, but still nothing about not using it.
> http://legacy.python.org/dev/peps/pep-0394/
> 
> So is there anything officially preferred, and should that go into pep 8?

Some editors can use shebang lines to control syntax highlighting or linting
(mine for example will lint different for python2 vs python3 shebangs).

> 
> 
> Special case with pip
> - --
> 
> I was looking through my installed packages and wondered quite
> much about pip:
> 
> Pip has a shebang in the __init__ file, but no shebang in the
> __main__ file.
> 
> I guess this is wrong and should be in the executable file,
> which is __main__ .
> 

*puts on pip developer hat*

I’m guessing that comes from when pip used to be a single file and from before
we dropped support for Pythons that didn’t support python -m. It’s probably
nonsense cruft that’s just been left over and hadn’t been managed to be deleted
now.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] shebang policy, and pip

[Donald Stufft]

> On Oct 8, 2014, at 3:36 PM, Wes Turner  wrote:
> 
> 
> On Oct 8, 2014 7:20 AM, "Donald Stufft"  <mailto:don...@stufft.io>> wrote:
> >
> >
> > > On Oct 8, 2014, at 6:16 AM, Christian Tismer  > > <mailto:tis...@stackless.com>> wrote:
> > >
> > >
> > > -BEGIN PGP SIGNED MESSAGE-
> > > Hash: SHA512
> > >
> > > Howdy,
> > >
> > > this question is a bit about general policy which is not yet
> > > covered in the python recommendations:
> > >
> > > I see projects which do check-ins like "get rid of shebang lines"
> > > and they remove those lines from non-script sources.
> > >
> > > It is not always clear to me what to do, so I tend to leave those
> > > lines in per default, in order not to waste time thinking about it,
> > > but well, today I was confronted with that.
> > >
> > > Digging a bit deeper shows the following:
> > >
> > > python docs:
> > > No mention of shebang, but for Windows.
> > > https://docs.python.org/3/search.html?q=shebang&check_keywords=yes&area=default
> > >  
> > > <https://docs.python.org/3/search.html?q=shebang&check_keywords=yes&area=default>
> > > https://docs.python.org/3/using/windows.html?highlight=shebang 
> > > <https://docs.python.org/3/using/windows.html?highlight=shebang>
> > >
> > > Google's python style guide also says when a shebang is needed, but
> > > does not forbid it.
> > >
> > > Pep 394 explains how to use shebang, but still nothing about not using it.
> > > http://legacy.python.org/dev/peps/pep-0394/ 
> > > <http://legacy.python.org/dev/peps/pep-0394/>
> > >
> > > So is there anything officially preferred, and should that go into pep 8?
> >
> > Some editors can use shebang lines to control syntax highlighting or linting
> > (mine for example will lint different for python2 vs python3 shebangs).
> 
> Does it support shebang lines like:
> 
> #!/usr/bin/env python
> 
> ?
> 

Sure. Though it doesn’t resolve it to determine what that would actually 
execute,
it just uses the name of the python in the shebang as heuristics.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[Donald Stufft]

> On Oct 10, 2014, at 6:59 PM, Steve Dower  wrote:
> 
> Cross compilation is a valid issue, but I hope that build services like 
> Appveyor also help out here. There is regular talk about the PSF/PyPI 
> providing something similar, though I have doubts about its feasibility under 
> any model other than renting a preconfigured VM. I don't see any reason why 
> projects couldn't apply to the PSF for a grant to cover Appveyor costs or the 
> expenses of similar services.

I know very little about Windows, but we can spin up arbitrary Windows boxes, 
build something on them, and then shut them down as long as there’s some way to 
automate logging into a Windows box and running some commands… Normally I’d do 
this with SSH but I don’t know if Windows has anything like that.

IOW we can totally spin up preconfigured VMs for a Windows build service.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Disabling SSL 3.0

[Donald Stufft]

A big security breach of SSL 3.0 just dropped a little while ago (named POODLE).
With this there is now no ability to securely connect via SSL 3.0. I believe
that we should disable SSL 3.0 in Python similarly to how SSL 2.0 is disabled,
where it is disabled by default unless the user has explicitly re-enabled it.

The new attack essentially allows reading the sensitive data from within a SSL
3.0 connection stream. It takes roughly 256 requests to break a single byte so
the attack is very practical. You can read more about the attack here at the
google announcement [1] or the whitepaper [2].

[1] 
http://googleonlinesecurity.blogspot.com/2014/10/this-poodle-bites-exploiting-ssl-30.html
[2] https://www.openssl.org/~bodo/ssl-poodle.pdf

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] https://docs.python.org/3/using/index.html not linking correctly

[Donald Stufft]

I'm on my phone but docs is served via fastly. Issues could be POP specific. 


> On Oct 20, 2014, at 7:38 PM, Oleg Broytman  wrote:
> 
>> On Tue, Oct 21, 2014 at 12:29:45AM +0100, MRAB  
>> wrote:
>>> On 2014-10-21 00:09, Eli Bendersky wrote:
>>> 
>>> On Mon, Oct 20, 2014 at 1:01 PM, Terry Reedy >> > wrote:
>>> 
>>>   If I go to https://docs.python.org/3/using/index.html and click on
>>>   any of the TOC entries, I get 'connecting' indefinitely.  This
>>>   problem seems unique to this file. I tried several other index files
>>>   and clicking am entry brings up the corresponding page almost
>>>   immediately.
>>> 
>>> Works fine for me, Terry, in both Chrome and Firefox. Could be something
>>> in your browser/caching? Try "private mode" or whatever that's called in
>>> your browser of choice.
>> Just tried the first link:
>> 
>> https://docs.python.org/3/using/cmdline.html
>> 
>> I also get 'Connecting' indefinitely.
>> 
>> So I tested the link here:
>> 
>> http://www.downforeveryoneorjustme.com/
>> 
>> It said: "It's not just you! http://https looks down from here."
> 
>   I think this is a limitation of isup.me: it doesn't like https://
> URLs -- either use http:// or no protocol prefix at all.
> 
> Oleg.
> -- 
> Oleg Broytmanhttp://phdru.name/p...@phdru.name
>   Programmers don't die, they just GOSUB without RETURN.
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/donald%40stufft.io
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] XP buildbot problem cloning from hg.python.org

[Donald Stufft]

Is this using HTTPS or SSH.

> On Oct 24, 2014, at 11:47 PM, David Bolen  wrote:
> 
> Starting yesterday, my XP buildbot began failing to execute clone
> operations against hg.python.org.  There's not a lot of data being
> given aside from a transaction abort message (and my buildbot log
> showing the hg command exiting), and I'm wondering if something may be
> amiss on the server or its configuration?
> 
> Note that this is a full clone (which for some reason the Windows
> buildbots seem to fall back on with some frequency) and can take quite
> a while.  My Windows 7 buildbot is ok so far but it's still doing
> incremental pulls over the same time period.
> 
> I've got two separate Internet connections here and have tried routing
> over both so I don't think it's a network issue.  I've completely
> flushed the local build trees and rebooted the buildbot.
> 
> Is there anything that might be available on the server to see if
> there are errors being logged?  Or anything that could have changed
> configuration wise recently (maybe timeout related or something)?  I'm
> running a bit low of items to try to change or reset on the buildbot
> side.
> 
> Thanks.
> 
> -- David
> 
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/donald%40stufft.io

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] XP buildbot problem cloning from hg.python.org

[Donald Stufft]

What version of OpenSSL is it using.

> On Oct 25, 2014, at 1:00 AM, David Bolen  wrote:
> 
> David Bolen  writes:
> 
>> which appears to die mid-stream while receiving the manifests.
>> 
>> So I'm sort of hoping there might be some record server-side as to why
>> things are falling apart mid-way.
> 
> Just to follow-up to myself, I get the same same error trying to do a
> clone from my own personal XP machine rather than the buildbot (which
> is a VM).  I've had the issue with hg 1.6.2, 2.5.2 and 3.1.2.
> 
> However, the same clones completely successfully under OSX and Linux.
> 
> So that's sort of strange.
> 
> -- David
> 
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/donald%40stufft.io

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] XP buildbot problem cloning from hg.python.org

[Donald Stufft]

I have an idea, can you run https://bpaste.net/show/c5d7cd102f5b and
tell me what it outputs? Both on a machine that works and one that
doesn’t.

> On Oct 25, 2014, at 2:14 AM, David Bolen  wrote:
> 
> Donald Stufft  writes:
> 
>> What version of OpenSSL is it using.
> 
> I'm using the pre-built Windows Mercurial installer, but if I unpack
> the included library.zip, the SSLEAY32.DLL shows version 0.9.8r.
> 
> This is from the 3.1.2 install I just did a few hours ago.  It appears
> that hg 2.5.2 on my other XP box also has 0.9.8r.  The prior buildbot
> version (1.6.2) looks like it had 0.9.8o.
> 
> I also got around to trying a manual clone on the Windows 7 buildbot,
> and it worked fine, even with the older hg 1.6.2.
> 
> So it seems to correlate with XP more than anything else at the moment.
> 
> -- David
> 
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/donald%40stufft.io

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[Donald Stufft]

> On Oct 29, 2014, at 11:37 AM, Steve Dower  wrote:
> 
> Antoine Pitrou wrote:
>> On Wed, 29 Oct 2014 11:07:53 -0400
>> Tres Seaver  wrote:
>>> -BEGIN PGP SIGNED MESSAGE-
>>> Hash: SHA1
>>> 
>>> On 10/29/2014 10:31 AM, R. David Murray wrote:
>>> 
>>>> If you are writing code targeted for Windows, I think you are very 
>>>> likely to have an MSDN subscription of some sort if your package 
>>>> includes C code.  I'm sure it's not 100%, though.
>>> 
>>> My experience with distributing distributions-with-extensions 
>>> indicates that the vast majority of Windows developers who are 
>>> "downstream" users for those distributions *cannot* build them from 
>>> source:  if there is no MSI / bdist_win (maybe now wheel), they won't use 
>>> the project.
>>> 
>>> (Note that "having an MSDN subscription" is not the same as "knowing 
>>> how to configure which compiler such that it can bulid extensions 
>>> against an installed Python binary").
>> 
>> I don't think you have to configure anything. Just install the right Visual 
>> Studio version and it's done.
> 
> The tricky part here is the *right* Visual Studio version. As many have 
> noted, there were bugs/missing components in some of the older Express 
> editions (like the 64-bit compiler integration), and even the compiler pack 
> we released recently doesn't actually help building Python itself, though 
> it's good for extensions. In general, VS 2008 Professional and VS 2010 
> Professional are easiest to set up if you can get them, the C++ Express 
> editions are fairly easy to set up, and the Windows SDK is difficult but 
> possible (and won't currently build CPython either, as the build depends on 
> VS - I'm also fixing this for 3.5).
> 
> My ideal target (Utopia refined to be achievable) is for python-dev to handle 
> the Python binaries themselves (already done) and to make them easy to bundle 
> with applications/etc (I'm working on this for 3.5), and for PyPI to include 
> pre-built wheels for Windows where necessary.
> 
> Note that I don't require package developers to build their own wheels, 
> though I think that they are generally the right people to be doing it. It 
> would be nice to create a culture of delegation, so that "someone" could be a 
> trusted builder for a range of packages (for example, I'd love it if 
> Continuum/Enthought/similar could provide their builds of numpy/scipy as 
> wheels and those projects would be willing to have them be the official PyPI 
> downloads). This is roughly how the python.org installers are handled, and 
> while it may cause some lag between source and binary releases, I think the 
> release cycles are slow enough that most users would only notice that "pip 
> install numpy" now works.

For the record, a long term “down the road” kind of thing I want to do is have 
PyPI have a build farm which can build Wheels. So that people upload a source 
distribution to PyPI and it kicks off Wheel builds on the various platforms 
automatically.

What this looks like is a complete unknown, all I have is the general idea.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Impact of Windows PowerShell OneGet ?

[Donald Stufft]

> On Oct 29, 2014, at 3:34 PM, Glenn Linderman  wrote:
> 
> New package manager from M$... article here 
> <http://www.neowin.net/news/windows-10-oneget-a-linux-style-package-management-framework>.
> 
> It seems doubtful that M$ will eliminate .msi (their obscure, hard to 
> configure and use, installation format), so it seems doubtful that the 
> addition of OneGet will _force_ any changes to Python packaging.
> 
> However, it does open the question in my mind about whether there will be any 
> _benefits_ of OneGet that would inspire helpful, useful changes to Python 
> packaging. They speak of "trusted repositories", and the like, and it sounds 
> like a the various *nix package managers (apt-get, et alia), but perhaps 
> allowing multiple repositories rather than just a single source vendor 
> repository (I'm actually not sure if *nix package managers allow multiple 
> repositories or not, but from the way people talk about them, it always 
> sounds like a "distribution" also provides "a repository" of additional 
> packages).
> 
> "trusted repositories" sounds more like Perl's CPAN.
> 
> One of the links 
> <http://blogs.technet.com/b/windowsserver/archive/2014/04/03/windows-management-framework-v5-preview.aspx>
>  contains this quote: "This first version of OneGet installs and searches 
> software from Chocolatey repositories.  Support of additional repositories 
> will come in subsequent versions."
> 
> I have no clue what a Chocolatey repository is (yet, will Google), but 
> unknown others will come, it says... whether it is possible to write a 
> "repository plugin" such that Perl's CPAN or Python's PyPI or other 
> preexisting repositories can be accessed is not clear.
> 
> The relationship between PowerShell and OneGet is not clear either... is 
> OneGet written in PowerShell, or is PowerShell just one way to invoke OneGet, 
> or???
> 
> Just a heads up.
> 

It appears to be a package manager manager. Chocolatey is one of the third 
party package managers available on Windows.

I also just learned that OneGet is apparently OSS and developed on github 
(https://github.com/OneGet/oneget <https://github.com/OneGet/oneget>).

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[Donald Stufft]

This sounds like something good for packaging.python.org


> On Oct 29, 2014, at 4:05 PM, Paul Moore  wrote:
> 
> On 29 October 2014 15:31, Nathaniel Smith  wrote:
>>> You can use Express editions of Visual Studio.
>> 
>> IIUC, the express edition compilers are 32-bit only, and what you actually
>> want are the "SDK compilers":
>> https://github.com/cython/cython/wiki/64BitCythonExtensionsOnWindows
>> 
>> These are freely downloadable by anyone, no msdn subscription required, but
>> only if you know where to find them!
>> 
>> AFAICT the main obstacle to using MSVC to build python extensions (assuming
>> it can handle your code at all) is not anything technical, but rather that
>> there's no clear and correct tutorial on how to do it, and lots of confusion
>> and misinformation circulating.
> 
> Would it help if I wrote a document explaining how to set up the MS
> compilers (free and paid for) to allow building of Python extensions?
> 
> There are a few provisos:
> 
> 1. A lot of it will be pretty trivial: "If you have Vistal Studio
> (full edition), install it. Done."
> 2. It will be out of date very fast as the situation for Python 3.5+
> will be trivial across the board.
> 3. I don't have anywhere particularly authoritative to host it (maybe
> the Python Wiki?) and it could easily get lost in the huge swamp of
> variously outdated, over-complicated, or otherwise alternative
> documents available. Ideally I'd like someone to suggest an "official"
> location I could use.
> 
> I don't want to do this if it won't be useful, as it'll take me a bit
> of effort to confirm the process for the only non-trivial scenario
> (64-bit Python 3.3/3.4 with free tools). But if people think it would
> help, that's fine, I volunteer.
> 
> Paul
> 
> PS Even if I don't get positive feedback, I may just say "to heck with
> it" and do it anyway, because it *is* so trivial :-) I just won't
> promise.
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/donald%40stufft.io
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[Donald Stufft]

> On Oct 29, 2014, at 6:09 PM, Paul Moore  wrote:
> 
> On 29 October 2014 20:26, Donald Stufft  wrote:
>> This sounds like something good for packaging.python.org
> 
> Yeah, I wondered about that. I'll work up a patch for that. But the
> more I think about it, it really is trivial:
> 
> - For non-free MSVC, install the appropriate version, and everything just 
> works.
> - For Python 2.7 (32 or 64 bit), install the compiler for Python 2.7
> package and everything just works as long as you're using setuptools.
> - For 32 bit Python 3.2-3.4, install Visual Studio Express and
> everything just works.
> - For 64 bit Python 3.2-3.4, install the SDK, set some environment
> variables, and everything just works.
> - For Python 3.5+, install the current Visual Studion Express and
> everything just works.
> 
> (I propose to deem Python 2.7 without setuptools as "unsupported")
> 
> The only things I can see that need expansion are:
> 
> 1. The precise versions to use (trivial to add, I'm just to lazy to
> check right now).
> 2. Where to get VS 2010 Express (as it's no longer supported or
> available from MS).
> 3. How to set up the SDK environment for 64-bit Python 3.2-3.4.
> 
> Before I do a write-up, I want to set up clean VMs with these
> configurations, so that I can confirm the details. But I don't expect
> any problems, as I've done them all before.
> 
> Paul.

I think it’d be good even if considered trivial. I know I certainly have no
idea which pieces I needed to do that.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[Donald Stufft]

> On Oct 29, 2014, at 7:02 PM, Ethan Furman  wrote:
> 
> On 10/29/2014 03:46 PM, Paul Moore wrote:
>> On 29 October 2014 22:19, Ethan Furman  wrote:
>>> 
>>>   - where one should be at when one starts the compile process
>> 
>> I don't understand this. It's just "pip wheel foo" to build a wheel
>> for foo (which will be downloaded), or "pip wheel ." or  "python
>> setup.py bdist_wheel" as you prefer for a local package.
> 
> Hmmm...  That looks like it's for installing/compiling somebody else's 
> package.  Is that last command sufficient to prepare one's own wheel for 
> uploading to PyPI, or there something else to do?
> 

Generally for uploading to PyPI you do ``python setup.py bdist_wheel``, though 
I don’t think there’d be any bad thing if you used pip wheel.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 2.7.9 schedule redux

[Donald Stufft]

> On Oct 31, 2014, at 7:54 PM, Benjamin Peterson  wrote:
> 
> I'm updating the 2.7 release PEP with the following release dates
> 
> 2.7.9rc1 Nov 22
> 2.7.9 final December 5
> 
> I expect PEPs 476 and 477 can be done by then?
> 
> Given that 2.7.9 has had some large changes, having multiple rcs is not
> unlikely. Hopefully, we can get a final out by 2015, though.

Yes on PEP 477, the actual back porting is done, I’m just slacking on doing
the tests because we used unittest.mock in 3.4.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 2.7.9 schedule redux

[Donald Stufft]

> On Nov 1, 2014, at 8:05 AM, Nick Coghlan  wrote:
> 
> On 1 November 2014 09:59, Donald Stufft  wrote:
>> 
>>> On Oct 31, 2014, at 7:54 PM, Benjamin Peterson  wrote:
>>> 
>>> I'm updating the 2.7 release PEP with the following release dates
>>> 
>>> 2.7.9rc1 Nov 22
>>> 2.7.9 final December 5
>>> 
>>> I expect PEPs 476 and 477 can be done by then?
>>> 
>>> Given that 2.7.9 has had some large changes, having multiple rcs is not
>>> unlikely. Hopefully, we can get a final out by 2015, though.
>> 
>> Yes on PEP 477, the actual back porting is done, I’m just slacking on doing
>> the tests because we used unittest.mock in 3.4.
> 
> mock is only a single file, so making it available to the Python 2.7
> regression test suite as something like test._mock_backport sounds
> tolerable to me.
> 
> Another challenge is that the functional tests (i.e. the ones that
> actually run pip from the bundled wheel file, rather than mocking it
> out) are in test_venv, since they rely on venv to provide a degree of
> isolation from the Python instance running the test suite.
> 
> However, given that pip's own tests cover compatibility with the 2.x
> series, the Python 3 test suite covers the functional integration with
> ensurepip, and the mock-based tests for ensurepip cover the 2.7
> backport itself, I think we still have an acceptable level of
> regression testing coverage, even without being able to backport the
> venv based functional tests.

Ok. The actual delta between ensurepip in 3.4 and in 2.7.9 is quite
small too. It’s really just removing things like def foo(*, bar=None)
and tempfile.TemporaryDirectory. No logic changes, just syntax/helper
stuff.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] problem with hg.python.org?

[Donald Stufft]

> On Nov 1, 2014, at 10:37 PM, Terry Reedy  wrote:
> 
> My normally dependable pull.bat script has 3 times given me this.
> 
> F:\Python\dev>hg --repository F:\Python\dev\5\py35 pull --verbose --config 
> ui.merge=internal:merge ssh://h...@hg.python.org/cpython
> pulling from ssh://h...@hg.python.org/cpython
> searching for changes
> all local heads known remotely
> adding changesets
> adding manifests
> adding file changes
> transaction abort!
> rollback completed
> abort: data/Doc/howto/webservers.rst.i@5f4ad429ae9f: unknown parent!
> remote: Unable to write to standard output: The pipe is being closed.
> 
> Since I have no 'data/Doc/i@5f9f, I am guessing that the problem is 
> not local.
> 
> -- 
> Terry Jan Reedy
> 
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/donald%40stufft.io

The internet suggests trying hg verify, does that do anything?

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?

[Donald Stufft]

> On Nov 21, 2014, at 10:26 AM, Barry Warsaw  wrote:
> 
> On Nov 21, 2014, at 10:36 PM, Nick Coghlan wrote:
> 
>> I'd been taking "must be hosted in PSF infrastructure" as a hard
>> requirement, but MAL pointed out earlier this evening that in the age
>> of DVCS's, that requirement may not make sense: if you avoid tightly
>> coupling your automation to a particular DVCS host's infrastructure,
>> then reverting back to self-hosting (if that becomes necessary for
>> some reason) is mostly just a matter of "hg push".
>> 
>> If that "must be self-hosted" constraint is removed, then the obvious
>> candidate for Mercurial hosting that supports online editing + pull
>> requests is the PSF's BitBucket account.
> 
> For the record, I object to moving *official* PSF resources to proprietary,
> closed-source infrastructure that we do not control or have access to[*].
> 
> As nice and friendly as BitBucket or any other code hosting source is today,
> there are many reasons why I think this is a bad idea for *official*
> branches.  We are beholden to their policies and operations, which may not
> align with PSF policies or principles today or in the future.  We will not be
> able to customize the experience for our own needs.  We will not have access
> to the underlying resources should we need them for any purpose.  We cannot
> take action ourselves if some problem occurs, e.g. banning an offensive user.
> 
> You're right that in a world of dvcs, branches can be mirrored anywhere.  For
> that reason, I have no problem allowing developers to use non-PSF controlled
> resources *unofficially* if it makes their work easier and doesn't conflict
> with their own principles.  However, in such cases, I still believe that the
> official, master, blessed repositories remain on PSF controlled
> infrastructure.  Surely that too is possible in the world of dvcs, right?
> 
> Cheers,
> -Barry
> 
> [*] Please note that I am not objecting to our use of lower-level resources
> donated by our generous sponsors.  It's a fine line perhaps, but I have no
> problem with a wiki running on a VM hosted on some donated hardware, since we
> still have full access to the machine, the OS, and the application software.

Personally I care less about proprietary and closed-source and care a lot more
about lock-in. Thus my big problem using Bitbucket for these things is that if
we ever want to *leave* bitbucket it becomes a lot harder because you have a
bunch of links and such pointing at bitbucket instead of a python.org domain.
They do offer a redirect feature but that is dependent on them not taking that
away in the future. (They also offer a CNAME feature but if you use it you lose
the ability to use TLS, which is also a non starter for me). Sadly this also
leaves out my favorite host site of Github :/. Something like Github Enterprise
or Atlassian stash which are able to be migrated away from are better in this
regards.

Ironically we do use a propetiary/closed-source/hosted solution for
https://status.python.org/ but it’s not terribly difficult to migrate away from
that if we ever wanted to.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators

[Donald Stufft]

> On Nov 21, 2014, at 11:31 AM, Ethan Furman  wrote:
> 
> On 11/21/2014 05:47 AM, Raymond Hettinger wrote:
>> 
>> Also, the proposal breaks a reasonably useful pattern of calling 
>> next(subiterator)
>> inside a generator and letting the generator terminate when the data stream  
>> ends.
>> 
>> Here is an example that I have taught for years:
>> 
>>def [...]
>>it1 = iter(iterable1)
>>it2 = iter(iterable2)
>>while True:
>>v1 = next(it1)
>>v2 = next(it2)
>>yield v1, v2
> 
> Stepping back a little and looking at this code, sans header, let's consider 
> the possible desired behaviors:
> 
>  - have an exact match-up between the two iterators, error otherwise
>  - stop when one is exhausted
>  - pad shorter one to longer one
> 
> Two of those three possible options are going to require dealing with the 
> StopIteration that shouldn't escape -- is the
> trade of keeping one option short and simple worth the pain caused by the 
> error-at-a-distance bugs caused when a
> StopIteration does escape that shouldn't have?
> 

I don’t have an opinion on whether it’s enough of a big deal to actually change
it, but I do find wrapping it with a try: except block and returning easier
to understand. If you showed me the current code unless I really thought about
it I wouldn't think about the fact that the next() calls can cause the
generator to terminate.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?

[Donald Stufft]

s
that the patch is perfectly acceptable and you just need to merge it in, the
Github flow is:

1. Press big green merge button, rejoice.
2. Possibly manually merge into other branches
   - This can be done via PRs if you want as well, since you can make a PR that
 merges one branch into another via the web UI and press the big green
 button on that too.

The current flow is:

1. Download the .patch file.
2. Apply the patch file.
   - Unlike in the PR style flow, you can’t know ahead of time if this
 patch is going to cleanly apply or not. This may involve needing
 to rework the patch which makes it hard to judge the relative level
 of effort to land any one patch.
3. Commit and push the changes.
4. Possibly manually merge into other branches.

Again the PR flow here takes almost no time, like 10 seconds once you’ve 
reviewed
the PR and decided it’s good to land. Maybe another minute or two if you’re 
going
to use the web to merge it into other branches, or a few extra minutes if you’re
going to do it manually. The current flow is again probably a good 10-15 minutes
worth of effort to land even a tiny one letter change.

I’m not going to say that we need to switch to Github/Bitbucket to get this sort
of ease of use however those platforms do come with it and it is pretty awesome.
There are self hosted options that offer similar things (often times not quite 
as
good in the ease of use though they might be more featureful in other areas).

Mostly what I’m trying to say is that documenting a field that essentially 
requires
the end user to not only figure out how to use mercurial, but figure out how to
host a public repository somewhere is not even really within the same order of
magnitude in ease of use.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?

[Donald Stufft]

> On Nov 21, 2014, at 3:59 PM, Ned Deily  wrote:
> 
> In article <19336614-0e4f-42bf-a918-1807bb7f3...@stufft.io>,
> Donald Stufft  wrote:
> [...]
>> Well you can’t document your way out of a bad UX. The thing you’re
>> competing with (on Github at least) is:
>> 
>> 1. I notice a docs change I can make
>> 2. I click the “Edit” button and it automatically creates a fork,
>>   and opens up a web based editor where I can make changes to
>>   that file.
>> 3. I fill out the commit message and hit Save.
>> 4. An automatic pull request is opened up with my changes.
>> 
>> I can submit a fix for a typo in the docs in like ~30 seconds assuming
>> I already have a Github account. That sort of instant win is a great
>> way to get people started contributing to a project and can lead later
>> to bigger more substantial contributions.
> [...]
>> Mostly what I’m trying to say is that documenting a field that essentially 
>> requires
>> the end user to not only figure out how to use mercurial, but figure out how 
>> to
>> host a public repository somewhere is not even really within the same order 
>> of
>> magnitude in ease of use.
> 
> Sure, I get that.  But we're not even talking here about the main Python 
> docs since they are part of the CPython repos, only ancillary repos like 
> PEPs and the developer's guide.  The level of activity on those is quite 
> small.  So, thinking about it a bit more, PEPs don't normally have bug 
> tracker issues associated with them so I suppose my concerns about issue 
> tracker aren't a major concern for them.  The dev guide does get issues 
> opened about it and I suppose they could be managed.  But, without 
> tackling the CPython repo workflow (a *much* bigger deal), is the 
> divergence in workflows worth it?  I dunno.

Yea for the smaller repositories I don’t have a whole lot of opinion
about if the benefit buys us much, especially since one of the goals
is new-person friendliness but the problem is that it doesn’t translate
to contributing to CPython itself.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?

[Donald Stufft]

> On Nov 22, 2014, at 9:59 AM, Nick Coghlan  wrote:
> 
> 
> On 22 Nov 2014 07:37, "Donald Stufft"  <mailto:don...@stufft.io>> wrote:
> > > On Nov 21, 2014, at 3:59 PM, Ned Deily  > > <mailto:n...@acm.org>> wrote:
> > > Sure, I get that.  But we're not even talking here about the main Python
> > > docs since they are part of the CPython repos, only ancillary repos like
> > > PEPs and the developer's guide.  The level of activity on those is quite
> > > small.  So, thinking about it a bit more, PEPs don't normally have bug
> > > tracker issues associated with them so I suppose my concerns about issue
> > > tracker aren't a major concern for them.  The dev guide does get issues
> > > opened about it and I suppose they could be managed.  But, without
> > > tackling the CPython repo workflow (a *much* bigger deal), is the
> > > divergence in workflows worth it?  I dunno.
> 
> I also think the tutorial and howto guides should be broken out of the main 
> CPython repo &  made version independent (with internal version specific 
> notes).
> 
> That offers no compelling advantages right now, but becomes far more 
> beneficial if it comes with a switch to enabling online editing.
> 
> > Yea for the smaller repositories I don’t have a whole lot of opinion
> > about if the benefit buys us much, especially since one of the goals
> > is new-person friendliness but the problem is that it doesn’t translate
> > to contributing to CPython itself.
> 
> OK, different question. Has anyone here actually even *read* the workflow 
> PEPs I wrote? They were on the agenda for the language summit, but got bumped 
> due to lack of time (which I'm still annoyed about, given the comparatively 
> inconsequential things that chewed up a whole lot of the day).
> 
> I've only had a couple of folks from outside the core dev community express 
> interest in them. Personally, the lack of online editing support annoys me 
> immensely whenever I need to work on PEPs or the devguide. I also think it's 
> ridiculous that we have dozens (hundreds?) of folks running community 
> workshops, and all creating their own custom documentation, rather than us 
> finding a way to better enable their collaboration on the official tutorial.
> 
> The BitBucket proposal in this thread came out of a desire to avoid adding 
> yet more work to an understaffed group of primarily volunteers maintaining 
> the infrastructure (the paid admins are more focused on incident response and 
> general infrastructure support, rather than spinning up new workflow 
> services).
> 
> My preferred answer remains setting up a srlf-hosted forge.python.org 
> <http://forge.python.org/>, but I've seen little evidence we have the 
> capacity to deploy & maintain such a service effectively, given the relative 
> lack of interest shown in the idea by almost everyone I've spoken to about 
> it. Any progress has only come with a lot of pushing from me, and I just 
> don't have the personal bandwidth to sustain that at this point. That's why 
> the related PEPs were deferred, and the only responses I've received 
> regarding potentially taking them over have come from folks outside the core 
> development community, which really doesn't help very much in removing my 
> availability as a bottleneck in the workflow improvement process.
> 
> If nobody wants to maintain a self-hosted forge, or even enable the folks 
> that have expressed interest in setting it up & maintaining it, then the 
> right answer is "don't do it" - we should use a commercial service instead.
> 
> 

I think I told you before, but if not I’ll say it now :)

From the infrastructure side spinning up a new VM is pretty easy, what we need
is someone to write some salt states in the psf-salt repository that will
deploy an instance of whatever software. I don't have time to figure out the
intracities of the various software and deploy them but I can help with
figuring out our infra layout. The other thing the infrastructure side would
need is some guidance on what size/flavor of Rackspace VM it needs and what
additional services it would need (we have a PostgreSQL database already, does
it need extra disk space? A Queue? Object Store?). There is a more or less
working vagrant setup in the psf-salt[1] repository (a few of the roles don't
work yet without manual configuration, like the hg role) but for what someone
would need for setting up a new service it should all be there.

I'm not sure what else we can do to enable someone who *does* want to work on
it to be able to set it up. However I'm not against using a comme

Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?

[Donald Stufft]

> On Nov 23, 2014, at 12:19 AM, Guido van Rossum  wrote:
> 
> This thread seems to beg for a decision. I think Donald Stufft has it exactly 
> right: we should move to GitHub, because it is the easiest to use and most 
> contributors already know it (or are eager to learn thee). Honestly, the time 
> for core devs (or some other elite corps of dedicated volunteers) to sysadmin 
> their own machines (virtual or not) is over. We've never been particularly 
> good at this, and I don't see us getting better or more efficient.
> 
> Moving the CPython code and docs is not a priority, but everything else 
> (PEPs, HOWTOs etc.) can be moved easily and I am in favor of moving to 
> GitHub. For PEPs I've noticed that for most PEPs these days (unless the 
> primary author is a core dev) the author sets up a git repo first anyway, and 
> the friction of moving between such repos and the "official" repo is a pain.
> 

Heck I am a Core dev technically and I setup a git repo first until it’s moved 
into the hg.python.org <http://hg.python.org/>.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?

[Donald Stufft]

> On Nov 23, 2014, at 12:59 AM, Nick Coghlan  wrote:
> 
> On 23 November 2014 at 15:19, Guido van Rossum  wrote:
>> This thread seems to beg for a decision. I think Donald Stufft has it
>> exactly right: we should move to GitHub, because it is the easiest to use
>> and most contributors already know it (or are eager to learn thee).
>> Honestly, the time for core devs (or some other elite corps of dedicated
>> volunteers) to sysadmin their own machines (virtual or not) is over. We've
>> never been particularly good at this, and I don't see us getting better or
>> more efficient.
> 
> The learning curve on git is still awful - it offers no compelling
> advantages over hg, and GitHub doesn't offer any huge benefits over
> BitBucket for Sphinx based documentation (ReadTheDocs works just as
> well with either service).

It does have one very big compelling advantage. It’s way more popular.

Besides, the learning curve on hg isn’t any better, it’s just differently
hard.

> 
>> Moving the CPython code and docs is not a priority, but everything else
>> (PEPs, HOWTOs etc.) can be moved easily and I am in favor of moving to
>> GitHub. For PEPs I've noticed that for most PEPs these days (unless the
>> primary author is a core dev) the author sets up a git repo first anyway,
>> and the friction of moving between such repos and the "official" repo is a
>> pain.
> 
> Note that if folks prefer Git, BitBucket supports both. I would object
> strongly to unilaterally forcing existing contributors to switch from
> Mercurial to git.

Going to all the trouble to move to an external repository and choosing the
least popular option out of the two main ones seems like a bad idea in
general.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?

[Donald Stufft]

> On Nov 23, 2014, at 1:03 AM, Donald Stufft  wrote:
> 
>> 
>> On Nov 23, 2014, at 12:59 AM, Nick Coghlan  wrote:
>> 
>> On 23 November 2014 at 15:19, Guido van Rossum  wrote:
>>> This thread seems to beg for a decision. I think Donald Stufft has it
>>> exactly right: we should move to GitHub, because it is the easiest to use
>>> and most contributors already know it (or are eager to learn thee).
>>> Honestly, the time for core devs (or some other elite corps of dedicated
>>> volunteers) to sysadmin their own machines (virtual or not) is over. We've
>>> never been particularly good at this, and I don't see us getting better or
>>> more efficient.
>> 
>> The learning curve on git is still awful - it offers no compelling
>> advantages over hg, and GitHub doesn't offer any huge benefits over
>> BitBucket for Sphinx based documentation (ReadTheDocs works just as
>> well with either service).
> 
> It does have one very big compelling advantage. It’s way more popular.
> 
> Besides, the learning curve on hg isn’t any better, it’s just differently
> hard.
> 
>> 
>>> Moving the CPython code and docs is not a priority, but everything else
>>> (PEPs, HOWTOs etc.) can be moved easily and I am in favor of moving to
>>> GitHub. For PEPs I've noticed that for most PEPs these days (unless the
>>> primary author is a core dev) the author sets up a git repo first anyway,
>>> and the friction of moving between such repos and the "official" repo is a
>>> pain.
>> 
>> Note that if folks prefer Git, BitBucket supports both. I would object
>> strongly to unilaterally forcing existing contributors to switch from
>> Mercurial to git.
> 
> Going to all the trouble to move to an external repository and choosing the
> least popular option out of the two main ones seems like a bad idea in
> general.
> 

Also important to note, that if people want to use hg on their own, it’s pretty
easy to do that with https://hg-git.github.io/. Last I heard that works pretty
well still. I’ve yet to find one that works the other direction that doesn’t
choke on some repositories.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?

[Donald Stufft]

> On Nov 23, 2014, at 1:25 AM, Nick Coghlan  wrote:
> 
> On 23 November 2014 at 16:03, Donald Stufft  wrote:
>>> On Nov 23, 2014, at 12:59 AM, Nick Coghlan  wrote:
>>> 
>>> Note that if folks prefer Git, BitBucket supports both. I would object
>>> strongly to unilaterally forcing existing contributors to switch from
>>> Mercurial to git.
>> 
>> Going to all the trouble to move to an external repository and choosing the
>> least popular option out of the two main ones seems like a bad idea in
>> general.
> 
> Moving repos from hg.python.org to bitbucket.org is just a matter of
> switching some URLs around, and changing some backend systems to cope
> with the new location. The end result should be to make life better
> for existing contributors *and* new contributors using the web UI, and
> be largely transparent to folks using command line tools.
> 
> By contrast, proposals to switch from Mercurial to Git impose a
> *massive* burden on contributors that don't already know git. That
> significant increase in the time investment required will provide *NO*
> practical benefit for existing contributors (this is coming from
> someone that has used git and Mercurial in parallel for years - trust
> me, they're functionally isomorphic), and only make life marginally
> easier for potential new contributors (you can log in to BitBucket
> with your GitHub ID, and the functional isomorphism means that many
> folks already use tools like git-remote-hg  to use the git command
> line to interact with the hg.python.org Mercurial repos).

Yea, but then you lose out on the entire ecosystem built around Github.

Like you won’t be able to run travis tests on the docs to make sure that
any Pull Requests don’t silently start breaking the ability to build the
docs.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?

[Donald Stufft]

d Mercurial
> repos is a low risk change. It reduces maintenance overhead and lowers
> barriers to external contribution, both without alienating existing
> contributors by forcing them to change their workflows.
> 
> Proposing to *also* switch from Mercurial to git significantly
> increases the cost of the change, while providing minimal incremental
> benefit.
> 
> Regards,
> Nick.
> 
> -- 
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/donald%40stufft.io

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?

[Donald Stufft]

> On Nov 23, 2014, at 2:01 AM, Nick Coghlan  wrote:
> 
> On 23 November 2014 at 16:27, Donald Stufft  wrote:
>>> On Nov 23, 2014, at 1:25 AM, Nick Coghlan  wrote:
>>> By contrast, proposals to switch from Mercurial to Git impose a
>>> *massive* burden on contributors that don't already know git. That
>>> significant increase in the time investment required will provide *NO*
>>> practical benefit for existing contributors (this is coming from
>>> someone that has used git and Mercurial in parallel for years - trust
>>> me, they're functionally isomorphic), and only make life marginally
>>> easier for potential new contributors (you can log in to BitBucket
>>> with your GitHub ID, and the functional isomorphism means that many
>>> folks already use tools like git-remote-hg  to use the git command
>>> line to interact with the hg.python.org Mercurial repos).
>> 
>> Yea, but then you lose out on the entire ecosystem built around Github.
>> 
>> Like you won’t be able to run travis tests on the docs to make sure that
>> any Pull Requests don’t silently start breaking the ability to build the
>> docs.
> 
> Travis isn't the only CI system on the internet, and for pure Sphinx
> documentation cases, ReadTheDocs runs just as well off BitBucket as it
> does off GitHub.

Sure it’s not the only CI system, but as far as I know bitbucket doesn’t
have near the integration possible with CI systems. I make a PR on github
I get it tested and the merge button turns green to let me know that
the tests run. Travis is popular enough that I’ve seen bitbucket projects
hosting a mirror on github *just* for travis.

> 
> I personally think changing version control systems would be an
> incredibly bad idea, and consider it completely out of scope for
> discussions of Mercurial repo hosting arrangements. There's a lot that
> could be done, with much lower impact, just by changing the way we
> manage the existing Mercurial repos. When we can't even work out the
> practical details of getting those implemented, suggestions of
> "git/GitHub will fix it!" sound like mere wishful thinking.
> 
> Cheers,
> Nick.
> 
> -- 
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?

[Donald Stufft]

> On Nov 23, 2014, at 2:35 AM, Nick Coghlan  wrote:
> 
> On 23 November 2014 at 17:14, Donald Stufft  wrote:
>>> On Nov 23, 2014, at 2:01 AM, Nick Coghlan  wrote:
>>> Travis isn't the only CI system on the internet, and for pure Sphinx
>>> documentation cases, ReadTheDocs runs just as well off BitBucket as it
>>> does off GitHub.
>> 
>> Sure it’s not the only CI system, but as far as I know bitbucket doesn’t
>> have near the integration possible with CI systems. I make a PR on github
>> I get it tested and the merge button turns green to let me know that
>> the tests run. Travis is popular enough that I’ve seen bitbucket projects
>> hosting a mirror on github *just* for travis.
> 
> In the absence of a proposal to change version control systems
> (again), the lack of Mercurial hosting on GitHub makes it rather a
> moot point. Given that we can barely muster up any enthusiasm for
> rehosting *without* changing version control systems (and the number
> of CI systems that integrate with hg.python.org repos other than the
> main CPython one is exactly zero), any proposal that involves doing
> even *more* work seems doubly doomed.
> 

I’d volunteer to do the work to get the PEPs, and possibly other repositories
onto Github if we so decided to do so. Don’t let the lack of volunteer stop
that because I will find the time to do it if need be.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?

[Donald Stufft]

I 
> would like to a list and to have the difficulty of moving them mentioned to 
> know what the impact would be.

Off the top of my head for the docs I know of 
https://github.com/python/docsbuild-scripts/blob/master/build_docs.py 
<https://github.com/python/docsbuild-scripts/blob/master/build_docs.py> which 
will need to be updated to git clone instead of hg clone.

For the record here is the list of commit hooks on the devguide: 
https://bpaste.net/show/3b151d0ddcb4 <https://bpaste.net/show/3b151d0ddcb4> but 
as I said above, those will need to be written on either platform because to my 
knowledge none of the hosted sites allow you to upload arbitrary commit hooks 
that get executed on their machines.

I’m struggling to think of what other automation we might have for 
documentation. There’s no buildbot on anything that isn’t CPython.

>  
> Orchestrating this kind of infrastructure enhancement for Red Hat *is* my day 
> job, and you almost always want to go for the lowest impact, lowest risk 
> approach that will alleviate the bottleneck you're worried about while 
> changing the smallest possible number of elements in the overall workflow 
> management system.
> 
> 
> Sure, but I would never compare our infrastructure needs to Red Hat. =) You 
> also have to be conservative in order to minimize downtown and impact for 
> cost reasons. As an open source project we don't have those kinds of worry; 
> we just have to worry about keeping everyone happy.

I think the other consideration here is that when your workforce is volunteers 
and not people you’re paying to do things you have to be cognizant of what the 
volunteers are willing to do as well. Nick has said he’s having problem getting 
interested parties who are motivated to make the move however I am willing to 
put in the work to move to Github, I’m not however willing to do it for 
Bitbucket because it’s frankly about as bad to me personally as the current 
arrangement is. This isn’t to say there isn’t someone who wouldn’t do the work 
for bitbucket of course, just going off Nick saying he’s having problems 
finding someone willing to do that work.

>  
> That underlying calculation doesn't really change much even when the units 
> shift from budget dollars to volunteers' time and energy.
> 
> So here is what I want to know to focus this discussion:
> 
> First, what new workflow are you proposing regardless of repo hosting 
> provider? Are you proposing we maintain just mirrors and update the devguide 
> to tell people to fork on the hosting provider, make their changes, generate 
> a patch (which can be as simple as telling people how find the raw diff on 
> the hosting provider), and then upload the patch the issue tracker just like 
> today? Are you going farther and saying we have people send PRs on the 
> hosting site, have them point to their PR in the issue tracker, and then we 
> accept PRs (I'm going on the assumption we are not dropping our issue 
> tracker)?
> 
> Second, to properly gauge the impact of switching from git to hg from an 
> infrastructure perspective, what automation scripts do we have and how 
> difficult would it be to update them to use git instead of hg? This is 
> necessary simply to know where we would need to update URLs, let alone change 
> in DVCS.
> 
> Third, do our release managers care about hg vs. git strongly? They probably 
> use the DVCS the most directly and at a lower level by necessity compared to 
> anyone else.
> 
> Fourth, do any core developers feel strongly about not using GitHub? Now 
> please notice I said "GitHub" and not "git"; I think the proper way to frame 
> this whole discussion is we are deciding if we want to switch to Bitbucket or 
> GitHub who provide a low-level API for their version control storage service 
> through hg or git, respectively. I personally dislike git, but I really like 
> GitHub and I don't even notice git there since I use GitHub's OS X app; as I 
> said, I view this as choosing a platform and not the underlying DVCS as I 
> have happily chosen to access the GitHub hosting service through an app that 
> is not git (it's like accessing a web app through it's web page or its REST 
> API).
> 
> At least for me, until we get a clear understanding of what workflow changes 
> we are asking for both contributors and core developers and exactly what work 
> would be necessary to update our infrastructure for either Bitbucket or 
> GitHub we can't really have a reasonable discussion that isn't going to be 
> full of guessing.
> 
> And I'm still in support no matter what of breaking out the HOWTOs and the 
> tutorial into their own repos for easier updating (having to update the 
> Python porting HOWTO in three 

Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?

[Donald Stufft]

> On Nov 23, 2014, at 3:03 PM, Georg Brandl  wrote:
> 
> The next point is that there is no easy way to change the target branch of
> a pull request (on github or bitbucket).  People will usually make patches
> against the master branch unless told differently explicitly, which means
> that the pull request will also be against the master branch.  Which means,
> close the PR, ask submitter to resubmit against 3.x branch, or do it
> yourself.

This in particular is not really true on Github at least. By default PRs are
made against whichever branch you have configured as the default branch in
the Github UI. This does default to master but it doesn’t have to be, for
instance pip has this configured for the develop branch. Although I think 
this specific point is moot because if things were on Git it’d probably make
the most sense to have the default integration branch be ``master`` just like
for the Hg repos they use default.

Even if someone makes a PR against the wrong branch, it “degrades” into
essentially the same UX as you have now, you can turn a PR into a patch by
adding a .patch or .diff to the end of the PR URL and then you have a patch
file. In additional github makes it easy to check out PRs directly with only a
minor bit of one time configuration in your local clone. I can checkout any PR
by doing ``git checkout pr/1`` (replacing 1 with whatever the number is).

Honestly the worst part about someone sending a PR to the wrong branch is it
degrades into the same terrible UX that *every* patch has to go through on
a hg.python.org repository right now.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?

[Donald Stufft]

> On Nov 23, 2014, at 4:08 PM, Georg Brandl  wrote:
> 
> On 11/23/2014 09:38 PM, Donald Stufft wrote:
>> 
>>> On Nov 23, 2014, at 3:03 PM, Georg Brandl  wrote:
>>> 
>>> The next point is that there is no easy way to change the target branch of
>>> a pull request (on github or bitbucket).  People will usually make patches
>>> against the master branch unless told differently explicitly, which means
>>> that the pull request will also be against the master branch.  Which means,
>>> close the PR, ask submitter to resubmit against 3.x branch, or do it
>>> yourself.
>> 
>> This in particular is not really true on Github at least. By default PRs are
>> made against whichever branch you have configured as the default branch in
>> the Github UI. This does default to master but it doesn’t have to be, for
>> instance pip has this configured for the develop branch. Although I think 
>> this specific point is moot because if things were on Git it’d probably make
>> the most sense to have the default integration branch be ``master`` just like
>> for the Hg repos they use default.
> 
> Sure, although as is the majority of commits to CPython are bugfixes, which
> normally go to 2.7, 3.4 and 3.5 (master).
> 
>> Even if someone makes a PR against the wrong branch, it “degrades” into
>> essentially the same UX as you have now, you can turn a PR into a patch by
>> adding a .patch or .diff to the end of the PR URL and then you have a patch
>> file. In additional github makes it easy to check out PRs directly with only 
>> a
>> minor bit of one time configuration in your local clone. I can checkout any 
>> PR
>> by doing ``git checkout pr/1`` (replacing 1 with whatever the number is).
>> 
>> Honestly the worst part about someone sending a PR to the wrong branch is it
>> degrades into the same terrible UX that *every* patch has to go through on
>> a hg.python.org repository right now.
> 
> I'm not saying it's worse, but most of the time it's no better for the
> committer, especially since the branches have to be juggled in most cases.

Right, well if you’re just merging one branch into another you can create a PR
from one branch to another directly in the Github Web UI, which will run tests
and then you can press the merge button on that PR too. It’s something like 3-4
clicks or so. If you want to selectively send a change from one branch to
another that will require manually making the change locally yea.

> 
> Although, when it's the same amount of work for the committer, but nicer for
> the contributor, that's a net win, I can see that.
> 
> What is absolutely essential though is a way to automatically open an issue
> on bugs.python.org for each PR, otherwise we have to look for issues in two
> different places.  (Sure, GH treats PRs like GH issues, but we wouldn't use
> the GH issue tracker.)

There’s a web hook that fires when a pull request is created, it could even be
smart enough to look for Issue numbers in the PR and commits so that it 
associates
it with an existing issue instead of creating a new issue.

You can see all of the events we can subscribe to here: 
https://developer.github.com/webhooks/#events

In a related note, while it’s not possible to *reject* a push if things don’t
match up it is entirely possible to detect it after the fact and send an email
out to folks or what have you. Similar to how tests themselves are done now.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?

[Donald Stufft]

> On Nov 23, 2014, at 6:57 PM, Steven D'Aprano  wrote:
> 
> On Sun, Nov 23, 2014 at 08:55:50AM -0800, Guido van Rossum wrote:
> 
>> But I strongly believe that if we want to do the right thing for the 
>> long term, we should switch to GitHub.
> 
> Encouraging a software, or social, monopoly is never the right thing for 
> the long term.
> 
> http://nedbatchelder.com/blog/201405/github_monoculture.html

I don’t think this is really all that big of a deal. If we want to move
off of Github doing so is easy. There are lots of (not nearly as good as
but probably still better than what we have now) OSS software that gives
you a github like flow. The only *data* that is really in there is what’s
stored in the repository itself (since I don’t think for anything major
we’d ever put issues there or use the wiki) which is trivial to move around.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?

[Donald Stufft]

> On Nov 24, 2014, at 6:44 AM, Ben Finney  wrote:
> 
> Nick Coghlan  writes:
> 
>> Are you volunteering to write a competing PEP for a migration to git
>> and GitHub?
> 
> Anyone who does decide to propose either Git or GitHub for hosting
> Python resources: Please don't conflate the two.
> 
> Git is a community-supported free-software DVCS system with many viable
> hosting platforms and providers.
> 
> GitHub is one proprietary hosting service with features that encourage
> vendor lock-in, and as Barry Warsaw pointed out it is not answerable to
> the PSF nor under the PSF's control. There are other hosting solutions
> without these problems.
> 
> Everyone here already knows this distinction at some level, but it's far
> too common to see people assume that an argument in favour of Git will
> also be a compelling case for GitHub. It isn't, and the case for the
> latter needs to be made quite separately from the case for the former.
> 
> No, I'm not offering to write such a PEP either. I'm requesting that we
> recognise that a promotion of GitHub needs to account for its downsides
> too, and needs to promote its specific benefits separately from the
> benefits of Git.
> 

In many cases Github is git’s killer feature which is why you see a lot
of people equate the two. It’s not unusual to see a project switch away
from X to git entirely so they can use Github.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?

[Donald Stufft]

> On Nov 24, 2014, at 7:09 AM, Nick Coghlan  wrote:
> 
> On 24 November 2014 at 22:01, Donald Stufft  wrote:
>>> On Nov 24, 2014, at 6:44 AM, Ben Finney  wrote:
>>> No, I'm not offering to write such a PEP either. I'm requesting that we
>>> recognise that a promotion of GitHub needs to account for its downsides
>>> too, and needs to promote its specific benefits separately from the
>>> benefits of Git.
>>> 
>> 
>> In many cases Github is git’s killer feature which is why you see a lot
>> of people equate the two. It’s not unusual to see a project switch away
>> from X to git entirely so they can use Github.
> 
> And this is why the "you can still get your data out" argument doesn't
> make any sense - if you aren't planning to rely on the proprietary
> APIs, GitHub is just a fairly mundane git hosting service, not
> significantly different in capabilities from Gitorious, or RhodeCode,
> or BitBucket, or GitLab, etc. So you may as well go with one of the
> open source ones, and be *completely* free from vendor lockin.

I don’t really agree unless what you’re saying is that the reason it
doesn’t make any sense is because you probably won’t want to get your
data out because Github’s features are compelling enough that you
don’t want to lose them.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?

[Donald Stufft]

> On Nov 24, 2014, at 4:12 AM, Nick Coghlan  wrote:
> 
> 
> On 24 Nov 2014 10:41, "Donald Stufft"  <mailto:don...@stufft.io>> wrote:
> >
> >
> > > On Nov 23, 2014, at 6:57 PM, Steven D'Aprano  > > <mailto:st...@pearwood.info>> wrote:
> > >
> > > On Sun, Nov 23, 2014 at 08:55:50AM -0800, Guido van Rossum wrote:
> > >
> > >> But I strongly believe that if we want to do the right thing for the
> > >> long term, we should switch to GitHub.
> > >
> > > Encouraging a software, or social, monopoly is never the right thing for
> > > the long term.
> > >
> > > http://nedbatchelder.com/blog/201405/github_monoculture.html 
> > > <http://nedbatchelder.com/blog/201405/github_monoculture.html>
> >
> > I don’t think this is really all that big of a deal. If we want to move
> > off of Github doing so is easy.
> 
> It's only easy to leave if you never adopt any GitHub-only services like 
> Travis CI. It's the same lockin play as the one Google uses: make it easy for 
> people to move their data, so they're less likely to notice you're locking 
> them in with proprietary APIs and ecosystems instead.
> 
> You unlikely to see significant amounts of VC funding pouring into a company 
> unless the investors see a possible chance to extract monopoly rents at some 
> point in the future.
> 
> Cheers,
> Nick.
> 


Dropping those services is not harder than running those services on our own 
anyways. It’s unlikely, for instance, that CPython itself would ever be able to 
use Travis so if we did move to Github and at some point in the future want to 
move away from Github we’d simply be reverting to the state we’re in now. We 
wouldn’t be worse off other than we’d have experienced something better than 
what we have now. I suppose purposely hobbling ourselves so we never get used 
to something better is a legitimate way to manage things.

It’s not different than saying spam filtering is a “vendor lockin” on gmail 
because if you want to switch away from it in the future you’ll have to set up 
your own spam filtering.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?

[Donald Stufft]

nfrastructure perspective, what automation scripts do we have and how
>> difficult would it be to update them to use git instead of hg? This is
>> necessary simply to know where we would need to update URLs, let alone
>> change in DVCS.
> 
> The problems with changing version control systems don't really become
> significant until we start talking about switching CPython itself,
> rather than the support repos.
> 
>> Third, do our release managers care about hg vs. git strongly? They probably
>> use the DVCS the most directly and at a lower level by necessity compared to
>> anyone else.
>> 
>> Fourth, do any core developers feel strongly about not using GitHub? Now
>> please notice I said "GitHub" and not "git"; I think the proper way to frame
>> this whole discussion is we are deciding if we want to switch to Bitbucket
>> or GitHub who provide a low-level API for their version control storage
>> service through hg or git, respectively. I personally dislike git, but I
>> really like GitHub and I don't even notice git there since I use GitHub's OS
>> X app; as I said, I view this as choosing a platform and not the underlying
>> DVCS as I have happily chosen to access the GitHub hosting service through
>> an app that is not git (it's like accessing a web app through it's web page
>> or its REST API).
> 
> Yes, I object strongly to the use of GitHub when there are
> commercially supported services written in Python like BitBucket and
> RhodeCode available if we want to go the external hosting route, and
> other options like the RhodeCode derived Kallithea if we want to run a
> self-hosted forge. RhodeCode are even PSF sponsors - I'm sure they'd
> be willing to discuss the possibility of hosting core development
> repos on their service.
> 
> If I was doing a full risk management breakdown, then RhodeCode would
> be the obvious winner, as not only are they PSF sponsors, but
> reverting to self-hosting on Kallithea would remain available as an
> exit strategy.
> 
> I only suggested BitBucket in this thread because the PSF already has
> some repos set up there, so that seemed easier than establishing a new
> set of repos on a RhodeCode hosted instance.

The PSF account on Bitbucket has 3 repositories. One hasn’t been touched
in two years, the other in a year, and the third isn’t in use either.

Compare that to Github which has:

- The Chef Cookbooks
- The Salt states
- The Python.org website
- The Log Archiver for PyPI/Python.org
- The documentation building scripts

Which are all actively being used/developed.

Beyond that, I don’t have hard numbers (though I could probably attempt to
get them) but my gut instinct is that the Python community is primarily
there as well.

> 
>> At least for me, until we get a clear understanding of what workflow changes
>> we are asking for both contributors and core developers and exactly what
>> work would be necessary to update our infrastructure for either Bitbucket or
>> GitHub we can't really have a reasonable discussion that isn't going to be
>> full of guessing.
> 
> All repos that migrated away from hg.python.org would move to a PR
> based workflow, rather than manual patch management on the issue
> tracker. The migrated repos would likely also use their integrated
> issue tracker rather than the main CPython one at bugs.python.org.
> 
> Externally hosted repos would likely retain a regularly updated mirror
> on hg.python.org to ensure the source remains available even in the
> event of problems affecting the external hosting provider.
> 
>> And I'm still in support no matter what of breaking out the HOWTOs and the
>> tutorial into their own repos for easier updating (having to update the
>> Python porting HOWTO in three branches is a pain when it should be
>> consistent across Python releases).
> 
> Agreed.
> 
> Cheers,
> Nick.
> 
> -- 
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?

[Donald Stufft]

> On Nov 24, 2014, at 11:28 AM, Brett Cannon  wrote:
> 
> On Mon Nov 24 2014 at 2:25:30 AM Nick Coghlan  <mailto:ncogh...@gmail.com>> wrote:
>  
> That
> outcome would be the antithesis of the PSF's overall mission, 
> 
> This might be a little controversial, but the PSF's mission should not 
> influence a decision of python-dev. If we had to break a tie then yes, I 
> would choose the Python-based solution. But if a superior solution existed 
> and it just happened to not be written in Python I'm not going to sacrifice 
> productivity and the overall health of the project just to promote an 
> inferior tool because they happened to choose Python.
> 
> The only reason we didn't go with Jira for our issue tracker was because of 
> pressure to not go with a closed-source solution and I was promised 
> volunteers from the FSF to help manage our issue tracker (which never 
> materialized, BTW).

This is really what I’m trying to do but I’m apparently not getting my point 
across very well. I want us to pick the best tool for the job regardless of 
what language it’s written in. I just so happen to think that the best tool for 
the job in this case is Github.

> Yes, I object strongly to the use of GitHub when there are
> commercially supported services written in Python like BitBucket and
> RhodeCode available if we want to go the external hosting route, and
> other options like the RhodeCode derived Kallithea if we want to run a
> self-hosted forge. RhodeCode are even PSF sponsors - I'm sure they'd
> be willing to discuss the possibility of hosting core development
> repos on their service.
> 
> That's fine if they want to talk about free support, but being a PSF sponsor 
> should not play into it in the slightest, else it's like buying influence.

Agreed here too (even then, Github has been a PyCon sponsor for awhile and has 
even ran their own Python conference in the past). I think it’s kind of shitty 
to reject and demean someone who has supported the Python community through 
various means just because their primary language isn’t Python.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?

[Donald Stufft]

> On Nov 24, 2014, at 2:25 AM, Nick Coghlan  wrote:
> 
> Are you volunteering to write a competing PEP for a migration to git and 
> GitHub?

If nobody steps up to do this (and another PEP isn’t accepted before then) I can
likely write something up over the upcoming holiday.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?

[Donald Stufft]

> On Nov 24, 2014, at 2:55 PM, Nick Coghlan  wrote:
> 
> 
> On 25 Nov 2014 02:28, "Brett Cannon"  <mailto:br...@python.org>> wrote:
> >
> >
> >
> > On Mon Nov 24 2014 at 2:25:30 AM Nick Coghlan  > <mailto:ncogh...@gmail.com>> wrote:
> >>
> >> On 24 November 2014 at 02:55, Brett Cannon  >> <mailto:br...@python.org>> wrote:
> >> > On Sun Nov 23 2014 at 6:18:46 AM Nick Coghlan  >> > <mailto:ncogh...@gmail.com>> wrote:
> >> >> Those features are readily accessible without changing the underlying
> >> >> version control system (whether self-hosted through Kallithea or 
> >> >> externally
> >> >> hosted through BitBucket or RhodeCode). Thus the folks that want to 
> >> >> change
> >> >> the version control system need to make the case that doing so will 
> >> >> provide
> >> >> additional benefits that *can't* be obtained in a less disruptive way.
> >> >
> >> > I guess my question is who and what is going to be disrupted if we go 
> >> > with
> >> > Guido's suggestion of switching to GitHub for code hosting? Contributors
> >> > won't be disrupted at all since most people are more familiar with GitHub
> >> > vs. Bitbucket (how many times have we all heard the fact someone has even
> >> > learned Mercurial just to contribute to Python?). Core developers might 
> >> > be
> >> > based on some learned workflow, but I'm willing to bet we all know git at
> >> > this point (and for those of us who still don't like it, myself included,
> >> > there are GUI apps to paper over it or hg-git for those that prefer a 
> >> > CLI).
> >> > Our infrastructure will need to be updated, but how much of it is that
> >> > hg-specific short of the command to checkout out the repo? Obviously
> >> > Bitbucket is much more minor by simply updating just a URL, but changing 
> >> > `hg
> >> > clone` to `git clone` isn't crazy either. Georg, Antoine, or Benjamin can
> >> > point out if I'm wrong on this, maybe Donald or someone in the
> >> > infrastructure committee.
> >>
> >> Are you volunteering to write a competing PEP for a migration to git and 
> >> GitHub?
> >
> >
> > Been there, done that, got the PEP number. I'm just trying to speak from 
> > the perspective of the person who drove us off of svn and on to hg (as well 
> > as drove us off of SourceForge to our own workflow stack). I personally 
> > just want a better workflow. As I said at the beginning, I read your PEPs 
> > and talked to you about them at PyCon and I want something like that to 
> > happen; push button patch acceptance along with CI of patches would go a 
> > long way to making accepting patches easier. But as others have pointed 
> > out, we just don't have the volunteer time to make them happen ATM, so I'm 
> > willing to entertain moving to GitHub or Bitbucket or whatever to improve 
> > our situation.
> 
> It may not have been Guido's intention, but his proposal to undercut the 
> entire Python based version control tooling ecosystem by deeming it entirely 
> unfit for our purposes has caused me to dramatically re-evaluate my own 
> priorities.
> 
> 

I think this is a misrepresentation of what Guido said. I’m pretty sure he just 
said that Github has “won” over the other sites (which if you define winning in 
terms of who has the mindshare, I think is indisputable) and that he prefers 
git over hg now.
> I consider the status quo to be only mildly annoying, which is why I'd been 
> willing to defer proposing improvements. If folks are seriously contemplating 
> proposing a move to GitHub instead, then that changes the equation 
> significantly.
> 
> Regards,
> Nick.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?

[Donald Stufft]

> On Nov 24, 2014, at 3:48 PM, Nick Coghlan  wrote:
> 
> 
> On 25 Nov 2014 06:25, "Donald Stufft"  <mailto:don...@stufft.io>> wrote:
> >
> >
> >> On Nov 24, 2014, at 2:55 PM, Nick Coghlan  >> <mailto:ncogh...@gmail.com>> wrote:
> >>
> >> It may not have been Guido's intention, but his proposal to undercut the 
> >> entire Python based version control tooling ecosystem by deeming it 
> >> entirely unfit for our purposes has caused me to dramatically re-evaluate 
> >> my own priorities.
> >>
> >>
> >
> > I think this is a misrepresentation of what Guido said. I’m pretty sure he 
> > just said that Github has “won” over the other sites (which if you define 
> > winning in terms of who has the mindshare, I think is indisputable) and 
> > that he prefers git over hg now.
> 
> That's how a monopoly play works - you invest heavily in rapid growth, aim to 
> lock in an ecosystem through proprietary APIs, then use that powerful market 
> position to deliver monopoly rents to the initial investors.
> 
> The pattern repeats because "free-as-in-beer" is an extraordinarily effective 
> marketing tactic.
> 
> 

I don’t really think this is a fair comparison either.

“lock in” here doesn’t make any sense to me unless you define lock-in as any 
compelling feature what so ever. Wherever a feature Github added was able to be 
integrated with git itself it did so. PRs for instance could have been done 
using a proprietary CLI client that uploaded patches which couldn’t be easily 
exported. Can you please point out in which ways Github has tried to do “vendor 
lock in” and in addition can you also point out in which ways they don’t apply 
to Bitbucket which you were previously fine with?

As Guido pointed out, the social aspect to a DVCS is an important part of a 
DCVS and Git (and Github) has that. Others (including yourself) have pointed 
out that git vs hg itself is largely isomorphic as far as actual technical 
ability go when you go through and spend the time to figure out what extensions 
you need and configure them. It’s largely the *other* tooling around it and the 
relative sizes of the communities.

If you want to pick a toolchain that isn’t as popular nor as polished or as 
well put together for idealogical reasons than that’s fine, but at least be 
honest about the fact that you’re choosing to prioritize ideology over 
selecting the best toolchain.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?

[Donald Stufft]

> On Nov 24, 2014, at 8:59 PM, Ethan Furman  wrote:
> 
> On 11/24/2014 08:36 AM, Donald Stufft wrote:
>> On Nov 24, 2014, at 11:28 AM, Brett Cannon wrote:
>>> 
>>> This might be a little controversial, but the PSF's mission should not 
>>> influence a decision of python-dev.
> 
> Yet what we do can reinforce, or undermine, the PSF.
> 
> 
>>> The only reason we didn't go with Jira for our issue tracker was because of 
>>> pressure to not go with a closed-source
>>> solution [...]
> 
>> This is really what I’m trying to do but I’m apparently not getting my point 
>> across very well. I want us to pick the
>> best tool for the job regardless of what language it’s written in.
> 
> As an open source project it behooves us to support open source solutions, 
> even if a "better" closed-source solution exists.
> 
> It is sounding to me like GitHub is not, itself, an open solution, even 
> though they may support open source.

I’d agree if the tooling was comparable, but at the end of the day the closed 
source tool is better and more popular. It isn’t Python’s job to fall on the 
sword in the name of some greater ideology while other languages get to pick 
the tooling that best enables them to serve the faith that their users have put 
in them. “Practicality beats Purity” after all.

You might lament the fact that the closed source tool is the better option, but 
the right response to that is to make an OSS alternative that is more, or at 
least as, compelling as the closed source solution and then market that and 
win. 

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?

[Donald Stufft]

> On Nov 24, 2014, at 9:37 PM, Ethan Furman  wrote:
> 
> On 11/24/2014 06:27 PM, Donald Stufft wrote:
>> On Nov 24, 2014, at 8:59 PM, Ethan Furman wrote:
>>> 
>>> It is sounding to me like GitHub is not, itself, an open solution, even 
>>> though
>>> they may support open source.
>> 
>> I’d agree if the tooling was comparable, but at the end of the day the closed
>> source tool is better and more popular. It isn’t Python’s job to fall on the
>> sword in the name of some greater ideology while other languages get to pick
>> the tooling that best enables them to serve the faith that their users have 
>> put
>> in them. “Practicality beats Purity” after all.
>> 
>> You might lament the fact that the closed source tool is the better option, 
>> but
>> the right response to that is to make an OSS alternative that is more, or at
>> least as, compelling as the closed source solution and then market that and 
>> win.
> 
> Or, make a list of the must-haves from the (for whatever reason) 
> controversial choice, and implement them in our own
> infrastructure.
> 
> (Yeah, I guess I'm volunteering to help with that effort. ;)

Isn’t that essentially what I said? ;) Unless you were planning to make the 
implementations on our own infrastructure closed source. Sadly it’s not just a 
feature matrix that you need to fill out some check boxes, the closed source 
software in question just flat out *works* better than any of it’s open source 
counter parts. Tacking on some features onto an existing solution does not 
compare.

There’s also the social aspects of it as well which is a big concern too IMO. 
If you want to attract new contributors, not just keep the ones you already 
have sometimes that means going to where the new contributors are instead of 
telling them that they need to come to you.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github

[Donald Stufft]

rojects.

They claim to have seen a great benefit from this move, where it enables casual
contributors to easily move between different projects within their
sub-community without having to learn a special, bespoke workflow and a
different toolchain for each project. They've found that when people can use
their limited time on actually contributing instead of learning the different
tools and workflows that not only do they contribute more to one project, that
they also expand out and contribute to other projects. This move is also
attributed to making it commonplace for members of that community to go so far
as publishing their research and educational materials on Github as well.

This showcases the real power behind moving to a highly popular toolchain and
workflow, as each variance introduces yet another hurdle for new and casual
contributors to get past and it makes the time spent learning that workflow
less reusable with other projects.


Migration
=

Through the use of hg-git [#hg-git]_ we can easily convert a Mercurial
repository to a Git repository by simply pushing the Mercurial repository to
the Git repository. People who wish to continue to use Mercurual locally can
then use hg-git going into the future using the new Github URL, however they
will need to re-clone their repositories as using Git as the server seems to
trigger a one time change of the changeset ids.

As none of the selected repositories have any tags, branches, or bookmarks
other than the ``default`` branch the migration will simply map the ``default``
branch in Mercurial to the ``master`` branch in git.

In addition since none of the selected projects have any great need of a
complex bug tracker, they will also migrate their issue handling to using the
GitHub issues.

In addition to the migration of the repository hosting itself there are a
number of locations for each particular repository which will require updating.
The bulk of these will simply be changing commands from the hg equivilant to
the git equivilant.

In particular this will include:

* Updating www.python.org to generate PEPs using a git clone and link to
  Github.
* Updating docs.python.org to pull from Github instead of hg.python.org for the
  devguide.
* Enabling the ability to send an email to python-check...@python.org for each
  push.
* Enabling the ability to send an IRC message to #python-dev on Freenode for
  each push.
* Migrate any issues for these projects to their respective bug tracker on
  Github.

This will restore these repositories to similar functionality as they currently
have. In addition to this the migration will also include enabling testing for
each pull request using Travis CI [#travisci]_ where possible to ensure that
a new PR does not break the ability to render the documentation or PEPs.


User Access
===

Moving to Github would involve adding an additional user account that will need
to be managed, however it also offers finer grained control, allowing the
ability to grant someone access to only one particular repository instead of
the coarser grained ACLs available on hg.python.org.


References
==

.. [#openhub-stats] `Open Hub Statistics 
<https://www.openhub.net/repositories/compare>`
.. [#hg-git] `hg-git <https://hg-git.github.io/>`
.. [#travisci] `Travis CI <https://travis-ci.org/>`

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github

[Donald Stufft]

> On Nov 29, 2014, at 6:27 PM, Donald Stufft  wrote:
> 
> [lots of words]

Just FYI, I’ve pushed an update to the PEP. Nothing major just some grammatical
fixes and such. The revision is here: 
https://hg.python.org/peps/rev/6c6947dbd13f

For whatever it's worth, the person who submitted that patch used Github's
online editor to submit them and made a PR to my personal PEP repository where
I work on my PEPs (https://github.com/dstufft/peps/pull/3). If someone wanted
to see some of the features in action that is a nice PEP to look at. In
particular if you hit "Files Changed" then beside the view button is an icon
that looks like a piece of paper and when you over over it it'll say "Display
the Rich Diff". Clicking on that you'll see the diff of the rendered output
which lets you ignore things which were just reflowing the paragraphs and
such.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github

[Donald Stufft]

> On Nov 29, 2014, at 7:15 PM, Alex Gaynor  wrote:
> 
> Donald Stufft  stufft.io> writes:
> 
>> 
>> [words words words]
>> 
> 
> I strongly support this PEP. I'd like to share two pieces of information. Both
> of these are personal anecdotes:
> 
> For the past several years, I've been a contributor on two major projects 
> using
> mercurial, CPython and PyPy. PyPy has a strong culture of in-repo branching,
> basically all contributors regularly make branches in the main repo for their
> work, and we're very free in giving people commit rights, so almost everyone
> working on PyPy in any way has this level of access. This workflow works ok. I
> don't love it as much as git, but it's fine, it's not an impediment to my 
> work.
> 
> By contrast, CPython does not have this type of workflow, there are almost no
> in-tree branches besides the 2.7, 3.4, etc. ones. Despite being a regular hg
> user for years, I have no idea how to create a local-only branch, or a branch
> which is pushed to a remote (to use the git term). I also don't generally
> commit my own work to CPython, even though I have push privledges, 
> because I
> prefer to *always* get code review on my work. As a result, I use a git mirror
> of CPython to do all my work, and generate patches from that.
> 
> The conclusion I draw from this is that hg's workflow is probably fine if
> you're a committer on the project, or don't ever need to maintain multiple
> patches concurrently (and thus can just leave everything uncommitted in the
> repo). However, the hg workflow seems extremely defficient at non-committer
> contributors.

I also don’t know how to do this. When I’m doing multiple things for CPython
my “branching” strategy is essentially using hg diff to create a patch file
with my “branch” name (``hg diff > my-branch.patch``), then revert all of my
changes (``hg revert —all —no-backup``), then either work on a new “branch”
or switch to an old “branch” by applying the corresponding patch
(``patch -p1 < other-branch.patch``).

> 
> The seconds experience I have is that of Django's migration to git and github.
> For a long time we were on SVN, and we were very resistant to moving to 
> DVCS in
> general, and github in particular. Multiple times I said that I didn't see how
> exporting a patch and uploading it to trac was more difficult than sending a
> pull request. That was very wrong on my part.
> 
> My primary observation is not about new contributors though, it's actually
> about the behavior of core developers. Before we were on github, it was fairly
> rare for core developers to ask for reviews for anything besides *gigantic*
> patches, we'd mostly just commit stuff to trunk. Since the switch to github,
> I've seen that core developers are *far* more likely to ask for reviews of
> their work before merging.

I’ve also seen this effect, not just in Django but that I also notice my own
behavior. Projects where I have commit access but which aren’t on Github I
find myself less likely to look for others to review them and find myself
just committing directly to master/default/trunk while I do tend to look for
reviews and create PRs to give others the chance to review on Github.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github

[Donald Stufft]

> On Nov 29, 2014, at 7:43 PM, Antoine Pitrou  wrote:
> 
> On Sun, 30 Nov 2014 00:15:55 + (UTC)
> Alex Gaynor  wrote:
>> 
>> The seconds experience I have is that of Django's migration to git and 
>> github.
>> For a long time we were on SVN, and we were very resistant to moving to 
>> DVCS in
>> general, and github in particular. Multiple times I said that I didn't see 
>> how
>> exporting a patch and uploading it to trac was more difficult than sending a
>> pull request. That was very wrong on my part.
>> 
>> My primary observation is not about new contributors though, it's actually
>> about the behavior of core developers. Before we were on github, it was 
>> fairly
>> rare for core developers to ask for reviews for anything besides *gigantic*
>> patches, we'd mostly just commit stuff to trunk. Since the switch to github,
>> I've seen that core developers are *far* more likely to ask for reviews of
>> their work before merging.
> 
> I don't know anything about Django's old SVN setup, but our code review
> tool (Rietveld) is actually quite good at that, and IMHO slightly
> better than github, since it will only send an e-mail at the final
> submission - by contrast, each individual comment you leave on github
> fires a separate notification, which can feel like spam.
> 
> Our main problem for reviews these days is the lack of core developer
> time. For example, Serhiy often asks for reviews (he has many patches
> pending), which I personally don't have a lot of time to provide.
> 

I think one of the issues with Reitveld isn’t related to Reitveld itself at
all, it’s all the *other* stuff you have to do to get a patch into Reitveld to
allow someone to review it at all. Generating a patch and uploading it to
Roundup is a pain and it’s far easier to just commit things directly to
default.

As far as Reitveld vs Github itself goes for reviews, I don’t personally agree.
Sending a single notification vs a notification per comment is going to be a
very subjective point of view, I like more notifications for the simple fact
that if I'm actively working on a PR while someone is reviewing it I can fix
the issues as quickly as they are finding them.

However if you move past that one thing, the Github PR review has a number of
benefits over Reitveld itself. Since the repositories this PEP currently deals
with are largely documentation-esque repositories the "Prose" diff is
incredibly useful for viewing a content-based diff instead of a diff designed
more for software where you don't normally reflow lines as much. In addition
the Github diff viewer also makes it trivial to expand the context of the diff
so you can see more of the surrounding file, even allowing you to expand it
to the entire file. This is super useful when the automatic amount of context
can't really give you enough information for reviewing the change or not.

Another difference is in how the review comments are presented. With Reitveld
the inline comments go away for each patchset you upload, regardless of if
you've addressed the concerns in that comment or not. They do get bubbled up
into the overall "Messages" view, however this has the opposite problem where
it gives you all of the messages regardless of if they are still a problem
with the latest code or not. In contrast the Github PR will hide old comments
when they lines they addressed have changed, but still make it easy to see the
old comments and the context in which they were made.

There's also the UI itself, it's somewhat dated and I think it's not entirely
unobvious that it suffers from something a lot of OSS projects suffer from in
that it is a "Developer" made UI. I don't think it's a big secret that
developers tend to make UIs that are not as nice as proper UX folks make, but
OSS has historically had a problem attracting that kind of talent (and is
often openly hostile towards it).

Finally there's the transferability of knowledge at play too. I likely would
not review someone else's patch unless I felt strongly about it on Reitveld
largely because I don't know Reitveld very well and I'd rather spend the time
that I'd need to learn to use it effectively working on other projects where
I already know the entire toolchain well. This is one of the larger points in
this PEP that the benefits to using popular tooling is that not only does
people's existing knowledge transfer easily into your project, but the time
they spend learning your tooling also transfers easily outside of this project.
This makes it much more attractive to learn the tooling since the hypothetical
person would be able to take that knowledge and apply it elsewhere.

It is my expe

Re: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github

[Donald Stufft]

> On Nov 29, 2014, at 8:12 PM, Nick Coghlan  wrote:
> 
> 
> On 30 Nov 2014 09:28, "Donald Stufft"  <mailto:don...@stufft.io>> wrote:
> >
> > As promised in the "Move selected documentation repos to PSF BitBucket
> > account?" thread I've written up a PEP for moving selected repositories from
> > hg.python.org <http://hg.python.org/> to Github.
> >
> > You can see this PEP online at: https://www.python.org/dev/peps/pep-0481/ 
> > <https://www.python.org/dev/peps/pep-0481/>
> >
> > I've also reproduced the PEP below for inline discussion.
> 
> Given that hg.python.org <http://hg.python.org/> isn't going anywhere, you 
> could also use hg-git to maintain read-only mirrors at the existing URLs and 
> minimise any breakage (as well as ensuring a full historical copy remains 
> available on PSF infrastructure). Then the only change needed would be to set 
> up appropriate GitHub web hooks to replace anything previously based on a 
> commit hook rather than periodic polling.
> 
> 

Ah yes, I meant to include that and just forgot to do it when I went to test
hg-git to see how well it worked and whether I got different commit hashes on
different machines. I also thought about adding a git.python.org 
<http://git.python.org/> which just
acted as a read-only mirror of what was on Github, but I don’t know if that’s
actually generally useful or not.

> The PEP should also cover providing clear instructions for setting up 
> git-remote-hg with the remaining Mercurial repos (most notably CPython), as 
> well as documenting a supported workflow for generating patches based on the 
> existing CPython GitHub mirror.
> 
> 

I can add this. I’ve never actually tried using git-remote-hg with CPython
itself because I’ve made it segfault on other Mercurial repositories and I
never figured out why so I just generally fight my way through using Mercurial
on projects that themselves use Mercurial. I will absolutely test to see if
git-remote-hg works with CPython and I can document using that to contribute to
CPython. I’m not sure it needs to be part of the PEP or not? Feels like
something that would be better inside the devguide itself but I’m not opposed
to putting it both locations.

> Beyond that, GitHub is indeed the most expedient option. My two main reasons 
> for objecting to taking the expedient path are:
> 

It's not entirely about expedience. I think a lot of the reason why we should
look towards outsourcing some of these items is that volunteer time is not
a fungible resource. Volunteers are generally only willing to work on things
which they personally care about. This is entirely unlike a business where you
have employees who will generally work on whatever you tell them to because
that's what you're paying them for. To this end I personally don't really have
an interest in trying to create a better code hosting platform than Github when
Github is doing an amazing job in my opinion and they satisify my needs fine.
Given the *current* state of tooling it appears that there are not a lot of
people who both care about making that piece of software exist and are capable
of competing with Github in terms of quality.

> 1. I strongly believe that the long term sustainability of the overall open 
> source community requires the availability and use of open source 
> infrastructure. While I admire the ingenuity of the "free-as-in-beer" model 
> for proprietary software companies fending off open source competition, I 
> still know a proprietary platform play when I see one (and so do venture 
> capitalists looking to extract monopoly rents from the industry in the 
> future). (So yes, I regret relenting on this principle in previously 
> suggesting the interim use of another proprietary hosted service)
> 
> 

I somewhat agree. However I’m less concerned specifically about where projects
are hosted exactly and more about the *ability* to move to a completely OSS
infrastructure. In particular if at somepoint we need to move off of Github we
can totally do that, it’s not particularly difficult. Currently you lose the
higher quality polish of Github if you do that however if at some point in the
future Github either turns evil or an OSS software offers a truly compelling
alternative to Github then there is really nothing stopping a migration to
another platform. As I said in the PEP I view this as a “let’s cross that
bridge if/when we get to it”. The main thing we should look at is things that
would be difficult to migrate away from. For code hosting in particular most of
the truly valuable data is stored within the DVCS so migrating the bulk of the
data is as simple as pushing the repository to a new location. The other data
is within the issues, for these repositories I sug

Re: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github

[Donald Stufft]

> On Nov 29, 2014, at 9:01 PM, Donald Stufft  wrote:
> 
>> 
>> The PEP should also cover providing clear instructions for setting up 
>> git-remote-hg with the remaining Mercurial repos (most notably CPython), as 
>> well as documenting a supported workflow for generating patches based on the 
>> existing CPython GitHub mirror.
>> 
>> 
> 
> I can add this. I’ve never actually tried using git-remote-hg with CPython
> itself because I’ve made it segfault on other Mercurial repositories and I
> never figured out why so I just generally fight my way through using Mercurial
> on projects that themselves use Mercurial. I will absolutely test to see if
> git-remote-hg works with CPython and I can document using that to contribute 
> to
> CPython. I’m not sure it needs to be part of the PEP or not? Feels like
> something that would be better inside the devguide itself but I’m not opposed
> to putting it both locations.


Nevermind. I’m not going to add this because git-remote-hg is busted with the 
latest
version of Mercurial and I don’t think it’s really needed for this PEP (though 
would be
good for the devguide at some point).

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github

[Donald Stufft]

> On Nov 30, 2014, at 12:06 AM, Ben Finney  wrote:
> 
> Nick Coghlan  writes:
> 
>> 1. I strongly believe that the long term sustainability of the overall
>> open source community requires the availability and use of open source
>> infrastructure.
> 
> I concur. This article http://mako.cc/writing/hill-free_tools.html>
> makes the arguments well, IMO.
> 
>> 2. I also feel that this proposal is far too cavalier in not even
>> discussing the possibility of helping out the Mercurial team […] we'd
>> prefer to switch to something else entirely rather than organising a
>> sprint with them at PyCon to help ensure that our existing Mercurial
>> based infrastructure is approachable for git & GitHub users?
> 
> Exactly. For such a core tool, instead of pushing proprietary platforms
> at the expense of software freedom, the sensible strategy for a project
> (Python) that hopes to be around in the long term is to use and improve
> the free software platforms.

I think there is a big difference here between using a closed source VCS
or compiler and using a closed source code host. Namely in that the
protocol is defined by git so switching from one host to another is easy.

It’s akin to saying that if we chose to run the PyPI services on a Windows
machine that it is somehow makes it less-free even though we could
have chosen to run it on a “free” OS and we weren’t doing much, if anything,
to tie us to that particular OS.

If it makes people feel better we can continue to support the existing
mechanisms of contribution, then people can choose between interacting
with a “non free” host and “free” tooling. I suspect most people will choose
the “non-free” tooling.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github

[Donald Stufft]

> On Nov 30, 2014, at 7:31 AM, Paul Moore  wrote:
> 
> On 29 November 2014 at 23:27, Donald Stufft  wrote:
>> In previous years there was concern about how well supported git was on 
>> Windows
>> in comparison to Mercurial. However git has grown to support Windows as a 
>> first
>> class citizen. In addition to that, for Windows users who are not well 
>> aquanted
>> with the Windows command line there are GUI options as well.
> 
> I have little opinion on the PEP as a whole, but is the above
> statement true? From the git website, version 2.2.0 is current, and
> yet the downloadable Windows version is still 1.9.4. That's a fairly
> significant version lag for a "first class citizen".
> 
> I like git, and it has a number of windows-specific extensions that
> are really useful (more than Mercurial, AFAIK), but I wouldn't say
> that the core product supported Windows on an equal footing to Linux.
> 
> Paul

I think so yes. I may be wrong, however while 1.9.4 may be the latest
downloadable version of git for Windows, there is no downloadable
version of the Linux clients at all, they just tell you to go use
your package manager which for instance is version 1.7 on Debian. On
OS X the latest version is 2.0.1.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github

[Donald Stufft]

> On Nov 30, 2014, at 2:08 AM, Larry Hastings  wrote:
> 
> 
> On 11/29/2014 04:37 PM, Donald Stufft wrote:
>> On Nov 29, 2014, at 7:15 PM, Alex Gaynor  
>> <mailto:alex.gay...@gmail.com> wrote:
>>> Despite being a regular hg
>>> user for years, I have no idea how to create a local-only branch, or a 
>>> branch
>>> which is pushed to a remote (to use the git term).
>> I also don’t know how to do this.
> 
> Instead of collectively scratching your heads, could one of you guys do the 
> research and figure out whether or not hg supports this workflow?  One of the 
> following two things must be true:
> hg supports this workflow (or a reasonable fascimile), which may lessen the 
> need for this PEP.
> hg doesn't support this workflow, which may strengthen the need for this PEP.
> Saying "I've been using hg for years and I don't know whether it supports 
> this IMPORTANT THING" is not a particularly compelling argument.
> 

Comments like this make me feel like I didn’t explain myself very well in the
PEP.

While I do think that supporting this workflow via an extension is worse than
supporting it in core, this isn’t why this PEP exists. The current workflow for
contributing is painful, for the repositories this is talking about if I’m a
non-comitter I have to email patches to a particular closed mailing list and
then sit around and wait.

The Pull Request based workflow is *massively* better than uploading/emailing
patches around. So the question then becomes, if we're going to move to a PR
based workflow how do we do it? PEP 474 says that we should install some
software that works with Mercurial and supports Pull Requests. Another thread
suggested that we should just use to bitbucket whicih also supports Mercurial
and use that.

This PEP says that git and Github have the popular vote, which is extremely
compelling as a feature because:

* Popularity is the one thing you can't replicate featurewise. This feature or
  that you can find a way to work something like it out.
* With popularity comes the fact that people will already know the tooling
  involved and how to use it. This means that a new contributor whom already
  knows git will spend less time learning our specific toolset and more time
  being able to meaningfully contribute.
* With popularity also comes transferability, If I don't currently know a VCS
  tool and I learn git (and github) for the sake of these repositories then
  that knowledge will directly transfer to greater than 60% of the top 100
  projects on PyPI.
* With popularity also comes an increased amount of people writing about it,
  asking questions, etc. This means that if I have a problem while using this
  tool I am more likely to find someone who already had this problem and they
  are more likely to have found someone to help them solve it than I am with
  a less popular tool.

I'm not sure why people seem to try and focus on the fact that with this
extension or that you can make Mercurial replicate git better when that isn't
really the major point of the PEP at all. If Mercurial and git were neck in
neck in terms of mindshare and bitbucket and Github were as well then this
PEP probably wouldn't exist because on a techincal level I think the two tools
are probably close enough. However when most of the world is using one tool
and you're using another, that does represent a barrier to entry.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github

[Donald Stufft]

> On Nov 30, 2014, at 6:18 AM, Ben Finney  wrote:
> 
> Donald Stufft  writes:
> 
>> I think there is a big difference here between using a closed source
>> VCS or compiler and using a closed source code host. Namely in that
>> the protocol is defined by git so switching from one host to another
>> is easy.
> 
> GitHub deliberately encourages proprietary features that create valuable
> data that cannot be exported — the proprietary GitHub-specific pull
> requests being a prime example.

Once a Pull Request is merged there is little benefit to the PR itself,
however if you want the diff you can create a .patch file by appending
.diff or .patch to the end of the PR URL.

> 
> So it is only true to say one can export the data to a different host if
> one entirely abandons all those features which create GitHub-proprietary
> data.

That’s not true at all. There is an API that allows you to access *all*
of the data that is stored there. Allowing you to replicate it to whatever
system you wish. 

> 
> If you want to re-write the PEP to be clear you are only talking about
> the repository hosting of GitHub, and *not* any of its extra features
> that make it so attractive to use their proprietary service, I'd be
> interested to see that.
> 
> On the other hand, if you want to promote those proprietary features as
> part of the PEP, then it is disingenuous to claim the data can be easily
> exported to a different host.
> 
> -- 
> \“Do not enter the lift backwards, and only when lit up.” |
>  `\—elevator, Leipzig |
> _o__)  |
> Ben Finney
> 
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/donald%40stufft.io

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github

[Donald Stufft]

> On Nov 30, 2014, at 11:28 AM, Brett Cannon  wrote:
> 
> 
> 
> On Sat Nov 29 2014 at 7:16:34 PM Alex Gaynor  <mailto:alex.gay...@gmail.com>> wrote:
> Donald Stufft  stufft.io <http://stufft.io/>> writes:
> 
> >
> > [words words words]
> >
> 
> I strongly support this PEP. I'd like to share two pieces of information. Both
> of these are personal anecdotes:
> 
> For the past several years, I've been a contributor on two major projects 
> using
> mercurial, CPython and PyPy. PyPy has a strong culture of in-repo branching,
> basically all contributors regularly make branches in the main repo for their
> work, and we're very free in giving people commit rights, so almost everyone
> working on PyPy in any way has this level of access. This workflow works ok. I
> don't love it as much as git, but it's fine, it's not an impediment to my 
> work.
> 
> By contrast, CPython does not have this type of workflow, there are almost no
> in-tree branches besides the 2.7, 3.4, etc. ones. Despite being a regular hg
> user for years, I have no idea how to create a local-only branch, or a branch
> which is pushed to a remote (to use the git term). I also don't generally
> commit my own work to CPython, even though I have push privledges,
> because I
> prefer to *always* get code review on my work. As a result, I use a git mirror
> of CPython to do all my work, and generate patches from that.
> 
> The conclusion I draw from this is that hg's workflow is probably fine if
> you're a committer on the project, or don't ever need to maintain multiple
> patches concurrently (and thus can just leave everything uncommitted in the
> repo). However, the hg workflow seems extremely defficient at non-committer
> contributors.
> 
> One way to come close to that using hg is to have your own clone that you 
> never push to hg.python.org/cpython <http://hg.python.org/cpython> (e.g. 
> cloning the Bitbucket clone of cpython or hosting on hg.python.org 
> <http://hg.python.org/> a personal clone). You can then specify the repo and 
> branch on the issue tracker to generate your patch: 
> https://docs.python.org/devguide/triaging.html#mercurial-repository 
> <https://docs.python.org/devguide/triaging.html#mercurial-repository> . After 
> that it's just like any other patch workflow for core devs. It's not quite as 
> nice as maybe using named branches where you can just do a final hg 
> merge/commit to get your changes committed, but if you're not going to commit 
> your branches then you might as well get the automatic patch generation perk 
> in the issue tracker rather than using git (unless there is some other git 
> feature you use that you can't get in hg).

This doesn’t really work in a Pull Request workflow though I think is the point.
Uploading patches is it’s own kind of terrible but yes if you’re just uploading
patches you can probably do that. I don’t make branches in Mercurial because
i’m afraid I’m going to push a permanent branch to hg.python.org 
<http://hg.python.org/> and screw
something up.

>  
> 
> The seconds experience I have is that of Django's migration to git and github.
> For a long time we were on SVN, and we were very resistant to moving to
> DVCS in
> general, and github in particular. Multiple times I said that I didn't see how
> exporting a patch and uploading it to trac was more difficult than sending a
> pull request. That was very wrong on my part.
> 
> My primary observation is not about new contributors though, it's actually
> about the behavior of core developers. Before we were on github, it was fairly
> rare for core developers to ask for reviews for anything besides *gigantic*
> patches, we'd mostly just commit stuff to trunk. Since the switch to github,
> I've seen that core developers are *far* more likely to ask for reviews of
> their work before merging.
>  
> Why specifically? Did you have a web UI for reviewing patches previously? Do 
> you have CI set up for patches now and didn't before? What features did you 
> specifically gain from the switch to GitHub that you didn't have before? IOW 
> was it the "magic" of GitHub or some technical solution that you got as part 
> of the GitHub package and thus could theoretically be replicated on 
> python.org <http://python.org/>?

In most of the cases there was some form of a web UI for reviewing patches. I
don’t believe there was a CI setup for patches previously and most projects do
end up with some sort of CI on Github since Travis makes it particularly easy.
Honestly though, I feel like the main reason for it is just that Github’s
solution is easy to use, it works w

Re: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github

[Donald Stufft]

-specific fix -- then I 
> could easily push a "Commit" button and be done with it (although this 
> assumes single branch committing; doing this across branches makes all of 
> this difficult unless we finally resolve our Misc/NEWS conflict issues so 
> that in some instances it can be automated). Instead I have to wait until I 
> have a clone I can push from, download a patch, apply it, run the unit tests 
> myself, do the commit, and then repeat a subset of that to whatever branches 
> make sense. It's a lot of work for which some things could be automated.

Well there’s two sides to the contribution process.

There’s making things better/easier for people who *aren’t* committers and
there is making things better/easier for people who *are* committers. Tacking
extra things on to what we already have to improve the life of committers is
easier in many ways. As committers they’ve likely already taken the time to
learn the bespoke workflow that the Python project uses and have already gotten
through that particular hurdle. Looking to standardize around popular tools is
mostly about making it easier for *new* people and making it so that if they
learn this set of tools they can go an immediately apply that to most of the
other Python projects out there, or that if they are already contributing to
those other Python projects they are probably aware of this particular
toolchain and workflow and can apply that knowledge directly to the Python
project.

Moving to some of these tools happens to come with it features like really nice
CI integration and a nice "Merge" button that also make it a lot nicer for the
committer side of things.

I think it's also hard to get a representation of the people for whom the
bespoke workflow and less popular tooling are a problem for in a discussion
on python-dev. My guess is most of those people would not have signed up for
python-dev since, unless they were willing to take the time to learn that,
so there is an amount of selection bias at play here as well.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github

[Donald Stufft]

> On Nov 30, 2014, at 11:55 AM, Barry Warsaw  wrote:
> 
> On Nov 30, 2014, at 09:54 AM, Ian Cordasco wrote:
> 
>> - Migrating "data" from GitHub is easy. There are free-as-in-freedom
>> tools to do it and the only cost is the time it would take to monitor
>> the process
> 
> *Extracting* data may be easy, but migrating it is a different story.  As the
> Mailman project has seen in trying to migrate from Confluence to Moin, there
> is a ton of very difficult work involved after extracting the data.  Parsing
> the data, ensuring that you have all the bits you need, fitting it into the
> new system's schema, working out the edge cases, adapting to semantic
> differences and gaps, ensuring that all the old links are redirected, and so
> on, were all exceedingly difficult[*].
> 
> Even converting between two FLOSS tools is an amazing amount of work.  Look at
> what Eric Raymond did with reposurgeon to convert from Bazaar to git.

I fail to see how this is a reasonable argument to make at all since, as you
mentioned, converting between two FLOSS tools can be an amazing amount of work.
Realistically the amount of work is going to be predicated on whether or not
there is a tool that already handles the conversion for you. Assuming of course
that the data is available to you at all.

As a particularly relevant example, switching from Mercurial to Git is as easy
as installing hg-git, creating a bookmark for master that tracks default, and
then pushing to a git repository.

> 
> It's a good thing that your data isn't locked behind a proprietary door, for
> now.  That's only part of the story.  But also, because github is a closed
> system, there's no guarantee that today's data-freeing APIs will still exist,
> continue to be functional for practical purposes, remain complete, or stay at
> parity with new features.

This feels like Chicken Little-ing. If Github closed it’s APIs then you could
still get at that data by scraping the web interface. However why would Github
do that? That would piss off the vast majority of OSS projects who are currently
hosted there and is likely to cause a pretty big migration off of Github for
fear that Github is going to attempt to lock people onto Github. The popularity
of Github *is* Github’s killer feature and doing something that is going to
send the vast bulk of your users running for the hills sounds like something 
that
they would have to be particularly stupid to do.

> 
> Cheers,
> -Barry
> 
> [*] And our huge gratitude goes to Paul Boddie for his amazing amount of work
> on the project.
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/donald%40stufft.io

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github

[Donald Stufft]

> On Nov 30, 2014, at 12:09 PM, Donald Stufft  wrote:
> 
>> 
>> On Nov 30, 2014, at 11:55 AM, Barry Warsaw  wrote:
>> 
>> On Nov 30, 2014, at 09:54 AM, Ian Cordasco wrote:
>> 
>>> - Migrating "data" from GitHub is easy. There are free-as-in-freedom
>>> tools to do it and the only cost is the time it would take to monitor
>>> the process
>> 
>> *Extracting* data may be easy, but migrating it is a different story.  As the
>> Mailman project has seen in trying to migrate from Confluence to Moin, there
>> is a ton of very difficult work involved after extracting the data.  Parsing
>> the data, ensuring that you have all the bits you need, fitting it into the
>> new system's schema, working out the edge cases, adapting to semantic
>> differences and gaps, ensuring that all the old links are redirected, and so
>> on, were all exceedingly difficult[*].
>> 
>> Even converting between two FLOSS tools is an amazing amount of work.  Look 
>> at
>> what Eric Raymond did with reposurgeon to convert from Bazaar to git.
> 
> I fail to see how this is a reasonable argument to make at all since, as you
> mentioned, converting between two FLOSS tools can be an amazing amount of 
> work.
> Realistically the amount of work is going to be predicated on whether or not
> there is a tool that already handles the conversion for you. Assuming of 
> course
> that the data is available to you at all.
> 
> As a particularly relevant example, switching from Mercurial to Git is as easy
> as installing hg-git, creating a bookmark for master that tracks default, and
> then pushing to a git repository.

When looking for a tool that did this (specifically Github -> Gitlab because the
two are most similar) I found
https://gitlab.com/sigmavirus24/issues-migration/blob/master/migrate.py
which happens to be written by Ian. I would guess that he is likely speaking 
from
experience about migrating off of Github to go to Gitlab.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github

[Donald Stufft]

> On Nov 30, 2014, at 11:30 AM, Donald Stufft  wrote:
> 
> Comments like this make me feel like I didn’t explain myself very well in the
> PEP.


It’s been pointed out to me that Mercurial bookmarks have been core since 1.8
and since I felt like the technical arguments were really secondary to the 
social
arguments I’ve gone ahead and removed that section from the PEP.

I’ve also added the read only Mercurial repository on hg.python.org 
<http://hg.python.org/>

See changes at https://hg.python.org/peps/rev/d31fe28e2766

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com