Re: [Python-Dev] Supported versions of OpenSSL

[Benjamin Peterson]

On Sun, Aug 28, 2016, at 22:42, Christian Heimes wrote:
> On 2016-08-29 04:38, Ned Deily wrote:
> > On Aug 28, 2016, at 19:06, Benjamin Peterson  wrote:
> >> On Sun, Aug 28, 2016, at 13:40, Christian Heimes wrote:
> >>> Here is the deal for 2.7 to 3.5:
> >>>
> >>> 1) All versions older than 0.9.8 are completely out-of-scope and no
> >>> longer supported.
> >> +1
> >>> 2) 0.9.8 is semi-support. Python will still compile and work with 0.9.8.
> >>> However we do NOT promise that is secure to run 0.9.8. We also require a
> >>> recent version. Patch level 0.9.8zc from October 2014 is reasonable
> >>> because it comes with SCSV fallback (CVE-2014-3566).
> >> I think we should support 0.9.8 for 2.7 and drop it for 3.6.
> > 
> > Sounds good to me, too.  I think we should also not change things for 3.5.x 
> > at this point, e.g. continue to support 0.9.8 there.
> 
> 
> In my proto-PEP I'm talking about different levels of support: full,
> build-only and unsupported. Full support means that the combination of
> Python and OpenSSL versions is reasonable secure and recommended.
> 
> On the other hand build-only support doesn't come with any security
> promise. The ssl and hashlib module are source compatible with OpenSSL
> 0.9.8. You can still compile Python, do https connections but they might
> not be secure. It's "Warranty void" mode.

I'm not sure having such "support" is a good idea. If we're not able to
support a security module securely, it's probably better if it doesn't
compile at all.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] File system path encoding on Windows

[Nick Coghlan]

On 30 August 2016 at 13:29, Steve Dower  wrote:
> The other discussion about OpenSSL and LTS systems is also interesting. Do
> we really expect users to take their fully functioning systems and blindly
> upgrade to a new major version of Python expecting everything to just work?
> That seems very unlikely to me, and also doesn't match my experience (but I
> can't quantify that in any useful way, so take it as you wish).

As far as I'm aware, new developers running new Windows and Mac OS X
releases from python.org and new versions of non-LTS Linux distros are
the folks most impacted by the 3-6 month post-release window where the
rest of the Python ecosystem plays catch-up to any breaking changes in
the new release - while a lot of major libraries and frameworks do
compatibility testing during the beta period, it's far from being a
universal practice.

Existing developers are able to stick with the versions they already
know work, and upgrade at a time that suits them, rather than getting
the newest version just because it's the newest version.

Cheers,
Nick.

P.S. My working assumption is that these two observations are not
unrelated phenomena ;)

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] File system path encoding on Windows

[Steve Dower]

On 29Aug2016 1810, Nick Coghlan wrote:

On 30 August 2016 at 08:38, Victor Stinner  wrote:

Hi,

tl; dr: just drop byte support and help developers to use Unicode in
their application!


My view (and Steve's) is that this approach is likely to result in
Linux-centric projects just dropping even nominal native Windows
support, rather than more Python software that handles Unicode on
Windows (/the CLR/the JVM) correctly.


Yeah, this basically sums it up. If I could be sure that the Python 
developers who are 99% Linux/1% Windows (i.e. run unit tests once and 
then release) weren't going to see dropping byte support completely as a 
hostile action, I'd much rather go that way.


But let's definitely take note that platform-specific deprecation 
warnings are probably not a good idea for cross-platform functionality.



What Steve is proposing here is essentially a way of providing more
*nix like CPython behaviour on Windows


Yep. What actually spurred me into action on this was a Twitter rant 
from one of Twisted's developers about paths on Windows. So I presume 
that Twisted is probably okay *now* (and hopefully because they 
explicitly decode from network traffic into str before accessing the 
file system...)


Using bytes has essentially always been using an arbitrarily-encoded str 
on Windows. The active code page is not an equivalent of "give me the 
path as raw bytes" as it is on POSIX, but my change will make it so that 
it is. There'll be a performance penalty, but otherwise using bytes for 
paths will become reliable.


Unfortunately, any implicitly-encoded cross-version interoperability 
will have to be broken by such a change. There's just no way around it. 
But I've seen no evidence that it's common, and there are two 
workarounds available (set the environment variable, or change your code 
to specify the encoding used).



However, this view is also why I don't agree with being aggressive in
making this behaviour the default on Windows - I think we should make
it readily available as a provisional feature through a single
cross-platform command line switch and environment setting (e.g. "-X
utf8" and "PYTHONASSUMEUTF8") so folks that need it can readily opt in
to it, but we can defer making it the default until 3.7 after folks
have had a full release cycle's worth of experience with it in the
wild.


Given the people who would need to opt-in to the behaviour are merely 
the recipients of a library written by someone else, I don't think this 
is the right approach. Stephen Turnbull in an earlier post referred to 
organisations that fully control their systems in order to ensure that 
the implicit encodings all match. These are also the people who can 
apply an environment variable to avoid a behaviour change.


However, someone who just installed an HTTP library that was developed 
on POSIX and perhaps not even tested on Windows should not have to flick 
the switch themselves. In contrast, if it is known that 3.6 *definitely* 
changed something here, we will certainly see more effort applied to 
making sure libraries are updated. (Compare these two bug reports: "your 
library breaks on Python 3.6" vs "your library breaks on Python 3.6 when 
I set this environment variable". The fix for the latter is quite 
reasonably going to be "don't do that".)


The other discussion about OpenSSL and LTS systems is also interesting. 
Do we really expect users to take their fully functioning systems and 
blindly upgrade to a new major version of Python expecting everything to 
just work? That seems very unlikely to me, and also doesn't match my 
experience (but I can't quantify that in any useful way, so take it as 
you wish).


Cheers,
Steve

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Supported versions of OpenSSL

[Ned Deily]

On Aug 29, 2016, at 22:59, Nick Coghlan  wrote:
> The other thing I've been looking at is how well documented the
> process is for building with a custom OpenSSL instead of the system
> one, and as near as I can tell, it isn't documented at all - the top
> level README doesn't mention it, and because the compilation is
> handled by setup.py rather than directly by make, there's no configure
> option for it the way there is for "--with-system-expat",
> "--with-system-libmpdec" and "--with-system-ffi".

There is at least one open issue with a supplied patch to provide a --with-ssl 
configure option:

http://bugs.python.org/issue21541

The patch may need some freshening up.  There are also other open issues on 
configuring third-party libraries in general.

> By contrast (and assuming I understand the situation correctly), the
> Windows build is already set up around the assumption that you'll need
> to build OpenSSL yourself.
> 
> So, in addition to updating PEP 11 with guidelines for determining the
> OpenSSL constraints for each release, I think we'll also need a README
> update to cover building against a non-system OpenSSL.

For OS X, we can copy the info about building with OpenSSL from Homebrew or 
MacPorts from the Developer's Guide. There are other things that could be done. 
The OS X installer build knows how to build OpenSSL as well as other needed 
libs.  I'd like to refactor that to make those libs usable for non-installer 
builds.  I've also thought it would be nice to provide simpler configure 
options to use Homebrew, MaPorts, or build-your-own.  That's probably not going 
to happen in time for 3.6b1, though.

--
  Ned Deily
  n...@python.org -- []

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Supported versions of OpenSSL

[Nick Coghlan]

On 30 August 2016 at 08:56, Barry Warsaw  wrote:
> On Aug 29, 2016, at 12:33 PM, Cory Benfield wrote:
>
>>Can someone explain to me why this is a use-case we care about?
>
> I do think it would be nice to be able to compile newer versions of Python on
> stock LTS releases, especially for people developing software that they want
> to support on a wide-range of Python 3 versions.

Based on this discussion, I've been thinking about how we could
articulate these build support constraints in PEP 11, similar to the
https://www.python.org/dev/peps/pep-0011/#microsoft-windows section we
have for Windows.

For that, I think it makes sense to key off source compatibility with
the most recent releases of long term community or commercially
supported Linux distros that have folks directly involved in CPython
development (specifically Debian stable, Ubuntu LTS, and RHEL/CentOS).

The other thing I've been looking at is how well documented the
process is for building with a custom OpenSSL instead of the system
one, and as near as I can tell, it isn't documented at all - the top
level README doesn't mention it, and because the compilation is
handled by setup.py rather than directly by make, there's no configure
option for it the way there is for "--with-system-expat",
"--with-system-libmpdec" and "--with-system-ffi".

By contrast (and assuming I understand the situation correctly), the
Windows build is already set up around the assumption that you'll need
to build OpenSSL yourself.

So, in addition to updating PEP 11 with guidelines for determining the
OpenSSL constraints for each release, I think we'll also need a README
update to cover building against a non-system OpenSSL.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Supported versions of OpenSSL

[Cory Benfield]

> On 29 Aug 2016, at 15:31, M.-A. Lemburg  wrote:
> 
> Ubuntu 14.04 is a widely deployed system and newer Python version
> should run on such widely deployed systems without having to
> replace important vendor maintained system libraries such as
> OpenSSL.

That's quite the non-sequitur. You never answered my question: what does 
"widely deployed" mean? At what level of deployment do we need to support the 
system configuration with no changes?

Do we need to support compiling out of box with Windows 10? Because we don't: 
if they want SSL, we need them to compile and install an OpenSSL. Do we need to 
support compiling out of the box on macOS 10.12 Sierra? Because we don't: if 
they want SSL they need to install their own OpenSSL.

At a certain point we need to give up on systems that do not provide up to date 
copies of important libraries, or say that if you want to use Python on them 
you need to compile without our support libraries. 

> Python 3.7 starts shipping around June 2018 (assuming the 18 month
> release cycle). Ubuntu 14.04 EOL is April 2019, so in order to
> be able to use Python 3.7 on such a system, you'd have to upgrade
> to a more recent LTS version 10 months before the EOL date (with
> all the associated issues) or lose vendor maintenance support and
> run with your own copy of OpenSSL.

Yes, that's true. But on the other hand, that LTS release is *already out* at 
this time, and has been for four months. By the time of the release of Python 
3.7 it will have been released for two years and two months. The request to 
upgrade is not unreasonable. 

> Sure, but Ubuntu will continue to support OpenSSL 1.0.1
> until 2019, backporting important security fixes as necessary and
> that's what's important.

Then Ubuntu can ship us an engineer who is willing to support the SSL module 
with OpenSSL 1.0.1 going forward. The one engineer we have has said he is 
unwilling to do it.

> This doesn't sound like a feature worth breaking compatibility
> to me.

Does the compatibility guarantee apply to libraries that Python will link 
against?

Cory
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Supported versions of OpenSSL

[Nick Coghlan]

On 30 August 2016 at 02:33, Cory Benfield  wrote:
>
>> On 29 Aug 2016, at 04:09, M.-A. Lemburg  wrote:
>>
>> On 28.08.2016 22:40, Christian Heimes wrote:
>>> ...
>>> I like to reduce the maintenance burden and list of supported OpenSSL
>>> versions ASAP. OpenSSL has deprecated 0.9.8 and 1.0.0 last year. 1.0.1
>>> will reach EOL by the end of this year,
>>> https://www.openssl.org/policies/releasestrat.html . However OpenSSL
>>> 0.9.8 is still required for some platforms (OSX).
>>> ...
>>> For upcoming 3.6 I would like to limit support to 1.0.2+ and require
>>> 1.0.2 features for 3.7.
>>> ...
>>
>> Hmm, that last part would mean that Python 3.7 will no longer compile
>> on e.g. Ubuntu 14.04 LTS which uses OpenSSL 1.0.1 as default version.
>> Since 14.04 LTS is supported until 2019, I think it would be better
>> to only start requiring 1.0.2 in Python 3.8.
>
> Can someone explain to me why this is a use-case we care about?

I can think of a few key cases where it's potentially going to have an impact:

- availability of new Python versions in public PaaS environments
- availability of new Python versions in free-for-open-source public CI services
- the "maximum supported Python version" for iterations of the manylinux project

Fortunately, public PaaS providers and public CI providers tend to
fall into the "upgrade Linux when new LTS releases are available, not
when the old ones are about to lose support" category, so a policy of
targeting the latest LTS releases (perhaps with a "more than 12 months
old" caveat) rather than the oldest still commercially supported ones
should be sufficient on that front (if the LTS vendors decide they'd
like us to change that policy, they always have the option of funding
it accordingly). Currently, that means:

- Debian 8
- Ubuntu 16.04 (or 14.04 with the "12 months old" caveat)
- RHEL/CentOS 7

And then trusting that supporting those will also cover other Linux
distros that aren't as directly involved in upstream CPython
development.

That means the main potential source of problems I see here is the
manylinux project, but fortunately that concern was accounted for by
explicitly excluding OpenSSL from the defined portable ABI:
https://www.python.org/dev/peps/pep-0513/#security-implications

So it's already a manylinux requirement that anyone wanting to use ssl
in a portable way needs to either use the standard library module, or
else something like PyOpenSSL (which bundles its own copy, independent
of what the system provides)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] File system path encoding on Windows

[Nick Coghlan]

On 30 August 2016 at 11:10, Daniel Holth  wrote:
> A nice recent blog post about paths encoding in a media player.
> http://beets.io/blog/paths.html . It's not merely the porting that makes it
> hard.

Interestingly, with Steve's proposed changes, their approach of using
bytes paths internally should "just work" on Windows in UTF-8 mode,
even without their syspath converter. So I may be being unduly
pessimistic in assuming this will be useful primarily for network
services - cross-platform desktop development may benefit as well.

(The additional info doesn't change my view that we'd benefit from a
release cycle's worth of availability as an opt-in provisional
behaviour though)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] File system path encoding on Windows

[Nick Coghlan]

On 30 August 2016 at 08:38, Victor Stinner  wrote:
> Hi,
>
> tl; dr: just drop byte support and help developers to use Unicode in
> their application!

My view (and Steve's) is that this approach is likely to result in
Linux-centric projects just dropping even nominal native Windows
support, rather than more Python software that handles Unicode on
Windows (/the CLR/the JVM) correctly.

What Steve is proposing here is essentially a way of providing more
*nix like CPython behaviour on Windows, just as the Windows Subsystem
for Linux is Microsoft's latest iteration on providing such an
environment at the kernel layer.

This is why interoperability with other Windows programs isn't a major
concern for this proposal - projects with Windows developers actually
working on them that are designed to operate with other applications
running on the same device (especially via files) should absolutely be
using strings rather than bytes for interacting with the operating
system.

Instead, the target audience I see for these changes is
network-centric projects where all the primary developers are running
either Linux or Mac OS X on their personal devices, and deploying to
Linux servers. If you tell them "You need to make major changes to
your project to properly support native deployment on Windows", then
they will say "OK, we'll just drop the claim of Windows support, so
we're not giving people false expectations" rather than actually
making the change. That then excludes a lot of potential contributors
from their project (since they need to be able to run Cygwin, WSL,
containers or VMs to get a compatible test environment), and it also
means folks running Windows based infrastructure can't readily run the
service up on an existing system without using like Cygwin, Docker, or
the new WSL.

However, this view is also why I don't agree with being aggressive in
making this behaviour the default on Windows - I think we should make
it readily available as a provisional feature through a single
cross-platform command line switch and environment setting (e.g. "-X
utf8" and "PYTHONASSUMEUTF8") so folks that need it can readily opt in
to it, but we can defer making it the default until 3.7 after folks
have had a full release cycle's worth of experience with it in the
wild.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] File system path encoding on Windows

[Daniel Holth]

A nice recent blog post about paths encoding in a media player.
http://beets.io/blog/paths.html . It's not merely the porting that makes it
hard.

On Mon, Aug 29, 2016, 19:32 Victor Stinner  wrote:

> 2016-08-24 17:44 GMT+02:00 Steve Dower :
> > I know Nick and Victor like the idea of a -X flag (or a direct -utf8
> flag),
> > but I prefer more specific environment variables:
> >
> > - PYTHONWINDOWSLEGACYSTDIO (for the console changes)
> > - PYTHONWINDOWSLEGACYPATHENCODING (assuming getfilesystemencoding() is
> utf8)
> > - PYTHONWINDOWSLEGACYLOCALEENCODING (assuming getpreferredencoding() is
> > utf8)
>
> Hum, you propose the opposite behaviour of my proposed "-X utf8"
> switch :-) You propose to force UTF-8 by default, but add a new flag
> to revert to the old behaviour. Right?
>
> About (...)LEGACYSTDIO: I don't think that we need such option. Using
> Unicode is the obvious choice if stdout/stderr is the Windows console
> no? I'm not talking about the case where stdout/stderr is redirected
> (into a file or a pipe). Are you aware of any case where the "legacy
> stdio" is better than the "new" (win_unicode_console-like) stdio?
>
> (...)PATHENCODING and (...)LOCALEENCODING: please don't do that. I
> tried once in Python 3.2 on Linux, and it failed badly:
> https://mail.python.org/pipermail/python-dev/2010-October/104517.html
>
> I don't think that it's a good idea to add an option to use two
> different encodings inside the same process. It's too common to
> exchange data between the "path encoding world" and "locale encoding
> world".
>
> Hum, currently, is it possible to get two different encodings for
> "path encoding" and "locale encoding"? Or both encodings are the ANSI
> code page? I only recall that the console uses a different encoding,
> the OEM code page.
>
> Victor
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Update on PEP 523 and adding a co_extra field to code objects

[Brett Cannon]

On Mon, Aug 29, 2016, 17:06 Terry Reedy  wrote:

> On 8/29/2016 5:38 PM, Brett Cannon wrote:
>
> > who objected to the new field did either for memory ("it adds another
> > pointer to the struct that won't be typically used"), or for conceptual
> > reasons ("the code object is immutable and you're proposing a mutable
> > field"). The latter is addressed by not exposing the field in Python and
>
> Am I correct is thinking that you will also not add the new field as an
> argument to PyCode_New?
>

Correct.


>  > clearly stating that code should never expect the field to be filled.
>
> I interpret this as "The only code that should access the field should
> be code that put something there."
>

Yep, seems like a reasonable rule to follow.

-brett


> --
> Terry Jan Reedy
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Update on PEP 523 and adding a co_extra field to code objects

[Terry Reedy]

On 8/29/2016 5:38 PM, Brett Cannon wrote:


who objected to the new field did either for memory ("it adds another
pointer to the struct that won't be typically used"), or for conceptual
reasons ("the code object is immutable and you're proposing a mutable
field"). The latter is addressed by not exposing the field in Python and


Am I correct is thinking that you will also not add the new field as an 
argument to PyCode_New?


> clearly stating that code should never expect the field to be filled.

I interpret this as "The only code that should access the field should 
be code that put something there."


--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Supported versions of OpenSSL

[Terry Reedy]

On 8/29/2016 5:20 PM, Christian Heimes wrote:

On 2016-08-29 23:00, Gregory P. Smith wrote:



Lets make 3.7 require a higher version. The common OSS OS distros of its
time will be better prepared.


Especially is warned.


My multissl test script allows me to compile and test _ssl.c and
_hashopenssl.c with multiple versions of OpenSSL and LibreSSL in less
than a minute. For 3.6 I have verified source compatibility with 0.9.8zc
(maybe even older) up to 1.1.0.

My argument with MAL is about future features for 3.7. I'm not planning
to require 1.0.2 APIs for 3.6 yet. This may change in case new security
issues are found. I might clean up the ssl module and require 0.9.8zc+
during beta, though.


So, in 3.6, and maybe even 3.5.3, add a deprecation warning when OSSL < 
1.0.2 is used.



--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] File system path encoding on Windows

[Victor Stinner]

2016-08-24 17:44 GMT+02:00 Steve Dower :
> I know Nick and Victor like the idea of a -X flag (or a direct -utf8 flag),
> but I prefer more specific environment variables:
>
> - PYTHONWINDOWSLEGACYSTDIO (for the console changes)
> - PYTHONWINDOWSLEGACYPATHENCODING (assuming getfilesystemencoding() is utf8)
> - PYTHONWINDOWSLEGACYLOCALEENCODING (assuming getpreferredencoding() is
> utf8)

Hum, you propose the opposite behaviour of my proposed "-X utf8"
switch :-) You propose to force UTF-8 by default, but add a new flag
to revert to the old behaviour. Right?

About (...)LEGACYSTDIO: I don't think that we need such option. Using
Unicode is the obvious choice if stdout/stderr is the Windows console
no? I'm not talking about the case where stdout/stderr is redirected
(into a file or a pipe). Are you aware of any case where the "legacy
stdio" is better than the "new" (win_unicode_console-like) stdio?

(...)PATHENCODING and (...)LOCALEENCODING: please don't do that. I
tried once in Python 3.2 on Linux, and it failed badly:
https://mail.python.org/pipermail/python-dev/2010-October/104517.html

I don't think that it's a good idea to add an option to use two
different encodings inside the same process. It's too common to
exchange data between the "path encoding world" and "locale encoding
world".

Hum, currently, is it possible to get two different encodings for
"path encoding" and "locale encoding"? Or both encodings are the ANSI
code page? I only recall that the console uses a different encoding,
the OEM code page.

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Update on PEP 523 and adding a co_extra field to code objects

[Brett Cannon]

On Mon, 29 Aug 2016 at 15:12 Maciej Fijalkowski  wrote:

> Hi Brett
>
> For what is worth, vmprof and similar tools would love such field
> (there is an open question how can you use vmprof *and* another tool,
> but later)
>

That's great to hear! Glad the solution has multiple use-cases.

-Brett


>
> On Mon, Aug 29, 2016 at 11:38 PM, Brett Cannon  wrote:
> > For quick background for those that don't remember, part of PEP 523
> proposed
> > adding a co_extra field to code objects along with making the frame
> > evaluation function pluggable
> > (https://www.python.org/dev/peps/pep-0523/#expanding-pycodeobject). The
> idea
> > was that things like JITs and debuggers could use the field as a scratch
> > space of sorts to store data related to the code object. People who
> objected
> > to the new field did either for memory ("it adds another pointer to the
> > struct that won't be typically used"), or for conceptual reasons ("the
> code
> > object is immutable and you're proposing a mutable field"). The latter is
> > addressed by not exposing the field in Python and clearly stating that
> code
> > should never expect the field to be filled.
> >
> > For the former issue of whether the memory was worth it, Dino has been
> > testing whether the field is necessary for performance from a JIT
> > perspective. Well, Dino found the time to test Pyjion without the
> co_extra
> > field and it isn't pretty. With the field, Pyjion is faster than stock
> > Python in 15 benchmarks
> > (https://github.com/Microsoft/Pyjion/tree/master/Perf). Removing the
> > co_extra field and using an unordered_map from the C++ STL drops that
> number
> > to 2. Performance is even worse if we try and use a Python dictionary
> > instead.
> >
> > That means we still want to find a solution to attach arbitrary data to
> code
> > objects without sacrificing performance. One proposal is what's in PEP
> 523
> > for the extra field. Another option is to make the memory allocator for
> code
> > objects pluggable and introduce a new flag that signals that the object
> was
> > created using a non-default allocator. Obviously we prefer the former
> > solution due to its simplicity. :)
> >
> > Anyway, we would like to get this settled this week so that I can get
> > whatever solution we agree to (if any) in next week in time for Python
> 3.6b1
> > feature freeze that would be greatly appreciated.
> >
> > ___
> > Python-Dev mailing list
> > Python-Dev@python.org
> > https://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> > https://mail.python.org/mailman/options/python-dev/fijall%40gmail.com
> >
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] File system path encoding on Windows

[Victor Stinner]

2016-08-20 21:31 GMT+02:00 Nick Coghlan :
> Reading your summary meant this finally clicked with something Victor
> has been considering for a while: a "Force UTF-8" switch that told
> Python to ignore the locale encoding on Linux, and instead assume
> UTF-8 everywhere (command line parameter parsing, environment variable
> processing, filesystem encoding, standard streams, etc)
>
> It's essentially the same problem you have on Windows, just with
> slightly different symptoms and consequences.

Yes and no, but more no than yes :-)

On Linux, the issue is quite simple: most major Linux distributions
switched to UTF-8 by default, network shares use UTF-8, filenames are
stored as UTF-8, applications expect UTF-8, etc. I proposed once a "-X
utf8" switch, but more as a convenient workaround for badly configured
system which encode data to UTF-8, but the locale encoding is not
properly configured *in some cases*. The switch does a single thing:
ignore the locale encoding, and force UTF-8 as the locale encoding.

Steve's proposition is specific to Windows, and Windows is a different
world. On Windows, there is one unique distribution: the Microsoft
flavor, and UTF-8 was and is *never* used as the ANSI code page (which
is more and less the same thing that UNIX locale encoding). Using
UTF-8 is something new, not really common in the Windows world. Steve
said that UTF-8 is common in the .NET (but I don't know well Windows
community/universe).

I proposed to Steve to work on an unified "-X utf8" option to
explicitly force UTF-8 on Linux and Windows. But Steve looks to prefer
to force UTF-8 *by default*, but add a new option to revert the old
behaviour.

I proposed the idea, but I'm not sure that we can have a single option
for Linux and Windows. Moreover, I never really worked on trying to
implement "-X utf8" on Linux, because it looks like the "misconfigured
system" are less and less common nowadays. I see very few user
requests in this direction.

By the way, except Steve, did someone complain about the ANSI code
page for bytes on Windows in Python? I recall one or two issues last 5
years about the os.listdir(bytes) issue, but these issues were
specific to Python 2 if I recall correctly?

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Update on PEP 523 and adding a co_extra field to code objects

[Brett Cannon]

On Mon, 29 Aug 2016 at 15:51 Guido van Rossum  wrote:

> Anyway, given the outcome of Dino's tests I have no objections to the
> PEP. (Though using Christian's hack would be cool.)
>

Great! I'll mark the PEP as accepted and get the implementation in for 3.6.


>
> On Mon, Aug 29, 2016 at 3:28 PM, Christian Heimes 
> wrote:
> > On 2016-08-29 23:38, Brett Cannon wrote:
> >> That means we still want to find a solution to attach arbitrary data to
> >> code objects without sacrificing performance. One proposal is what's in
> >> PEP 523 for the extra field. Another option is to make the memory
> >> allocator for code objects pluggable and introduce a new flag that
> >> signals that the object was created using a non-default allocator.
> >> Obviously we prefer the former solution due to its simplicity. :)
> >
> > May I remind you that you can have the field with no extra memory cost?
> > :)


Yes you may. :)


> The struct has sub-par alignments.
>

So the struct in question can be found at
https://github.com/python/cpython/blob/2d264235f6e066611b412f7c2e1603866e0f7f1b/Include/code.h#L10
.
The official docs say the fields can be changed at any time, so
re-arranging them shouldn't break any ABI compatibility promises:
https://docs.python.org/3/c-api/code.html#c.PyCodeObject . Would grouping
all the fields of the same type together, sorting them by individual field
size (i.e. PyObject*, void*, int, unsigned char*), and then adding the
co_extra field at the end of the grouping of PyObject * fields do what
you're suggesting?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Supported versions of OpenSSL

[Barry Warsaw]

On Aug 29, 2016, at 12:33 PM, Cory Benfield wrote:

>Can someone explain to me why this is a use-case we care about?

I do think it would be nice to be able to compile newer versions of Python on
stock LTS releases, especially for people developing software that they want
to support on a wide-range of Python 3 versions.

Sure, you can use your container of choice, and there are ways around this,
but every hurdle just makes it more of a pain to do.  So I'd say if we can
keep the compatibility without too much effort, and without sacrificing the
security or maintainability of the code, we should do it.  If we can't, we
can't.

Cheers,
-Barry
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Update on PEP 523 and adding a co_extra field to code objects

[Guido van Rossum]

Anyway, given the outcome of Dino's tests I have no objections to the
PEP. (Though using Christian's hack would be cool.)

On Mon, Aug 29, 2016 at 3:28 PM, Christian Heimes  wrote:
> On 2016-08-29 23:38, Brett Cannon wrote:
>> That means we still want to find a solution to attach arbitrary data to
>> code objects without sacrificing performance. One proposal is what's in
>> PEP 523 for the extra field. Another option is to make the memory
>> allocator for code objects pluggable and introduce a new flag that
>> signals that the object was created using a non-default allocator.
>> Obviously we prefer the former solution due to its simplicity. :)
>
> May I remind you that you can have the field with no extra memory cost?
> :) The struct has sub-par alignments.
>
> Christian
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/guido%40python.org



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] File system path encoding on Windows

[Victor Stinner]

Hi,

tl; dr: just drop byte support and help developers to use Unicode in
their application!


As you may already know, I dislike your whole project. But first of
all, IMHO you should open a separated thread to discuss changes
related to the Windows console. I consider that they are unrelated,
well defined and makes sense to everyone, no?


It would help the discussion to estimate how much code is going to
break on Windows if Python 3 drops byte support. Maybe try some common
and major applications like Django and Twisted? My expectation is that
very few code explicitly uses bytes path, and that most applicatons
already use Unicode, just because it's much more easier to use Unicode
than bytes on Python 3.

Mercurial does its best to keep filenames as bytes, unchanged. I tried
once to port Mercurial to Python 3, and I recall that it was very
painful to handle filenames as bytes. A simple example:
print(filename) raises a BytesWarning (or displays a quoted string
prefixed with 'b', not really the expected result).


You always use the same example os.listdir(bytes). Ok, this one is a
mess on Windows. But modifying the value of
sys.getfilesystemencoding() has a broad effect: it changes *all*
functions in Python (ok, not really "all", but I'm trying to say that
a lot of functions use it). It's not only filenames: hostnames
(sockets), input/output data from other applications (subprocess
pipes), file content, command line arguments, environment variables,
etc. also use the ANSI code page.

Well, I know well UNIX where everything is stored as bytes. But I know
that hopefully, Windows has cool functions providing data directly as
Unicode, avoiding any mojibake issues, at least for command line
arguments and environment variables. But while your application may be
carefuly with Unicode, in practice "data" is transfered between many
channels and suffers from conversions.

There is no such single channel well defined. For example, "pipes" are
a pain on Windows. There is no "Unicode pipe" to exchange *text*
between two processes. I only know "cmd /u" (which seems specific and
restricted to cmd.exec) or _setmode() which has its own set of issues:
http://bugs.python.org/issue16587

See also this old article
http://archives.miloush.net/michkap/archive/2008/03/18/8306597.html


If Python 3.6 is going to speak UTF-8 only, there is a high risk that
it is going to produce data unreadable by other applications. For
example, write a filename into a file. I expect that the filename will
be stored as bytes, so as UTF-8. If another application tries to
decode the file content from the ANSI code page, it's likely that you
will get a "fatal" Unicode decoding error at the first non-ASCII
character.

The status quo is not better, you cannot store Japanese filenames in a
file if your ANSI code page is french. But if you decide to produce a
UTF-8 file, the effect will be restricted to this file, not to *all*
input and output data! For example, distutils was slowly upgraded
(from "encoding not specified by any spec, but more likely the locale
encoding") to use UTF-8 everywhere ;-)


More generally, I understand that you propose to explicitly break the
interoperability with other applications. Python 2 can be used as an
example of "application" expecting data encoded to the ANSI code page.

Yeah, I expect that storing text as UTF-8 inside the same process will
reduce the number of unicode errors *inside the same process*. It's
likely that you will no more get unicode errors if you control all
data inside the same process.

But for me, interoperability is more important than benefits of your
proposed changes.


Another way to express my concern: I don't understand how you plan to
"transcode" data from/to other applications. Will subprocess guess the
input and output encoding and transcode between the guessed encoding
and UTF-8?

Even today with Python 3.5, it's already easy to get mojibake betwen
the ANSI code page, the OEM code page, and some other encodings used
by an application. In short, you introduce yet another encoding which
is rare on Windows and incompatible with other encodings (well, except
of ASCII which is subset of many encodings, most encoings are
incompatible with all other encodings).


To me, you are building a giant patch to hide the mess, whereas this
problem was already solved in 1991 with the release of Unicode 1.0.
I'm not saying that it's a mistake to use UTF-8 internally. It's more
than I don't think that it's worth it. There is a risk that it adds
extra work to "support UTF-8", whereas this energy would be better
used to support Unicode, no?


I suggest to either fix the problem by really dropping bytes support,
as "announced" in Python 3.3 with the deprecation, or do nothing (and
continue to suffer).

I'm a supporter of the former option :-) Force developers to fix their
application and learn Unicode the hard way :-)

>From what I heard, Unicode is not really a pain of Python 3. The pain
comes from porting 

Re: [Python-Dev] Update on PEP 523 and adding a co_extra field to code objects

[Christian Heimes]

On 2016-08-29 23:38, Brett Cannon wrote:
> That means we still want to find a solution to attach arbitrary data to
> code objects without sacrificing performance. One proposal is what's in
> PEP 523 for the extra field. Another option is to make the memory
> allocator for code objects pluggable and introduce a new flag that
> signals that the object was created using a non-default allocator.
> Obviously we prefer the former solution due to its simplicity. :)

May I remind you that you can have the field with no extra memory cost?
:) The struct has sub-par alignments.

Christian
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Update on PEP 523 and adding a co_extra field to code objects

[Maciej Fijalkowski]

Hi Brett

For what is worth, vmprof and similar tools would love such field
(there is an open question how can you use vmprof *and* another tool,
but later)

On Mon, Aug 29, 2016 at 11:38 PM, Brett Cannon  wrote:
> For quick background for those that don't remember, part of PEP 523 proposed
> adding a co_extra field to code objects along with making the frame
> evaluation function pluggable
> (https://www.python.org/dev/peps/pep-0523/#expanding-pycodeobject). The idea
> was that things like JITs and debuggers could use the field as a scratch
> space of sorts to store data related to the code object. People who objected
> to the new field did either for memory ("it adds another pointer to the
> struct that won't be typically used"), or for conceptual reasons ("the code
> object is immutable and you're proposing a mutable field"). The latter is
> addressed by not exposing the field in Python and clearly stating that code
> should never expect the field to be filled.
>
> For the former issue of whether the memory was worth it, Dino has been
> testing whether the field is necessary for performance from a JIT
> perspective. Well, Dino found the time to test Pyjion without the co_extra
> field and it isn't pretty. With the field, Pyjion is faster than stock
> Python in 15 benchmarks
> (https://github.com/Microsoft/Pyjion/tree/master/Perf). Removing the
> co_extra field and using an unordered_map from the C++ STL drops that number
> to 2. Performance is even worse if we try and use a Python dictionary
> instead.
>
> That means we still want to find a solution to attach arbitrary data to code
> objects without sacrificing performance. One proposal is what's in PEP 523
> for the extra field. Another option is to make the memory allocator for code
> objects pluggable and introduce a new flag that signals that the object was
> created using a non-default allocator. Obviously we prefer the former
> solution due to its simplicity. :)
>
> Anyway, we would like to get this settled this week so that I can get
> whatever solution we agree to (if any) in next week in time for Python 3.6b1
> feature freeze that would be greatly appreciated.
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/fijall%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Update on PEP 523 and adding a co_extra field to code objects

[Brett Cannon]

For quick background for those that don't remember, part of PEP 523
proposed adding a co_extra field to code objects along with making the
frame evaluation function pluggable (
https://www.python.org/dev/peps/pep-0523/#expanding-pycodeobject). The idea
was that things like JITs and debuggers could use the field as a scratch
space of sorts to store data related to the code object. People who
objected to the new field did either for memory ("it adds another pointer
to the struct that won't be typically used"), or for conceptual reasons
("the code object is immutable and you're proposing a mutable field"). The
latter is addressed by not exposing the field in Python and clearly stating
that code should never expect the field to be filled.

For the former issue of whether the memory was worth it, Dino has been
testing whether the field is necessary for performance from a JIT
perspective. Well, Dino found the time to test Pyjion without the co_extra
field and it isn't pretty. With the field, Pyjion is faster than stock
Python in 15 benchmarks (
https://github.com/Microsoft/Pyjion/tree/master/Perf). Removing the
co_extra field and using an unordered_map from the C++ STL drops that
number to 2. Performance is even worse if we try and use a Python
dictionary instead.

That means we still want to find a solution to attach arbitrary data to
code objects without sacrificing performance. One proposal is what's in PEP
523 for the extra field. Another option is to make the memory allocator for
code objects pluggable and introduce a new flag that signals that the
object was created using a non-default allocator. Obviously we prefer the
former solution due to its simplicity. :)

Anyway, we would like to get this settled this week so that I can get
whatever solution we agree to (if any) in next week in time for Python
3.6b1 feature freeze that would be greatly appreciated.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Supported versions of OpenSSL

[Christian Heimes]

On 2016-08-29 23:00, Gregory P. Smith wrote:
> 
> Given that you already said:
> 
> """
> For 3.6 I don't require any 1.0.2 feature yet. The 1.1.0 patch keeps
> code compatible with 0.9.8zc to 1.1.0. But as soon as I use new
> features, the ssl module will no longer be source and build compatible
> with < 1.0.2. There is also the point of OpenSSL 1.0.1. It reaches
> end-of-lifetime by the end if this year. 1.0.2 will be supported until 2019.
> 
> I'm tempted to require 1.0.2 for Python 3.6 but it's technically not
> necessary yet.
> """
> 
> That to me means we should keep support for 1.0.1 in Python 3.6 unless
> there are features in 1.0.2 that you find are an absolute must have
> within the next two weeks. We're going to be entering 3.6beta on
> September 12th and current stable distros do not ship with a more recent
> version so lets not make the lives of our developers and buildbot
> maintainers hell by forcing them to install a special version.
> 
> Lets make 3.7 require a higher version. The common OSS OS distros of its
> time will be better prepared.

My multissl test script allows me to compile and test _ssl.c and
_hashopenssl.c with multiple versions of OpenSSL and LibreSSL in less
than a minute. For 3.6 I have verified source compatibility with 0.9.8zc
(maybe even older) up to 1.1.0.

My argument with MAL is about future features for 3.7. I'm not planning
to require 1.0.2 APIs for 3.6 yet. This may change in case new security
issues are found. I might clean up the ssl module and require 0.9.8zc+
during beta, though.

Christian



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Supported versions of OpenSSL

[Gregory P. Smith]

On Mon, Aug 29, 2016 at 1:18 PM Christian Heimes 
wrote:

> On 2016-08-29 21:31, M.-A. Lemburg wrote:
> > On 29.08.2016 18:33, Cory Benfield wrote:
> >>
> >>> On 29 Aug 2016, at 04:09, M.-A. Lemburg  wrote:
> >>>
> >>> On 28.08.2016 22:40, Christian Heimes wrote:
>  ...
>  I like to reduce the maintenance burden and list of supported OpenSSL
>  versions ASAP. OpenSSL has deprecated 0.9.8 and 1.0.0 last year. 1.0.1
>  will reach EOL by the end of this year,
>  https://www.openssl.org/policies/releasestrat.html . However OpenSSL
>  0.9.8 is still required for some platforms (OSX).
>  ...
>  For upcoming 3.6 I would like to limit support to 1.0.2+ and require
>  1.0.2 features for 3.7.
>  ...
> >>>
> >>> Hmm, that last part would mean that Python 3.7 will no longer compile
> >>> on e.g. Ubuntu 14.04 LTS which uses OpenSSL 1.0.1 as default version.
> >>> Since 14.04 LTS is supported until 2019, I think it would be better
> >>> to only start requiring 1.0.2 in Python 3.8.
> >>
> >> Can someone explain to me why this is a use-case we care about?
> >
> > Ubuntu 14.04 is a widely deployed system and newer Python version
> > should run on such widely deployed systems without having to
> > replace important vendor maintained system libraries such as
> > OpenSSL.
>
> "Widely deployed" is true for a lot of old operating systems including
> Windows XP.
>
> > Python 3.7 starts shipping around June 2018 (assuming the 18 month
> > release cycle). Ubuntu 14.04 EOL is April 2019, so in order to
> > be able to use Python 3.7 on such a system, you'd have to upgrade
> > to a more recent LTS version 10 months before the EOL date (with
> > all the associated issues) or lose vendor maintenance support and
> > run with your own copy of OpenSSL.
>
> Why would you deploy an unsupported Python version on a LTS release? Why
> should compatibility be our concern?
>
> > Sure, but Ubuntu will continue to support OpenSSL 1.0.1
> > until 2019, backporting important security fixes as necessary and
> > that's what's important.
>
> I see an easy solution here: either pay or make Canonical backport all
> required features to OpenSSL 1.0.1. 
>
> > It's unfortunate that Python has to rely on a 3rd party library
> > for security, but we should at least make sure that our users
> > can rely on OS vendor support to keep the lib up to date with
> > security fixes.
>
> No, it is a good thing that we can rely on 3rd party libraries for
> security. Crypto and security is not our domain. It is incredible hard
> to develop and maintain crypto code. Also my proposal enforces OS
> vendors to supply up to date OpenSSL versions.
>
> >
> > On 29.08.2016 10:24, Christian Heimes wrote:
> >> By the way I knew that something like this would come up from you.
> >> Thank you that you satisfied my expectation. :p
> >
> > Sure, I want Python to be used on as many systems as possible,
> > both in terms of architecture and OS. The more the better.
> > If we don't have to drop support early, why should we ?
>
> MAL, I don't like your attitude. It feels like you want me and other
> contributors to waste time on this topic. That is not how this
> discussion is going to end. If *you* want to keep support for outdated
> OpenSSL versions, than it is *your* responsibility and *your* time. You
> cannot and will not put this burden on me.
>

Please keep your dialog civil Christian. That was unwarranted.

Nobody was forcing a burden upon you. 3.6 will remain buildable and usable
on common stable distros so long as we support 1.0.1 which sounds easy to
do.  For 3.7 we can move on and raise the minimum beyond that.

...

> opinion it is more than reasonable to ditch 1.0.1 and earlier.
>

Given that you already said:

"""
For 3.6 I don't require any 1.0.2 feature yet. The 1.1.0 patch keeps
code compatible with 0.9.8zc to 1.1.0. But as soon as I use new
features, the ssl module will no longer be source and build compatible
with < 1.0.2. There is also the point of OpenSSL 1.0.1. It reaches
end-of-lifetime by the end if this year. 1.0.2 will be supported until 2019.

I'm tempted to require 1.0.2 for Python 3.6 but it's technically not
necessary yet.
"""

That to me means we should keep support for 1.0.1 in Python 3.6 unless
there are features in 1.0.2 that you find are an absolute must have within
the next two weeks. We're going to be entering 3.6beta on September 12th
and current stable distros do not ship with a more recent version so lets
not make the lives of our developers and buildbot maintainers hell by
forcing them to install a special version.

Lets make 3.7 require a higher version. The common OSS OS distros of its
time will be better prepared.

-gps
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Supported versions of OpenSSL

[Christian Heimes]

On 2016-08-29 22:10, Random832 wrote:
> On Mon, Aug 29, 2016, at 04:09, M.-A. Lemburg wrote:
>> Hmm, that last part would mean that Python 3.7 will no longer compile
>> on e.g. Ubuntu 14.04 LTS which uses OpenSSL 1.0.1 as default version.
>> Since 14.04 LTS is supported until 2019, I think it would be better
>> to only start requiring 1.0.2 in Python 3.8.
> 
> If someone's going to compile a new version of Python for Ubuntu LTS,
> they can compile a new version of OpenSSL. How likely is it that someone
> will want to use the packaged version of OpenSSL, but not use the
> packaged version (which is 3.4) of Python?

Please let me rephrase the question. How likely is it that somebody
won't use a container to deploy more recent versions? It's 2016.

Christian


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Supported versions of OpenSSL

[Christian Heimes]

On 2016-08-29 21:31, M.-A. Lemburg wrote:
> On 29.08.2016 18:33, Cory Benfield wrote:
>>
>>> On 29 Aug 2016, at 04:09, M.-A. Lemburg  wrote:
>>>
>>> On 28.08.2016 22:40, Christian Heimes wrote:
 ...
 I like to reduce the maintenance burden and list of supported OpenSSL
 versions ASAP. OpenSSL has deprecated 0.9.8 and 1.0.0 last year. 1.0.1
 will reach EOL by the end of this year,
 https://www.openssl.org/policies/releasestrat.html . However OpenSSL
 0.9.8 is still required for some platforms (OSX).
 ...
 For upcoming 3.6 I would like to limit support to 1.0.2+ and require
 1.0.2 features for 3.7.
 ...
>>>
>>> Hmm, that last part would mean that Python 3.7 will no longer compile
>>> on e.g. Ubuntu 14.04 LTS which uses OpenSSL 1.0.1 as default version.
>>> Since 14.04 LTS is supported until 2019, I think it would be better
>>> to only start requiring 1.0.2 in Python 3.8.
>>
>> Can someone explain to me why this is a use-case we care about?
> 
> Ubuntu 14.04 is a widely deployed system and newer Python version
> should run on such widely deployed systems without having to
> replace important vendor maintained system libraries such as
> OpenSSL.

"Widely deployed" is true for a lot of old operating systems including
Windows XP.

> Python 3.7 starts shipping around June 2018 (assuming the 18 month
> release cycle). Ubuntu 14.04 EOL is April 2019, so in order to
> be able to use Python 3.7 on such a system, you'd have to upgrade
> to a more recent LTS version 10 months before the EOL date (with
> all the associated issues) or lose vendor maintenance support and
> run with your own copy of OpenSSL.

Why would you deploy an unsupported Python version on a LTS release? Why
should compatibility be our concern?

> Sure, but Ubuntu will continue to support OpenSSL 1.0.1
> until 2019, backporting important security fixes as necessary and
> that's what's important.

I see an easy solution here: either pay or make Canonical backport all
required features to OpenSSL 1.0.1. 

> It's unfortunate that Python has to rely on a 3rd party library
> for security, but we should at least make sure that our users
> can rely on OS vendor support to keep the lib up to date with
> security fixes.

No, it is a good thing that we can rely on 3rd party libraries for
security. Crypto and security is not our domain. It is incredible hard
to develop and maintain crypto code. Also my proposal enforces OS
vendors to supply up to date OpenSSL versions.

> 
> On 29.08.2016 10:24, Christian Heimes wrote:
>> By the way I knew that something like this would come up from you.
>> Thank you that you satisfied my expectation. :p
> 
> Sure, I want Python to be used on as many systems as possible,
> both in terms of architecture and OS. The more the better.
> If we don't have to drop support early, why should we ?

MAL, I don't like your attitude. It feels like you want me and other
contributors to waste time on this topic. That is not how this
discussion is going to end. If *you* want to keep support for outdated
OpenSSL versions, than it is *your* responsibility and *your* time. You
cannot and will not put this burden on me.

Python is running out of developers with OpenSSL expertise. It's Alex,
Antoine, Benjamin, Victor and me. Antoine and me haven't been active for
a while. Victor and Benjamin are mostly working on other topics. As far
as I can judge Alex, he rather works on PyCA than CPython stdlib.

I'm both interested and willing to improve Python's ssl stack, and I'm
going to do this in my own free time. Yes, I'm working for Red Hat's
security engineering, but I'm not getting paid to work on Python (except
for a couple of hours now and then when a bug is relevant for my daily
work). I will only contribute improvements and fixes on my own terms,
that means I'm not going to waste my time with outdated versions. In my
opinion it is more than reasonable to ditch 1.0.1 and earlier.

> This doesn't sound like a feature worth breaking compatibility
> to me.

It does.

Christian
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Supported versions of OpenSSL

[Random832]

On Mon, Aug 29, 2016, at 04:09, M.-A. Lemburg wrote:
> Hmm, that last part would mean that Python 3.7 will no longer compile
> on e.g. Ubuntu 14.04 LTS which uses OpenSSL 1.0.1 as default version.
> Since 14.04 LTS is supported until 2019, I think it would be better
> to only start requiring 1.0.2 in Python 3.8.

If someone's going to compile a new version of Python for Ubuntu LTS,
they can compile a new version of OpenSSL. How likely is it that someone
will want to use the packaged version of OpenSSL, but not use the
packaged version (which is 3.4) of Python?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Supported versions of OpenSSL

[M.-A. Lemburg]

On 29.08.2016 18:33, Cory Benfield wrote:
> 
>> On 29 Aug 2016, at 04:09, M.-A. Lemburg  wrote:
>>
>> On 28.08.2016 22:40, Christian Heimes wrote:
>>> ...
>>> I like to reduce the maintenance burden and list of supported OpenSSL
>>> versions ASAP. OpenSSL has deprecated 0.9.8 and 1.0.0 last year. 1.0.1
>>> will reach EOL by the end of this year,
>>> https://www.openssl.org/policies/releasestrat.html . However OpenSSL
>>> 0.9.8 is still required for some platforms (OSX).
>>> ...
>>> For upcoming 3.6 I would like to limit support to 1.0.2+ and require
>>> 1.0.2 features for 3.7.
>>> ...
>>
>> Hmm, that last part would mean that Python 3.7 will no longer compile
>> on e.g. Ubuntu 14.04 LTS which uses OpenSSL 1.0.1 as default version.
>> Since 14.04 LTS is supported until 2019, I think it would be better
>> to only start requiring 1.0.2 in Python 3.8.
> 
> Can someone explain to me why this is a use-case we care about?

Ubuntu 14.04 is a widely deployed system and newer Python version
should run on such widely deployed systems without having to
replace important vendor maintained system libraries such as
OpenSSL.

Python 3.7 starts shipping around June 2018 (assuming the 18 month
release cycle). Ubuntu 14.04 EOL is April 2019, so in order to
be able to use Python 3.7 on such a system, you'd have to upgrade
to a more recent LTS version 10 months before the EOL date (with
all the associated issues) or lose vendor maintenance support and
run with your own copy of OpenSSL.

OTOH, the situations isn't all that bad: people on such systems
will likely not go straight for the 3.7.0 release of Python, but
instead wait for 3.7.2+ for the dust to settle.

> The nature of a stable distribution is that all the packages are guaranteed 
> to work together. Once you start compiling your own software, you are out in 
> the cold: you are no longer covered by that guarantee. Why, then, should 
> someone compiling a version of Python that was barely conceived when Ubuntu 
> 14.04 was released expect that they can compile it from source using only 
> dependencies available in their mainline distribution?
> 
> I absolutely understand wanting to support Ubuntu 14.04 for any release of 
> Python that already compiles on Ubuntu 14.04: that makes sense. But new 
> releases should not be shackled to what is in LTS releases of operating 
> systems. If it were, a more coherent argument would be that we cannot drop 
> 0.9.8 support in any Python released before the middle of 2017 because RHEL 5 
> ships it, and we will not be able to require OpenSSL 1.0.2 until almost 2021 
> because RHEL 6 ships 1.0.1. After all, why does Ubuntu 14.04 get privileged 
> over RHEL? Both are supported LTS releases, after all.
> 
> I don’t believe that the Python dev team has ever promised that future 
> versions of Python will compile on a given LTS release. Am I mistaken?
> 
> While I’m here, I should note that Ubuntu 14.04 goes EOL in 2019, while 
> OpenSSL 1.0.1 loses support from upstream at the end of this year, which is 
> probably a good reason to stop supporting it in new Python versions. OpenSSL 
> 1.0.0 and 0.9.8 are already unsupported by upstream.

Sure, but Ubuntu will continue to support OpenSSL 1.0.1
until 2019, backporting important security fixes as necessary and
that's what's important.

It's unfortunate that Python has to rely on a 3rd party library
for security, but we should at least make sure that our users
can rely on OS vendor support to keep the lib up to date with
security fixes.

On 29.08.2016 10:24, Christian Heimes wrote:
> By the way I knew that something like this would come up from you.
> Thank you that you satisfied my expectation. :p

Sure, I want Python to be used on as many systems as possible,
both in terms of architecture and OS. The more the better.
If we don't have to drop support early, why should we ?

>> BTW: Are there any features in 1.0.2 that we need and would warrant
>> dropping support for 1.0.1 earlier than Ubuntu 14.04 LTS ?
>
> Yes, there are features I want to use, e.g. proper hostname
> verification. Python's post-handshake verification is a hack and leads
> to information disclosure.

This doesn't sound like a feature worth breaking compatibility
to me.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Aug 29 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/


::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  

Re: [Python-Dev] Supported versions of OpenSSL

[Brett Cannon]

On Mon, 29 Aug 2016 at 09:34 Cory Benfield  wrote:

>
> > On 29 Aug 2016, at 04:09, M.-A. Lemburg  wrote:
> >
> > On 28.08.2016 22:40, Christian Heimes wrote:
> >> ...
> >> I like to reduce the maintenance burden and list of supported OpenSSL
> >> versions ASAP. OpenSSL has deprecated 0.9.8 and 1.0.0 last year. 1.0.1
> >> will reach EOL by the end of this year,
> >> https://www.openssl.org/policies/releasestrat.html . However OpenSSL
> >> 0.9.8 is still required for some platforms (OSX).
> >> ...
> >> For upcoming 3.6 I would like to limit support to 1.0.2+ and require
> >> 1.0.2 features for 3.7.
> >> ...
> >
> > Hmm, that last part would mean that Python 3.7 will no longer compile
> > on e.g. Ubuntu 14.04 LTS which uses OpenSSL 1.0.1 as default version.
> > Since 14.04 LTS is supported until 2019, I think it would be better
> > to only start requiring 1.0.2 in Python 3.8.
>
> Can someone explain to me why this is a use-case we care about?
>
> The nature of a stable distribution is that all the packages are
> guaranteed to work together. Once you start compiling your own software,
> you are out in the cold: you are no longer covered by that guarantee. Why,
> then, should someone compiling a version of Python that was barely
> conceived when Ubuntu 14.04 was released expect that they can compile it
> from source using only dependencies available in their mainline
> distribution?
>

I also agree with this view. If you're on an LTS then you should expect
everything that comes with the LTS to work, not new software that you
pulled forward unless your OS distributor supports it, e.g. RHEL
collections.


>
> I absolutely understand wanting to support Ubuntu 14.04 for any release of
> Python that already compiles on Ubuntu 14.04: that makes sense. But new
> releases should not be shackled to what is in LTS releases of operating
> systems. If it were, a more coherent argument would be that we cannot drop
> 0.9.8 support in any Python released before the middle of 2017 because RHEL
> 5 ships it, and we will not be able to require OpenSSL 1.0.2 until almost
> 2021 because RHEL 6 ships 1.0.1. After all, why does Ubuntu 14.04 get
> privileged over RHEL? Both are supported LTS releases, after all.
>

No one gets special privileges beyond someone funding a core dev to put
into the effort to support a platform that won't shackle others.


>
> I don’t believe that the Python dev team has ever promised that future
> versions of Python will compile on a given LTS release. Am I mistaken?
>

No, literally the only support promise we make along these lines is that we
will support Windows versions that are still in extended support:
https://www.python.org/dev/peps/pep-0011/#id10 , and that promise only
works due to Windows not being a moving target in terms of
backwards-compatibility.


>
> While I’m here, I should note that Ubuntu 14.04 goes EOL in 2019, while
> OpenSSL 1.0.1 loses support from upstream at the end of this year, which is
> probably a good reason to stop supporting it in new Python versions.
> OpenSSL 1.0.0 and 0.9.8 are already unsupported by upstream.
>

I agree w/ Christian's proposal of matching what OpenSSL upstream supports
when releasing new versions of CPython. I don't see how supporting outdated
security code is good for anyone involved.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Supported versions of OpenSSL

[Cory Benfield]

> On 29 Aug 2016, at 04:09, M.-A. Lemburg  wrote:
> 
> On 28.08.2016 22:40, Christian Heimes wrote:
>> ...
>> I like to reduce the maintenance burden and list of supported OpenSSL
>> versions ASAP. OpenSSL has deprecated 0.9.8 and 1.0.0 last year. 1.0.1
>> will reach EOL by the end of this year,
>> https://www.openssl.org/policies/releasestrat.html . However OpenSSL
>> 0.9.8 is still required for some platforms (OSX).
>> ...
>> For upcoming 3.6 I would like to limit support to 1.0.2+ and require
>> 1.0.2 features for 3.7.
>> ...
> 
> Hmm, that last part would mean that Python 3.7 will no longer compile
> on e.g. Ubuntu 14.04 LTS which uses OpenSSL 1.0.1 as default version.
> Since 14.04 LTS is supported until 2019, I think it would be better
> to only start requiring 1.0.2 in Python 3.8.

Can someone explain to me why this is a use-case we care about?

The nature of a stable distribution is that all the packages are guaranteed to 
work together. Once you start compiling your own software, you are out in the 
cold: you are no longer covered by that guarantee. Why, then, should someone 
compiling a version of Python that was barely conceived when Ubuntu 14.04 was 
released expect that they can compile it from source using only dependencies 
available in their mainline distribution?

I absolutely understand wanting to support Ubuntu 14.04 for any release of 
Python that already compiles on Ubuntu 14.04: that makes sense. But new 
releases should not be shackled to what is in LTS releases of operating 
systems. If it were, a more coherent argument would be that we cannot drop 
0.9.8 support in any Python released before the middle of 2017 because RHEL 5 
ships it, and we will not be able to require OpenSSL 1.0.2 until almost 2021 
because RHEL 6 ships 1.0.1. After all, why does Ubuntu 14.04 get privileged 
over RHEL? Both are supported LTS releases, after all.

I don’t believe that the Python dev team has ever promised that future versions 
of Python will compile on a given LTS release. Am I mistaken?

While I’m here, I should note that Ubuntu 14.04 goes EOL in 2019, while OpenSSL 
1.0.1 loses support from upstream at the end of this year, which is probably a 
good reason to stop supporting it in new Python versions. OpenSSL 1.0.0 and 
0.9.8 are already unsupported by upstream.

Cory

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] File system path encoding on Windows

[Steve Dower]

On 28Aug2016 2043, Stephen J. Turnbull wrote:

tritium-l...@sdamon.com writes:

 > Once you get to var lengths like that, arcane single character flags start
 > looking preferable.  How about "PYTHONWINLEGACY" to just turn it all on or
 > off.  If the code breaks on one thing, it obviously isn't written to use the
 > other two, so might as well shut them all off.

Since Steve is thinking about three separate PEPs (among other things,
they might be implemented on different timelines), that's not really
possible (placing the features under control of one switch at
different times would be an unacceptable compatibility break).


Yeah, the likelihood of different timelines basically means three PEPs 
are going to be necessary. But I think we can have a single 
"PYTHONWINDOWSANSI" (or ...MBCS) flag to cover all three whenever they 
come in without it being a compatibility break, especially if (as Nick 
suggested) there are _PYTHONWINDOWSANSI(CONSOLE|PATH|LOCALE) flags too. 
But it does give us the ability to say "all ANSI or all UTF-8 are 
supported; mix-and-match at your own risk".



Anyway, it's not *obvious* that your premise is true, because code
isn't written to do any of those things.  It's written to process
bytes agnostically.  The question is what does the environment look
like.  Steve obviously has a perspective on environment which suggests
that these aspects are often decoupled because in Windows the actual
filesystem is never bytes-oriented.  I don't know if it's possible to
construct a coherent environment where these aspects are decoupled,
but I can't say it's impossible, either.


Actually, the three items are basically completely decoupled, though it 
isn't obvious.


* stdin/stdout/stderr are text wrappers by default (under my changes, 
using the console encoding when it's a console and the locale encoding 
when it's a file/pipe). There's no point reading bytes from the console, 
and redirected files or pipes are unaffected by the change.
* the file system encoding only affects paths passed into/returned from 
the OS as bytes, and...

* the locale encoding affects files opened in text mode, which means...
* if you open('rb') and read paths, the locale encoding has no effect on 
whether the bytes are the right encoding to be used as paths


So while there are scenarios that use multiple pieces of this, there 
should only be one change impacting any scenario:

* reading str paths from a file - locale encoding
* reading bytes paths from a file - filesystem encoding
* reading str paths from a pipe/redirected file - locale encoding
* reading bytes paths from a pipe/redirected file - filesystem encoding
* reading str paths from the console - console encoding
* reading bytes paths from the console (i.e. 
sys.stdin.buffer.raw.read()) - filesystem encoding


The last case doesn't make sense anyway right now, as 
sys.stdin.buffer.raw has no specified encoding and you can't reliably 
read paths from it. Perhaps there exist examples of where this is put to 
good use (bearing in mind it must be an actual console - not a 
redirection or pipe) - I would love to hear about them.


As far as I can tell, any other combination requires the Python 
developer to convert between str and bytes themselves, which may lead to 
errors if they have assumed that the encoding of the bytes would never 
change, but code that ignores encodings and uses bytes or str 
exclusively is only going to encounter one (bytes) or two (str) of the 
changes.


Cheers,
Steve

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] What do we do about bad slicing and possible crashes (issue 27867)

[Ethan Furman]

On 08/28/2016 09:26 AM, Nick Coghlan wrote:

On 28 August 2016 at 08:25, Terry Reedy  wrote:

Slicing can be made to malfunction and even crash with an 'evil' __index__
method. https://bugs.python.org/issue27867

The crux of the problem is this: PySlice_GetIndicesEx
receives a slice object and a sequence length.  Calling __index__ on the
start, stop, and step components can mutate the sequence and invalidate the
length.  Adjusting the int values of start and stop according to an invalid
length (in particular, one that is too long) will result in invalid results
or a crash.

Possible actions -- very briefly.  For more see end of
https://bugs.python.org/issue27867?@ok_message=msg 273801
0. Do nothing.
1. Detect length change and raise.


I suggest taking this path - it's the lowest impact, and akin to the
"dictionary changed size during iteration" runtime error.


+1.  Being able to do such strange things with list but not dict would be 
irritating and a nuisance (although maybe not attractive ;) .


__index__ having side effects is pathological code behaviour, so we
really just need to prevent the interpreter crash, rather than trying
to make it sense of it.


Agreed.

--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Supported versions of OpenSSL

[Chris Angelico]

On Mon, Aug 29, 2016 at 9:16 PM, Nick Coghlan  wrote:
> On 29 August 2016 at 21:05, Chris Angelico  wrote:
>> On Mon, Aug 29, 2016 at 8:52 PM, Nick Coghlan  wrote:
>>> For upcoming 3.6 I would like to limit support to 1.0.2+ and require
>>> 1.0.2 features for 3.7.
>>
>> What does "limit support" mean? Will it be possible to build CPython
>> 3.6 against OpenSSL 1.0.1?
>
> Christian clarified this later in the thread:
>
> - full support is stating confidently that software running that way
> is using network connections that are as secure as we know how to make
> them
> - build support is ensuring it builds, without vouching one way or the
> other for the security of the resulting network connections
> - no support is "it doesn't build, but even if it did, we wouldn't
> vouch for the security of the resulting connections"

Sorry, my bad for just skimming the thread. There are comments like this:

> I'm tempted to require 1.0.2 for Python 3.6 but it's technically not
> necessary yet.
>
> #if OPENSSL_VERSION_INFO < 0x01000200L
> #  error "OpenSSL 1.0.2+ required"
> #endif

that led me to think that 3.6 was planning to demand 1.0.2, but if the
intention is build support for 1.0.1, that would work.

Sorry for the noise!

ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Supported versions of OpenSSL

[Nick Coghlan]

On 29 August 2016 at 21:05, Chris Angelico  wrote:
> On Mon, Aug 29, 2016 at 8:52 PM, Nick Coghlan  wrote:
>> For upcoming 3.6 I would like to limit support to 1.0.2+ and require
>> 1.0.2 features for 3.7.
>
> What does "limit support" mean? Will it be possible to build CPython
> 3.6 against OpenSSL 1.0.1?

Christian clarified this later in the thread:

- full support is stating confidently that software running that way
is using network connections that are as secure as we know how to make
them
- build support is ensuring it builds, without vouching one way or the
other for the security of the resulting network connections
- no support is "it doesn't build, but even if it did, we wouldn't
vouch for the security of the resulting connections"

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Supported versions of OpenSSL

[Chris Angelico]

On Mon, Aug 29, 2016 at 8:52 PM, Nick Coghlan  wrote:
> On 29 August 2016 at 19:14, Chris Angelico  wrote:
>> On Mon, Aug 29, 2016 at 6:24 PM, Christian Heimes  
>> wrote:
>>> No, LTS support should not be our concern. If you need a brand new
>>> version of Python on an old LTS or Enterprise version of your OS, please
>>> contact your vendor and buy support. You don't get to run old metal and
>>> play with shiny new toys at the same time for free.
>>
>> I think I agree with you, but I'm not fully convinced. The current
>> stable Debian (Jessie) also ships OpenSSL 1.0.1, although 1.0.2 is
>> available in backports. The next stable release (Stretch) is expected
>> some time 2017.
>
> I agree keeping build compatibility with the latest Debian stable
> release is a sensible baseline, but 3.7 won't land until some time in
> 2018, so Christian's proposed timeline still works on that front (but
> jumping straight to requiring 1.0.2 in 3.6 wouldn't, suggesting that
> supporting the latest Debian stable release will be a genuinely useful
> guideline in helping to make decisions about minimum supported OpenSSL
> versions for new CPython releases).

Cool. I may have slightly misunderstood this from the OP:

> For upcoming 3.6 I would like to limit support to 1.0.2+ and require
> 1.0.2 features for 3.7.

What does "limit support" mean? Will it be possible to build CPython
3.6 against OpenSSL 1.0.1?

For 3.7, sure. Once Stretch has been stable for a year, it's not
unreasonable to declare that the newest Python won't run on oldstable.
Just not so sure about 3.6, where Stretch won't be stable yet. But
even there, Jessie has been stable since Apr 2015 (so, 18 months or so
by the time Python 3.6 comes out - that's one entire Python release),
so I could well understand not wanting to be shackled by it. Saying
"the latest CPython demands that you enable backports in your apt
sources.list" wouldn't be unreasonable.

ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Supported versions of OpenSSL

[Nick Coghlan]

On 29 August 2016 at 19:14, Chris Angelico  wrote:
> On Mon, Aug 29, 2016 at 6:24 PM, Christian Heimes  
> wrote:
>> No, LTS support should not be our concern. If you need a brand new
>> version of Python on an old LTS or Enterprise version of your OS, please
>> contact your vendor and buy support. You don't get to run old metal and
>> play with shiny new toys at the same time for free.
>
> I think I agree with you, but I'm not fully convinced. The current
> stable Debian (Jessie) also ships OpenSSL 1.0.1, although 1.0.2 is
> available in backports. The next stable release (Stretch) is expected
> some time 2017.

I agree keeping build compatibility with the latest Debian stable
release is a sensible baseline, but 3.7 won't land until some time in
2018, so Christian's proposed timeline still works on that front (but
jumping straight to requiring 1.0.2 in 3.6 wouldn't, suggesting that
supporting the latest Debian stable release will be a genuinely useful
guideline in helping to make decisions about minimum supported OpenSSL
versions for new CPython releases).

Assuming the checks for required OpenSSL features are able to be
implemented as feature guards rather than as a plain version check
(e.g. by issuing a warning if the detected SSL version is too low, but
otherwise letting the build continue), other distros will also be able
to keep new CPython versions building just by backporting the required
OpenSSL features.

More generally, it's important for folks to keep in mind that where
most commercial Linux redistributors invest directly in upstream
maintenance by assigning people to work on the kernel and other low
level infrastructure pieces full time (including the community distros
where those components are most actively developed), Python's
commercial redistributors aren't as well behaved, so we currently have
just the one full time developer working on PyPI and related PyPA
tooling, and a few folks that have successfully negotiated part-time
upstream CPython involvement as a condition of their employment.

That disparity means we have to start getting much better at
offloading security-sensitive work from CPython's volunteers to paid
Linux developers in cases where it's feasible for us to do so, even if
that in turn means requiring that Linux distros (both community and
commercial) pay much closer attention to keeping their network
security libraries up to date if they want their users to be readily
able to build and run the latest Python feature releases.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Supported versions of OpenSSL

[Chris Angelico]

On Mon, Aug 29, 2016 at 6:24 PM, Christian Heimes  wrote:
> No, LTS support should not be our concern. If you need a brand new
> version of Python on an old LTS or Enterprise version of your OS, please
> contact your vendor and buy support. You don't get to run old metal and
> play with shiny new toys at the same time for free.

I think I agree with you, but I'm not fully convinced. The current
stable Debian (Jessie) also ships OpenSSL 1.0.1, although 1.0.2 is
available in backports. The next stable release (Stretch) is expected
some time 2017. That's not the same as "running an old LTS" - that's
running the latest non-dev release. Is it acceptable to tell people to
install a library from backports if they want to compile the latest
Python?

ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Supported versions of OpenSSL

[Stefan Krah]

On Mon, Aug 29, 2016 at 10:24:42AM +0200, Christian Heimes wrote:
> On 2016-08-29 10:09, M.-A. Lemburg wrote:
> > On 28.08.2016 22:40, Christian Heimes wrote:
> >> ...
> >> I like to reduce the maintenance burden and list of supported OpenSSL
> >> versions ASAP. OpenSSL has deprecated 0.9.8 and 1.0.0 last year. 1.0.1
> >> will reach EOL by the end of this year,
> >> https://www.openssl.org/policies/releasestrat.html . However OpenSSL
> >> 0.9.8 is still required for some platforms (OSX).
> >> ...
> >> For upcoming 3.6 I would like to limit support to 1.0.2+ and require
> >> 1.0.2 features for 3.7.
> >> ...
> > 
> > Hmm, that last part would mean that Python 3.7 will no longer compile
> > on e.g. Ubuntu 14.04 LTS which uses OpenSSL 1.0.1 as default version.
> > Since 14.04 LTS is supported until 2019, I think it would be better
> > to only start requiring 1.0.2 in Python 3.8.
> 
> No, LTS support should not be our concern. If you need a brand new
> version of Python on an old LTS or Enterprise version of your OS, please
> contact your vendor and buy support. You don't get to run old metal and
> play with shiny new toys at the same time for free.

Well, it's an additional annoyance for sure:

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:Ubuntu 14.04.4 LTS
Release:14.04
Codename:   trusty


I'm not in the habit of updating my OS constantly.

[Before this attracts some of the usual lectures, I know how to install
OpenSSL, thanks.]


Stefan Krah




___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Supported versions of OpenSSL

[Christian Heimes]

On 2016-08-29 10:09, M.-A. Lemburg wrote:
> On 28.08.2016 22:40, Christian Heimes wrote:
>> ...
>> I like to reduce the maintenance burden and list of supported OpenSSL
>> versions ASAP. OpenSSL has deprecated 0.9.8 and 1.0.0 last year. 1.0.1
>> will reach EOL by the end of this year,
>> https://www.openssl.org/policies/releasestrat.html . However OpenSSL
>> 0.9.8 is still required for some platforms (OSX).
>> ...
>> For upcoming 3.6 I would like to limit support to 1.0.2+ and require
>> 1.0.2 features for 3.7.
>> ...
> 
> Hmm, that last part would mean that Python 3.7 will no longer compile
> on e.g. Ubuntu 14.04 LTS which uses OpenSSL 1.0.1 as default version.
> Since 14.04 LTS is supported until 2019, I think it would be better
> to only start requiring 1.0.2 in Python 3.8.

No, LTS support should not be our concern. If you need a brand new
version of Python on an old LTS or Enterprise version of your OS, please
contact your vendor and buy support. You don't get to run old metal and
play with shiny new toys at the same time for free.

By the way I knew that something like this would come up from you. Thank
you that you satisfied my expectation. :p

> BTW: Are there any features in 1.0.2 that we need and would warrant
> dropping support for 1.0.1 earlier than Ubuntu 14.04 LTS ?

Yes, there are features I want to use, e.g. proper hostname
verification. Python's post-handshake verification is a hack and leads
to information disclosure.

Christian
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Supported versions of OpenSSL

[M.-A. Lemburg]

On 28.08.2016 22:40, Christian Heimes wrote:
> ...
> I like to reduce the maintenance burden and list of supported OpenSSL
> versions ASAP. OpenSSL has deprecated 0.9.8 and 1.0.0 last year. 1.0.1
> will reach EOL by the end of this year,
> https://www.openssl.org/policies/releasestrat.html . However OpenSSL
> 0.9.8 is still required for some platforms (OSX).
> ...
> For upcoming 3.6 I would like to limit support to 1.0.2+ and require
> 1.0.2 features for 3.7.
> ...

Hmm, that last part would mean that Python 3.7 will no longer compile
on e.g. Ubuntu 14.04 LTS which uses OpenSSL 1.0.1 as default version.
Since 14.04 LTS is supported until 2019, I think it would be better
to only start requiring 1.0.2 in Python 3.8.

BTW: Are there any features in 1.0.2 that we need and would warrant
dropping support for 1.0.1 earlier than Ubuntu 14.04 LTS ?

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Aug 29 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/


::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com