[Python-Dev] Deleting with setting C API functions

[Serhiy Storchaka]

Slots like PyTypeObject.tp_setattr, PySequenceMethods.sq_ass_item, 
PyMappingMethods.mp_ass_subscript are used not only for setting 
attribute/item value, but also for deleting attribute/item (if value is 
NULL). This is not documented and should be. [1]  Correspondingly public 
API functions like PyObject_SetAttr, PyObject_SetItem, 
PySequence_SetItem, PySequence_SetSlice, PyMapping_SetItemString can be 
used for deleting. But all these functions have special counterparts for 
deleting: PyObject_DelAttr etc.


The question is wherever deleting ability of Set-functions is 
intentional, should we document this or deprecate and then delete?


[1] http://bugs.python.org/issue25701

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] Daily reference leaks (e5e507a357a6): sum=103

[Brett Cannon]

Someone just added a leak to pickle.

On Tue, 24 Nov 2015 at 01:45  wrote:

> results for e5e507a357a6 on branch "default"
> 
>
> test_asyncio leaked [3, 0, 0] memory blocks, sum=3
> test_functools leaked [0, 2, 2] memory blocks, sum=4
> test_pickle leaked [20, 20, 20] references, sum=60
> test_pickle leaked [12, 12, 12] memory blocks, sum=36
>
>
> Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R',
> '3:3:/home/psf-users/antoine/refleaks/reflogUP9efi', '--timeout', '7200']
> ___
> Python-checkins mailing list
> python-check...@python.org
> https://mail.python.org/mailman/listinfo/python-checkins
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] Daily reference leaks (e5e507a357a6): sum=103

[Serhiy Storchaka]

On 24.11.15 21:31, Brett Cannon wrote:

Someone just added a leak to pickle.


It always was here. I just added tests that expose it.

http://bugs.python.org/issue25725


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Deleting with setting C API functions

[Guido van Rossum]

Ooooh, that's probably really old code. I guess for the slots the
reasoning is to save on slots. For the public functions, alas it will
be hard to know if anyone is depending on it, even if it's
undocumented. Perhaps add a deprecation warning to these if the value
is NULL for one release cycle?

On Tue, Nov 24, 2015 at 1:21 PM, Serhiy Storchaka  wrote:
> Slots like PyTypeObject.tp_setattr, PySequenceMethods.sq_ass_item,
> PyMappingMethods.mp_ass_subscript are used not only for setting
> attribute/item value, but also for deleting attribute/item (if value is
> NULL). This is not documented and should be. [1]  Correspondingly public API
> functions like PyObject_SetAttr, PyObject_SetItem, PySequence_SetItem,
> PySequence_SetSlice, PyMapping_SetItemString can be used for deleting. But
> all these functions have special counterparts for deleting: PyObject_DelAttr
> etc.
>
> The question is wherever deleting ability of Set-functions is intentional,
> should we document this or deprecate and then delete?
>
> [1] http://bugs.python.org/issue25701
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance)

[Nick Coghlan]

On 25 Nov 2015 05:42, "Toshio Kuratomi"  wrote:

> >
>
> Yeah, I think you are correct in your understanding of what actual
> changes to the python distrribution are being proposed for
> redistributors in the PEP.  Reading through the PEP again, I'm not
> sure if I'm correct in thinking that this only applies to
> backporting... it may be that the environment section of the PEP
> applies to any python-2 while the config file section only applies to
> backporting.  Nick, could you clarify?

Yep, the environment variable section is the part MAL persuaded me was a
good idea during the original discussion of this PEP, based on the
interests of eGenix's customers. It's a cross-platform suggestion for
distributors wanting to provide a smoother upgrade path, applicable to both
backports and rebasing on 2.7.9+. (I originally didn't like it, but became
amenable once it was pointed out that anyone with the ability to set
process environment variables can already alter OpenSSL's settings to use a
different CA cert file or directory)

The config file recommendation is specific to Linux distro backports, with
the aim of saying "IF you backport PEP 476 to an older Python 2.7 point
release, THEN you should do it like this for cross-distro consistency". As
the current discussion shows, I'm *personally* an advocate for backporting,
but I don't think the PEP itself should reflect that - the suggestions to
adjust the tone are good ones, so I'll update the text accordingly.

Cheers,
Nick.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Deleting with setting C API functions

[Nick Coghlan]

On 25 November 2015 at 07:33, Guido van Rossum  wrote:
> Ooooh, that's probably really old code. I guess for the slots the
> reasoning is to save on slots. For the public functions, alas it will
> be hard to know if anyone is depending on it, even if it's
> undocumented. Perhaps add a deprecation warning to these if the value
> is NULL for one release cycle?

I did a quick scan for "PyObject_SetAttr", and it turns out
PyObject_DelAttr is only a convenience macro for calling
PyObject_SetAttr with NULL as the value argument. bltinmodule.c and
ceval.c also both include direct calls to PyObject_SetAttr with
"(PyObject *)NULL" as the value argument.

Investigating some of the uses that passed a variable as the value
argument, one case is the weakref proxy implementation, which uses
PyObject_SetAttr on the underlying object in its implementation of the
setattr slot in the proxy.

So it looks to me like replicating the NULL-handling behaviour of the
slots in the public Set* APIs was intentional, and it's just the
documentation of that detail that was missed (since most folks
presumably use the Del* convenience APIs instead).

Regards,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance)

[Toshio Kuratomi]

On Tue, Nov 24, 2015 at 10:08 AM, Paul Moore  wrote:

> I'm not actually sure that it's the place of this PEP to even comment
> on what the long term answer for such environments should be (or
> indeed, any answer, long term or not). I've argued the position that
> in some organisations it feels like security don't actually understand
> the issues of carefully balancing secure operation against flexible
> development practices,

I agree with this.

> but conversely it's certainly true that in many
> organisations, they *have* weighed the various arguments and made an
> informed decision on how to set up their internal network. It's
> entirely possible that self-signed certificates are entirely the right
> decision for their circumstances. Why would a Python PEP be qualified
> to comment on that decision?

I don't quite agree with this but it's probably moot in the face of
the previous and certain cornercases.  Self-signed certs work just
fine with the backports to python-2.7.9+ but you have to add the ca to
the clients.  An organization that has weighed the arguments and made
an informed decision to use self-signed certs should either do this
(to prevent MITM) or they should switch to using http instead of https
(because MITM isn't a feasible attack here).  The cornercases come
into play because you don't always control all of the devices and
services on your network.  The site could evaluate and decide that
MITM isn't a threat to their switch's configuration interface but that
interface might be served over https using a certificate signed by
their network vendor who doesn't give out their ca certificate (simply
stated: your security team knows what they're doing but your vendor's
does not).

> In my opinion, we should take all of the value judgements out of this
> paragraph, and just state the facts. How about:
>
> """
> In order to provide additional flexibility to allow infrastructure
> administrators to provide the appropriate solution for their
> environment, this PEP offers a way for administrators to upgrade to
> later versions of the Python 2.7 series without being forced to update
> their existing security certificate management infrastructure as a
> prerequisite.
> """

Two notes:

* python-2.7.9+ doesn't give you flexibility in this regard so
organizations do have to update their certificate management
infrastructure.  The cornercase described above becomes something that
has to be addressed at the code level.  Environments that are simply
misconfigured have to be fixed.  So in that regard, a value judgement
does seem appropriate here.  the judgement is "Listen guys, this PEP
advises redistributors on how they might provide a migration path for
you but it *does not bandaid the problem indefinitely*.  So long term,
you have to change your practices or you'll be out in the cold when
your redistributor upgrades to python-2.7.9+"

* Your proposed text actually removes the fix that I was adding --
this version of the paragraph implies that if your environment is
compatible with the redistributors' python-2.7.8 (or less) then it
will also be compatible with the redistributors' python-2.7.9+.  That
is not true.  Whether or not we take out any value judgement as to the
user's present environment this paragraph needs to be fixed to make it
clear that this only affects redistributor's packages which have
backported pep 476 to python-2.7.8 or older.  Once the redistributor
updates to a newer python sites which relied on this crutch will
break.

-Toshio
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance)

[Paul Moore]

On 24 November 2015 at 17:16, Toshio Kuratomi  wrote:
> The long term answer for such environments is to update their internal
>   certificate management to at least match the standards set by the public
>   internet, but in the meantime, it is desirable to offer these administrators
>   a way to continue receiving maintenance updates to the Python 2.7 series,
>   without having to gate that on upgrades to their certificate management
>   infrastructure.
>
> + The wording here seems to reflect a different scope than merely
> backporting by distros.  Perhaps we should change it to: "[...]set by
> the public internet.  Distributions may wish to help these sites
> transition by backporting the PEP 476 changes to earlier versions of
> python in a way that does not require the administrators to upgrade
> their certificate management infrastructure immediately.  This would
> allow sites to choose to use the distribution suppiied python in a
> backwards compatible fashion until their certificate management
> infrastructure was updated and then toggle their site to utilize the
> more secure features provided by PEP 476."

I'm not actually sure that it's the place of this PEP to even comment
on what the long term answer for such environments should be (or
indeed, any answer, long term or not). I've argued the position that
in some organisations it feels like security don't actually understand
the issues of carefully balancing secure operation against flexible
development practices, but conversely it's certainly true that in many
organisations, they *have* weighed the various arguments and made an
informed decision on how to set up their internal network. It's
entirely possible that self-signed certificates are entirely the right
decision for their circumstances. Why would a Python PEP be qualified
to comment on that decision?

In my opinion, we should take all of the value judgements out of this
paragraph, and just state the facts. How about:

"""
In order to provide additional flexibility to allow infrastructure
administrators to provide the appropriate solution for their
environment, this PEP offers a way for administrators to upgrade to
later versions of the Python 2.7 series without being forced to update
their existing security certificate management infrastructure as a
prerequisite.
"""

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] collections.Counter __add__ implementation quirk

[Michael Selik]

Raymond,
I think you made a typographical error in your Counter.update example.

>>> from collections import Counter
>>> c = Counter(a=4, b=2, c=0, d=-2)
>>> d = Counter(a=1, b=-5, c=-2, d=6)
>>> c.update(d)
>>> c
Counter({'a': 5, 'd': 4, 'c': -2, 'b': -3})

Pair programming ;-)


On Tue, Nov 24, 2015 at 1:02 AM Raymond Hettinger <
raymond.hettin...@gmail.com> wrote:

>
> > On Nov 23, 2015, at 10:43 AM, Vlastimil Brom 
> wrote:
> >
> >> Is there any particular reason counters drop negative values when you
> add
> >> them together?  I definitely expected them to act like ints do when you
> add
> >> negatives, and had to subclass it to get what I think is the obvious
> >> behavior.
> >> ___
> >> Python-Dev mailing list
> > ...
> > Hi,
> > this is probably more appropriate for the general python list rathere
> > then this developers' maillist, however, as I asked a similar question
> > some time ago, I got some detailed explanations for the the current
> > design decissions from the original developer; cf.:
> > https://mail.python.org/pipermail/python-list/2010-March/570618.html
> >
> > (I didn't check possible changes in Counter since that version (3.1 at
> > that time).)
>
> In Python3.2, Counter grew a subtract() method:
>
> >>> c = Counter(a=4, b=2, c=0, d=-2)
> >>> d = Counter(a=1, b=2, c=3, d=4)
> >>> c.subtract(d)
> >>> c
> Counter({'a': 3, 'b': 0, 'c': -3, 'd': -6})
>
> The update() method has been around since the beginning:
>
> >>> from collections import Counter
> >>> c = Counter(a=4, b=2, c=0, d=-2)
> >>> d = Counter(a=1, b=-5, c=-2, d=6)
> >>> c.update(d)
> >>> d
> Counter({'d': 6, 'a': 1, 'c': -2, 'b': -5})
>
>
> So, you have two ways of doing counter math:
>
> 1. Normal integer arithmetic using update() and subtract() does straight
> addition and subtraction, either starting with or ending-up with negative
> values.
>
> 2. Saturating arithmetic using the operators: + - & | excludes
> non-positive results.  This supports bag-like behavior (c.f. smalltalk) and
> multiset operations (https://en.wikipedia.org/wiki/Multiset).
>
>
> Raymond
>
>
>
>
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/mike%40selik.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance)

[Toshio Kuratomi]

On Mon, Nov 23, 2015 at 5:59 PM, Barry Warsaw  wrote:

> I'm concerned about accepting PEP 493 making a strong recommendation to
> downstreams.  Yes, in an ideal world we all want security by default, but I
> think the backward compatibility concerns of the PEP are understated,
> especially as they relate to a maintenance release of a stable long term
> support version of the OS.  I don't want PEP 493 to be a cudgel that people
> beat us up with instead of having an honest discussion of the difficult
> trade-offs involved.
>
It sounds like the implementation sections of the PEP are acceptable
but that the PEP's general tone seems to assume that distributors are
champing at the bit to backport and that the recommendations here make
it so that backporting is a no-brainer -- which does not seem to
reflect the real-world?

I think the tone could be changed to address that as it doesn't seem
like forcing distros to backport is a real goal of the PEP.  The main
purposes of the PEP seem to be:

* Enumerate several ways that distributors can backport these 2.7.9
features to older releases
* Allow programmers to detect the presence of the features from their code
* Give end-users the ability to choose between backwards compatibility
and enhanced security

Here's some ideas for changing the tone:

  Abstract
  

  PEP 476 updated Python's default handling of HTTPS certificates to be
  appropriate for communication over the public internet. The Python 2.7 long
  term maintenance series was judged to be in scope for this change, with the
  new behaviour introduced in the Python 2.7.9 maintenance release.

+ Change to "PEP 476 updated Python's default handling of HTTPS
certificates to validate that the certs belonged to the server".  This
way we're saying what the change is rather than making a value
judgement of whether people who don't choose to backport are
"appropriate" or not.  Appropriate-ness is probably best left as an
argument in the text of PEP 476.

  This PEP provides recommendations to downstream redistributors wishing to
  provide a smoother migration experience when helping their users to manage
  this change in Python's default behaviour.

+ Change to "downstream redistributors wishing to backport the
enhancements in a way that allows users to choose between backwards
compatible behaviour or more secure certificate handling."  As barry
noted, this PEP doesn't change the amount of work needed to migrate.
It does, however, give users some choice in when they are going to
perform that work.  Additionally, this isn't simply about distributors
who want to make the transition smoother... (there's no downstreams
that want to make it "more painful" are there? ;-)  It's really about
making backporting of the enhancements less painful for users.

  Rationale
  =

  PEP 476 changed Python's default behaviour to better match the needs and
  expectations of developers operating over the public internet, a category
  which appears to include most new Python developers. It is the position of
  the authors of this PEP that this was a correct decision.

  However, it is also the case that this change *does* cause problems for
  infrastructure administrators operating private intranets that rely on
  self-signed certificates, or otherwise encounter problems with the new default
  certificate verification settings.

+ per barry's mesage, it would be good to either devote a paragraph to
the backwards compatibility implications here or link to
https://www.python.org/dev/peps/pep-0476/#backwards-compatibility

  The long term answer for such environments is to update their internal
  certificate management to at least match the standards set by the public
  internet, but in the meantime, it is desirable to offer these administrators
  a way to continue receiving maintenance updates to the Python 2.7 series,
  without having to gate that on upgrades to their certificate management
  infrastructure.

+ The wording here seems to reflect a different scope than merely
backporting by distros.  Perhaps we should change it to: "[...]set by
the public internet.  Distributions may wish to help these sites
transition by backporting the PEP 476 changes to earlier versions of
python in a way that does not require the administrators to upgrade
their certificate management infrastructure immediately.  This would
allow sites to choose to use the distribution suppiied python in a
backwards compatible fashion until their certificate management
infrastructure was updated and then toggle their site to utilize the
more secure features provided by PEP 476."

[...]

  These designs are being proposed as a recommendation for
redistributors, rather
  than as new upstream features, as they are needed purely to support legacy
  environments migrating from older versions of Python 2.7. Neither approach
  is being proposed as an upstream Python 2.7 feature, nor as a feature in any
  version of Python 3 (whether 

Re: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance)

[Paul Moore]

On 24 November 2015 at 18:37, Toshio Kuratomi  wrote:
> I don't quite agree with this but it's probably moot in the face of
> the previous and certain cornercases.  Self-signed certs work just
> fine with the backports to python-2.7.9+ but you have to add the ca to
> the clients.  An organization that has weighed the arguments and made
> an informed decision to use self-signed certs should either do this
> (to prevent MITM) or they should switch to using http instead of https
> (because MITM isn't a feasible attack here).

There's an assumption (which is untrue for the big organisations I'm
familiar with) that security are involved with the full stack -
typically in my experience "corporate security" set rules that are
filtered down through many layers, and development teams have no
influence on those decisions - conversely, the people setting security
policies aren't aware of the details of development day to day needs.

It's easy to say that users just click on the "allow" button in their
browser if they want to use self-signed certificates, but not have any
viable solution for automation code that doesn't use a browser - and
such automation code is "not corporately supported", so policy won't
get updated to address the disconnect.

> The cornercases come
> into play because you don't always control all of the devices and
> services on your network.  The site could evaluate and decide that
> MITM isn't a threat to their switch's configuration interface but that
> interface might be served over https using a certificate signed by
> their network vendor who doesn't give out their ca certificate (simply
> stated: your security team knows what they're doing but your vendor's
> does not).

This sounds like a similar situation to what I described above. I'm
not sure I'd see these as corner cases, though - they are pretty much
day to day business in my experience :-(

>> In my opinion, we should take all of the value judgements out of this
>> paragraph, and just state the facts. How about:
>>
>> """
>> In order to provide additional flexibility to allow infrastructure
>> administrators to provide the appropriate solution for their
>> environment, this PEP offers a way for administrators to upgrade to
>> later versions of the Python 2.7 series without being forced to update
>> their existing security certificate management infrastructure as a
>> prerequisite.
>> """
>
> Two notes:
>
> * python-2.7.9+ doesn't give you flexibility in this regard so
> organizations do have to update their certificate management
> infrastructure.  The cornercase described above becomes something that
> has to be addressed at the code level.  Environments that are simply
> misconfigured have to be fixed.  So in that regard, a value judgement
> does seem appropriate here.  the judgement is "Listen guys, this PEP
> advises redistributors on how they might provide a migration path for
> you but it *does not bandaid the problem indefinitely*.  So long term,
> you have to change your practices or you'll be out in the cold when
> your redistributor upgrades to python-2.7.9+"

Hmm, maybe I misread the PEP (I only skimmed it - as I say, Linux is
of limited interest to me). I thought that the environment variable
gave developers a "get out" clause. Maybe it's not what we want them
to do (for some value of "we") but isn't that the point of the PEP?

Admittedly if distributions don't *implement* that part of the PEP
(and I understand Red Hat haven't) then people are still stuck. But
"this PEP offers a way" is not incompatible with "your vendor didn't
implement the PEP so you're still stuck, sorry"...

> * Your proposed text actually removes the fix that I was adding --
> this version of the paragraph implies that if your environment is
> compatible with the redistributors' python-2.7.8 (or less) then it
> will also be compatible with the redistributors' python-2.7.9+.  That
> is not true.  Whether or not we take out any value judgement as to the
> user's present environment this paragraph needs to be fixed to make it
> clear that this only affects redistributor's packages which have
> backported pep 476 to python-2.7.8 or older.  Once the redistributor
> updates to a newer python sites which relied on this crutch will
> break.

Sorry for that. Certainly getting the facts right is crucial, and it
looks like my suggestion didn't do that. But hopefully someone can fix
it up (if people think it's a good way to go).

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance)

[Toshio Kuratomi]

On Tue, Nov 24, 2015 at 10:56 AM, Paul Moore  wrote:
> On 24 November 2015 at 18:37, Toshio Kuratomi  wrote:

>> The cornercases come
>> into play because you don't always control all of the devices and
>> services on your network.  The site could evaluate and decide that
>> MITM isn't a threat to their switch's configuration interface but that
>> interface might be served over https using a certificate signed by
>> their network vendor who doesn't give out their ca certificate (simply
>> stated: your security team knows what they're doing but your vendor's
>> does not).
>
> This sounds like a similar situation to what I described above. I'm
> not sure I'd see these as corner cases, though - they are pretty much
> day to day business in my experience :-(
>
It sounds like you're coming from a Windows background and I'm coming
from a Linux background which might be a small disconnect here -- we
do seem to be in agreement that what's "right to do" isn't always easy
or possible for the client to accomplish so I think we should probably
leave it at that.

>>> In my opinion, we should take all of the value judgements out of this
>>> paragraph, and just state the facts. How about:
>>>
>>> """
>>> In order to provide additional flexibility to allow infrastructure
>>> administrators to provide the appropriate solution for their
>>> environment, this PEP offers a way for administrators to upgrade to
>>> later versions of the Python 2.7 series without being forced to update
>>> their existing security certificate management infrastructure as a
>>> prerequisite.
>>> """
>>
>> Two notes:
>>
>> * python-2.7.9+ doesn't give you flexibility in this regard so
>> organizations do have to update their certificate management
>> infrastructure.  The cornercase described above becomes something that
>> has to be addressed at the code level.  Environments that are simply
>> misconfigured have to be fixed.  So in that regard, a value judgement
>> does seem appropriate here.  the judgement is "Listen guys, this PEP
>> advises redistributors on how they might provide a migration path for
>> you but it *does not bandaid the problem indefinitely*.  So long term,
>> you have to change your practices or you'll be out in the cold when
>> your redistributor upgrades to python-2.7.9+"
>
> Hmm, maybe I misread the PEP (I only skimmed it - as I say, Linux is
> of limited interest to me). I thought that the environment variable
> gave developers a "get out" clause. Maybe it's not what we want them
> to do (for some value of "we") but isn't that the point of the PEP?
>
> Admittedly if distributions don't *implement* that part of the PEP
> (and I understand Red Hat haven't) then people are still stuck. But
> "this PEP offers a way" is not incompatible with "your vendor didn't
> implement the PEP so you're still stuck, sorry"...
>

Yeah, I think you are correct in your understanding of what actual
changes to the python distrribution are being proposed for
redistributors in the PEP.  Reading through the PEP again, I'm not
sure if I'm correct in thinking that this only applies to
backporting... it may be that the environment section of the PEP
applies to any python-2 while the config file section only applies to
backporting.  Nick, could you clarify?

The PEP is clear that it doesn't apply to python-3 or cross-distro.
So that means that sites still can't rely on this long-term (but their
long term would extend to the lifetime of their vendor supporting
python2 rather than when their vendor updated to 2.7.9+) and also that
developers can't depend on this if they're developing portable code.

>> * Your proposed text actually removes the fix that I was adding --
>> this version of the paragraph implies that if your environment is
>> compatible with the redistributors' python-2.7.8 (or less) then it
>> will also be compatible with the redistributors' python-2.7.9+.  That
>> is not true.  Whether or not we take out any value judgement as to the
>> user's present environment this paragraph needs to be fixed to make it
>> clear that this only affects redistributor's packages which have
>> backported pep 476 to python-2.7.8 or older.  Once the redistributor
>> updates to a newer python sites which relied on this crutch will
>> break.
>
> Sorry for that. Certainly getting the facts right is crucial, and it
> looks like my suggestion didn't do that. But hopefully someone can fix
> it up (if people think it's a good way to go).

Could be that I'm wrong -- will wait for Nick to clarify before I
think about what could be done to make this wording better.

-Toshio
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance)

[Paul Moore]

On 24 November 2015 at 03:46, Nick Coghlan  wrote:
> I think there are three relevant categories here:
>
> 1. Folks who assume that "https" means the same thing in Python that
> it means in web browsers, and are currently experiencing a silent
> security failure
> 2. Folks who already know it doesn't, and are relying on that to keep
> their infrastructure working
> 3. Folks currently in group 2 who would like to improve their
> infrastructure to default to verifying certificates

I'm not directly interested in this PEP (any code I write that is
relevant will run on Windows) but there's another group:

4. People who use https because it's the published API, but are only
looking at internal systems (where the threat window is minimised) and
the internal infrastructure is out of their control but insecure.
Browser users and other apps have to go through pain to work
(accepting self-signed certs, repeatedly responding "Yes" to security
warnings). Python code written for internal use just works at the
moment. Maybe it "shouldn't", in some abstract sense, but it *does*
and that is important to users.

This group may know that they are taking risks, and may want to
improve the infrastructure, but they *can't*. And this change breaks
their apps (possibly in ways they can't fix easily - not all client
environments offer ways to add trust, there's a reason tools like wget
have "ignore certificate checks" flags even though everyone knows they
are a bad idea).

The push for "secure by default" is important and needed, but needs to
be handled in a more pragmatic way, recognising that Python is used in
situations like the above, where people know they aren't doing the
right thing, but have no better options. Shutting down their least bad
option on a point of principle isn't helpful.

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance)

[Paul Moore]

On 24 November 2015 at 13:20, Nick Coghlan  wrote:
> I believe you're referring mainly to the original PEP 476 change there. In
> the context of PEP 493, this is another group that would potentially benefit
> from the suggested "security downgrade" environment variable (if any
> redistributors decide to implement that - RHEL doesn't as yet), since it
> would provide a way to restore the old behaviour without changing their
> client code or monkeypatching the SSL module as described in PEP 476.

I'm actually referring to the fact that your classification didn't
seem to include people who have no control over their infrastructure
(except in class 1 which implies ignorance rather than
powerlessness...). PEP 493 is of benefit to such people, so there's
now downside in explicitly noting this.

My concern is that *because* people consistently forget about the
class of people who have to put up with bad infrastructure but can't
do anything about it, we risk promoting a sense of "security as the
enemy" - which is the direct opposite of what we're trying to do.

I have no interest or opinion regarding this PEP itself, but I would
like to see "people who have to put up with whatever infrastructure
they are dumped with, and use Python to ease that burden" recognised
as an important class of user. They are very under-represented in
discussions, as it's usually big business closed source and similar
environments that are in that situation.

Simply adding "people who have no control over their broken
infrastructure" with a note that this PEP helps them, would be
sufficient here (and actually helps the case for the PEP, so why not?
;-))

Apologies, this is a bit of a hobby horse of mine, I'll pipe down now.

Paul.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance)

[Laura Creighton]

In a message of Tue, 24 Nov 2015 14:05:53 +, Paul Moore writes:
>Simply adding "people who have no control over their broken
>infrastructure" with a note that this PEP helps them, would be
>sufficient here (and actually helps the case for the PEP, so why not?
>;-))

But does it help them?  Or does it increase the power of those who
hand out certificates and who are intensely security conscious over
those who would like to get some work done this afternoon?

Laura
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance)

[Nick Coghlan]

On 25 November 2015 at 00:27, Laura Creighton  wrote:
> In a message of Tue, 24 Nov 2015 14:05:53 +, Paul Moore writes:
>>Simply adding "people who have no control over their broken
>>infrastructure" with a note that this PEP helps them, would be
>>sufficient here (and actually helps the case for the PEP, so why not?
>>;-))
>
> But does it help them?  Or does it increase the power of those who
> hand out certificates and who are intensely security conscious over
> those who would like to get some work done this afternoon?

In situations where IT are still the "Department of No", rather than
focusing on facilitating folks in getting their work done, I think the
most likely outcome of the configuration file recommendation in PEP
493 is preservation of the status quo: admins simply won't change the
config setting, even if they deploy a version of Linux that adopts the
approach suggested in the PEP. If they do enable full certificate
verification (or upgrade the environments they provide to a version of
Python that has it enabled by default) without doing appropriate
compatibility testing first, then they're going to hit the
compatibility problems Paul is talking about.

The aspect of the PEP that has the potential to help in the case of
poor infrastructure management is providing the ability to globally
turn off certificate verification on a per-process basis. It's the
networking equivalent of monkeypatching - you know there are risks
with doing it, but also judge the near term benefits to outweigh those
longer term risks in your particular situation.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance)

[M.-A. Lemburg]

I think the PEP is a good step forward to compromise between
the crypto purists (use whatever technologies makes us more
secure even if it breaks things) and those who cannot upgrade
their Python 2.7 because of the PEP 476 change, since it causes their
applications to fail (e.g. because the embedded devices they want to
interface to only support self signed certs).

I would still find having built-in support for the recommendations
in the Python stdlib a better approach, but PEP 493 is good enough
in at least solving real problems people are having.

PS: Would be great to have a PyPI package which implements these
recommendations so that you can simply add it as dependency - and
then please for Python 3 as well, since people with embedded
devices will want to be able to use Python 3 as well ;-)


On 24.11.2015 15:27, Laura Creighton wrote:
> In a message of Tue, 24 Nov 2015 14:05:53 +, Paul Moore writes:
>> Simply adding "people who have no control over their broken
>> infrastructure" with a note that this PEP helps them, would be
>> sufficient here (and actually helps the case for the PEP, so why not?
>> ;-))
> 
> But does it help them?  Or does it increase the power of those who
> hand out certificates and who are intensely security conscious over
> those who would like to get some work done this afternoon?
> 
> Laura
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/mal%40egenix.com
> 

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Nov 24 2015)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/


::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Fwd: Request for pronouncement on PEP 493 (HTTPS verification backport guidance)

[Toshio Kuratomi]

On Nov 24, 2015 6:28 AM, "Laura Creighton"  wrote:
>
> In a message of Tue, 24 Nov 2015 14:05:53 +, Paul Moore writes:
> >Simply adding "people who have no control over their broken
> >infrastructure" with a note that this PEP helps them, would be
> >sufficient here (and actually helps the case for the PEP, so why not?
> >;-))
>
> But does it help them?  Or does it increase the power of those who
> hand out certificates and who are intensely security conscious over
> those who would like to get some work done this afternoon?
>
My reading is that it will help more people but lockdown environments can
still trump their users if they wish.

If a distribution wishes to give users of older python versions the option
of verifying certificates then they will need to backport changes
authorized by previous peps.  By themselves, those changes would make it so
environment owners and application authors are in complete control.  If an
application is coded to do cert verification and the remote end has
certificates that aren't recognized as valid on the client end then the
user would have to change the client application code to be able to use it
in their environment (or figure out how to get the ca for the remote end
into their local certificate store... in extreme cases, this might be
impossible - the ca cert has been lost or belongs to another company).

This pep tells distributions how they might give the client end a bit more
power when they backport.  The settings file allows the client to toggle
verification site wide.  The environment variable allows clients to toggle
it per application invocation.  Both of these situations are better for a
client than having the backport and nothing else.  Both of these can be
shut down by an environment owner with sufficient authority to limit what's
running on the client (not sure the scope of the environment owner's powers
here so I thought I should acknowledge this factor).

So basically: backporting other peps (to increase security) will subtract
power from the clients.  This pep specifies several facilities the
backporters can implement to give some of that power back to the clients.

-Toshio

>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance)

[Christian Heimes]

On 2015-11-24 01:18, Nick Coghlan wrote:
> On 24 November 2015 at 05:35, Christian Heimes  wrote:
>> On 2015-11-17 01:00, Guido van Rossum wrote:
>>> Hm, making Christian the BDFL-delegate would mean two out of three
>>> authors *and* the BDFL-delegate all working for Red Hat, which clearly
>>> has a stake (and IIUC has already committed to this approach ahead of
>>> PEP approval). SO then it would look like this is just rubber-stamping
>>> Red Hat's internal decision process (if it's a process -- sounds more
>>> like an accident :-).
>>>
>>> So, Alex, do you want to approve this PEP?
>>
>> I haven't read this thread until now. Independently from your objection
>> I have raised the same concern with Nick today. I'd be willing to BDFL
>> the PEP but I'd rather have somebody outside of Red Hat.
> 
> Likewise, but the intersection between "wants to get PEP 476 into the
> hands of as many system operators as possible as soon as possible",
> "is a CPython core developer", and "doesn't work for Red Hat" is
> looking to be a rather select group :)

Right, with Antoine and Alex out of scope and you, Victor and me working
for Red Hat, the air is getting thin. Benjamin is familiar with the ssl
module. Or we can follow Alex's advice and ask somebody from the PyCA
group (Donald, Paul, lvh) or requests (Cory) to get some outside
perspective.


> Since we already know Red Hat are OK with the draft recommendations,
> and I missed the RHEL 7.2 release date anyway, perhaps Barry or
> Matthias might be interested in tilting at the Ubuntu 14.04 LTS stable
> release update windmill? I know there was previously a decision from
> Ubuntu Security not to backport PEPs 466 & 476 to 2.7.5 due to the
> stability risks [1], but the configuration file based approach
> recommended in PEP 493 is backwards compatible by default, with the
> decision to opt in to the improved settings after upgrading current
> systems being made by system administrators rather than the distro
> vendor. With around 3 1/2 years still to run on 14.04's support
> lifecycle, that has the potential to reach quite a few systems that
> otherwise wouldn't benefit from the change until well after Ubuntu
> 16.04 is released next year.

Yes, that makes a lot of sense.

Christian
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance)

[Donald Stufft]

> On Nov 24, 2015, at 7:53 AM, Christian Heimes  wrote:
> 
> On 2015-11-24 01:18, Nick Coghlan wrote:
>> On 24 November 2015 at 05:35, Christian Heimes  wrote:
>>> On 2015-11-17 01:00, Guido van Rossum wrote:
 Hm, making Christian the BDFL-delegate would mean two out of three
 authors *and* the BDFL-delegate all working for Red Hat, which clearly
 has a stake (and IIUC has already committed to this approach ahead of
 PEP approval). SO then it would look like this is just rubber-stamping
 Red Hat's internal decision process (if it's a process -- sounds more
 like an accident :-).
 
 So, Alex, do you want to approve this PEP?
>>> 
>>> I haven't read this thread until now. Independently from your objection
>>> I have raised the same concern with Nick today. I'd be willing to BDFL
>>> the PEP but I'd rather have somebody outside of Red Hat.
>> 
>> Likewise, but the intersection between "wants to get PEP 476 into the
>> hands of as many system operators as possible as soon as possible",
>> "is a CPython core developer", and "doesn't work for Red Hat" is
>> looking to be a rather select group :)
> 
> Right, with Antoine and Alex out of scope and you, Victor and me working
> for Red Hat, the air is getting thin. Benjamin is familiar with the ssl
> module. Or we can follow Alex's advice and ask somebody from the PyCA
> group (Donald, Paul, lvh) or requests (Cory) to get some outside
> perspective.
> 

Under normal circumstances I'd probably be willing to do it even though I have
doubts about the value of it. However, I'm less than two weeks away from
closing on a house and will be moving into it after that, so my time is very
limited right now.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance)

[Nick Coghlan]

On 24 Nov 2015 8:12 pm, "Paul Moore"  wrote:
>
> On 24 November 2015 at 03:46, Nick Coghlan  wrote:
> > I think there are three relevant categories here:
> >
> > 1. Folks who assume that "https" means the same thing in Python that
> > it means in web browsers, and are currently experiencing a silent
> > security failure
> > 2. Folks who already know it doesn't, and are relying on that to keep
> > their infrastructure working
> > 3. Folks currently in group 2 who would like to improve their
> > infrastructure to default to verifying certificates
>
> I'm not directly interested in this PEP (any code I write that is
> relevant will run on Windows) but there's another group:
>
> 4. People who use https because it's the published API, but are only
> looking at internal systems (where the threat window is minimised) and
> the internal infrastructure is out of their control but insecure.
> Browser users and other apps have to go through pain to work
> (accepting self-signed certs, repeatedly responding "Yes" to security
> warnings). Python code written for internal use just works at the
> moment. Maybe it "shouldn't", in some abstract sense, but it *does*
> and that is important to users.
>
> This group may know that they are taking risks, and may want to
> improve the infrastructure, but they *can't*. And this change breaks
> their apps (possibly in ways they can't fix easily - not all client
> environments offer ways to add trust, there's a reason tools like wget
> have "ignore certificate checks" flags even though everyone knows they
> are a bad idea).

I believe you're referring mainly to the original PEP 476 change there. In
the context of PEP 493, this is another group that would potentially
benefit from the suggested "security downgrade" environment variable (if
any redistributors decide to implement that - RHEL doesn't as yet), since
it would provide a way to restore the old behaviour without changing their
client code or monkeypatching the SSL module as described in PEP 476.

Regards,
Nick.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance)

[Christian Heimes]

On 2015-11-24 00:47, Nick Coghlan wrote:
> Updated version of the PEP posted: https://hg.python.org/peps/rev/8decac213ebf
> 
> On 24 November 2015 at 05:35, Christian Heimes  wrote:
>> 1) The example implementation of the function doesn't check the
>> sys.flags.ignore_environment. Internally CPython has specialized getenv
>> function that ignores env vars with PYTHON prefix when the flag is set.
>> PYTHON* env vars aren't removed from os.environ. Modules have to check
>> the flag.
> 
> I guarded the relevant sections in the examples with "if not
> sys.flags.ignore_environment:"

Thanks!


>> 2) The PEP is rather Linux-centric. What's the recommended path to the
>> config file on other platforms like BDS (/usr/local/etc/ is preferred
>> for additional dependencies on FreeBSD), OSX and Windows?
> 
> The environment variable section should be generic, and the
> configuration file section is largely specific to vendors offering
> long term commercial support options for Linux distros.
> 
> There was already a qualification in the "Recommended file location"
> section limiting the scope to *nix systems, so rather than trying to
> cover non-Linux systems, I've instead updated that qualification to
> limit the recommendation even further to specifically Linux systems.
> If *BSD folks pipe up saying they'd like to be included, then adding
> another filename recommendation wouldn't be difficult.
>> 3) What's the interaction between the location of the config file and
>> virtual envs? Would it make sense to search for the file in a venv's
>> etc/ first and then dispatch to global /etc/? That way venvs can
>> influence the setting, too.
> 
> This is a system level configuration setting to preserve backwards
> compatible behaviour in the system Python, so scoping it at the
> interpreter level was a deliberate design decision. However, I added a
> new section to both recommendations describing the lack of interaction
> with virtual environments.

I mentioned 2) and 3) because I suspect that some people will want to
use the new setting to configure applications. You know how people
think. They get a new shiny tool and then they abuse it as a hammer to
drive in all their problems. :)


>> 4) It makes sense to make the cert-verification.cfg file future-proof
>> and reserve it for other cert-related configuration in the future. For
>> example it could be used to define new contexts, set protocols, ciphers
>> or hashes for cert pinning. It should be enough to say that CPython
>> reserves the right to add more sections and keys later.
> 
> For future releases, I think a different filename makes sense, as we
> don't really want a global "turn off HTTPS certificate verification"
> flag. Perhaps something like "/etc/python/network-security.cfg", for
> example.

Are you planning to remove the "disable verification flag" in the future?


>> 5) I'm not particular fond of the section name [https]. For one It is
>> ambiguous because it doesn't distinguish between server certs and client
>> certs.
> 
> I added a note to clarify that the section name comes from the "https"
> URL schema passed to the relevant client APIs.
> 
>> It's also not correct. The default context is used for other
>> protocols like imap, smtp etc. over TLS.
> 
> That changed in PEP 476 - there's a separate private context for HTTPS
> standard library clients now, which allows HTTPS to verify hostnames
> by default, while other clients are still permissive. This is noted in
> PEP 476: https://www.python.org/dev/peps/pep-0476/#other-protocols
> 
> The reason for this is that browsers have provided a pretty good
> forcing function in getting folks to use properly signed certificates,
> at least on the public internet, but self-signed and improperly signed
> certificates are still significantly more common for other protocols.

I'm sorry, I forgot about PEP 476 and the separate context function for
HTTPS. Somehow I expected the setting to influence all TLS/SSL
connection, not just HTTPS. I should have read the topic line.

In the light of the new information the setting name makes sense.

Christian

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance)

[Paul Moore]

On 24 November 2015 at 14:27, Laura Creighton  wrote:
> In a message of Tue, 24 Nov 2015 14:05:53 +, Paul Moore writes:
>>Simply adding "people who have no control over their broken
>>infrastructure" with a note that this PEP helps them, would be
>>sufficient here (and actually helps the case for the PEP, so why not?
>>;-))
>
> But does it help them?  Or does it increase the power of those who
> hand out certificates and who are intensely security conscious over
> those who would like to get some work done this afternoon?

My reading is that if fully implemented (and Nick has already
confirmed that Red Hat didn't do this) it would add an environment
variable that would allow the user to (in effect) say "I can't fix my
security infrastructure, so just leave me alone and let me take the
risk".

So in theory this PEP would give back some of the ability to ignore
the problem that previous PEPs took away. (And by "ignore the
problem", here I mean "just try to get some work done in spite of a
security and infrastructure group that don't understand how to
implement security and infrastructure, dammit!")

Like it or not, in many organisations, security and development are a
huge "us and them" battle. For me, it's important that Python doesn't
take sides in that battle, while still offering education to anyone
willing to listen. (All I've learned about security is as a result of
working with Python - sadly, that knowledge has not made my job one
iota easier, it's just increased my stress levels :-()

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com