Re: [Python-Dev] Avoid formatting an error message on attribute error
2013/11/7 Steven D'Aprano : > My initial instinct here was to say that sounded like premature > optimization, but to my surprise the overhead of generating the error > message is actually significant -- at least from pure Python 3.3 code. I ran a quick and dirty benchmark by replacing the error message with None. Original: $ ./python -m timeit 'hasattr(1, "y")' 100 loops, best of 3: 0.354 usec per loop $ ./python -m timeit -s 's=1' 'try: s.y' 'except AttributeError: pass' 100 loops, best of 3: 0.471 usec per loop Patched: $ ./python -m timeit 'hasattr(1, "y")' 1000 loops, best of 3: 0.106 usec per loop $ ./python -m timeit -s 's=1' 'try: s.y' 'except AttributeError: pass' 1000 loops, best of 3: 0.191 usec per loop hasattr() is 3.3x faster and try/except is 2.4x faster on such micro benchmark. > Given that, I wonder whether it would be worth coming up with a more > general solution to the question of lazily generating error messages > rather than changing AttributeError specifically. My first question is about keeping strong references to objects (type object for AttributeError). Is it an issue? If it is an issue, it's maybe better to not modify the code :-) Yes, the lazy formatting idea can be applied to various other exceptions. For example, TypeError message is usually build using PyErr_Format() to mention the name of the invalid type. Example: PyErr_Format(PyExc_TypeError, "exec() arg 2 must be a dict, not %.100s", globals->ob_type->tp_name); But it's not easy to store arbitary C types for PyUnicode_FromFormat() parameters. Argument types can be char*, Py_ssize_t, PyObject*, int, etc. I proposed to modify (first/only) AttributeError, because it is more common to ignore the AttributeError than other errors like TypeError. (TypeError or UnicodeDecodeError are usually unexpected.) >> It would be nice to only format the message on demand. The >> AttributeError would keep a reference to the type. > > Since only the type name is used, why keep a reference to the type > instead of just type.__name__? In the C language, type.__name__ does not exist, it's a char* object. If the type object is destroyed, type->tp_name becomes an invalid pointer. So AttributeError should keep a reference on the type object. >> AttributeError.args would be (type, attr) instead of (message,). >> ImportError was also modified to add a new "name "attribute". > > I don't like changing the signature of AttributeError. I've got code > that raises AttributeError explicitly. The constructor may support different signature for backward compatibility: AttributeError(message: str) and AttributeError(type: type, attr: str). I'm asking if anyone relies on AttributeError.args attribute. Victor ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Avoid formatting an error message on attribute error
07.11.13 00:32, Victor Stinner написав(ла): I'm trying to avoid unnecessary temporary Unicode strings when possible (see issue #19512). While working on this, I saw that Python likes preparing an user friendly message to explain why getting an attribute failed. The problem is that in most cases, the caller doesn't care of the message: the exception is simply deleted. For example, hasattr() deletes immediatly the AttributeError. It would be nice to only format the message on demand. The AttributeError would keep a reference to the type. Keeping a strong reference to the type might change the behaviour of some applications in some corner cases. (Holding a reference to the object would be worse, and the type looks to be preferred over the type to format the error message.) See also: http://bugs.python.org/issue18156 : Add an 'attr' attribute to AttributeError http://bugs.python.org/issue18162 : Add index attribute to IndexError http://bugs.python.org/issue18163 : Add a 'key' attribute to KeyError ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue 19332: Guard against changing dict during iteration
On 7 Nov 2013 03:18, "Antoine Pitrou" wrote: > > Le 06/11/2013 06:41, Nick Coghlan a écrit : > >> >> The behaviour of mutating builtin containers while iterating over them >> is formally undefined beyond "it won't segfault" (one of the few such >> undefined behaviours in Python). The associated exceptions are thus >> strictly "best effort given other constraints". > > > Not sure what you mean with "formally undefined". For example, you can perfectly well change a list's contents while iterating over it, and I bet there's a lot of code that relies on that, as in:: > > for i, value in enumerate(mylist): > if some_condition(value): > mylist[i] = some_function(value) > > If you change "builtin containers" to "builtin unordered containers", then you probably are closer to the truth :) I meant changing the length rather than just the contents of an existing entry. Even lists get a little odd if you add and remove entries while iterating (although I agree the sequence case is much better defined than the unordered container case). Cheers, Nick. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Avoid formatting an error message on attribute error
On 7 Nov 2013 21:34, "Victor Stinner" wrote: > > 2013/11/7 Steven D'Aprano : > > My initial instinct here was to say that sounded like premature > > optimization, but to my surprise the overhead of generating the error > > message is actually significant -- at least from pure Python 3.3 code. > > I ran a quick and dirty benchmark by replacing the error message with None. > > Original: > > $ ./python -m timeit 'hasattr(1, "y")' > 100 loops, best of 3: 0.354 usec per loop > $ ./python -m timeit -s 's=1' 'try: s.y' 'except AttributeError: pass' > 100 loops, best of 3: 0.471 usec per loop > > Patched: > > $ ./python -m timeit 'hasattr(1, "y")' > 1000 loops, best of 3: 0.106 usec per loop > $ ./python -m timeit -s 's=1' 'try: s.y' 'except AttributeError: pass' > 1000 loops, best of 3: 0.191 usec per loop > > hasattr() is 3.3x faster and try/except is 2.4x faster on such micro benchmark. > > > Given that, I wonder whether it would be worth coming up with a more > > general solution to the question of lazily generating error messages > > rather than changing AttributeError specifically. > > My first question is about keeping strong references to objects (type > object for AttributeError). Is it an issue? If it is an issue, it's > maybe better to not modify the code :-) > > > Yes, the lazy formatting idea can be applied to various other > exceptions. For example, TypeError message is usually build using > PyErr_Format() to mention the name of the invalid type. Example: > > PyErr_Format(PyExc_TypeError, "exec() arg 2 must be a dict, not %.100s", > globals->ob_type->tp_name); > > But it's not easy to store arbitary C types for PyUnicode_FromFormat() > parameters. Argument types can be char*, Py_ssize_t, PyObject*, int, > etc. > > I proposed to modify (first/only) AttributeError, because it is more > common to ignore the AttributeError than other errors like TypeError. > (TypeError or UnicodeDecodeError are usually unexpected.) > > >> It would be nice to only format the message on demand. The > >> AttributeError would keep a reference to the type. > > > > Since only the type name is used, why keep a reference to the type > > instead of just type.__name__? > > In the C language, type.__name__ does not exist, it's a char* object. > If the type object is destroyed, type->tp_name becomes an invalid > pointer. So AttributeError should keep a reference on the type object. > > >> AttributeError.args would be (type, attr) instead of (message,). > >> ImportError was also modified to add a new "name "attribute". The existing signature continued to be supported, though. > > > > I don't like changing the signature of AttributeError. I've got code > > that raises AttributeError explicitly. > > The constructor may support different signature for backward > compatibility: AttributeError(message: str) and AttributeError(type: > type, attr: str). > > I'm asking if anyone relies on AttributeError.args attribute. The bigger problem is you can't change the constructor signature in a backwards incompatible way. You would need a new class method as an alternative constructor instead, or else use optional parameters. Cheers, Nick. > > Victor > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP 384 (stable api) question
PEP 384 describes the stable Python api, available when Py_LIMITED_API is defined. However, there are some (small) changes in the function prototypes available, one example is (in Python 3.3): PyObject* PyObject_CallFunction(PyObject *callable, char *format, ...) which changed in Python 3.4 to 'const char *format' for the third argument. I know that this is only a subtle difference, but in my case it gives compiler warnings when I compile my stuff (my stuff is a little bit special, I have to admit, but anyway). I thought that the stable API would keep exactly the same across releases - is this expectation wrong or is this a bug? Thanks, Thomas ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 384 (stable api) question
On 7 November 2013 12:44, Thomas Heller wrote: > PEP 384 describes the stable Python api, available when > Py_LIMITED_API is defined. > > However, there are some (small) changes in the function prototypes > available, one example is (in Python 3.3): > PyObject* PyObject_CallFunction(PyObject *callable, char *format, ...) > > which changed in Python 3.4 to 'const char *format' for the > third argument. > > I know that this is only a subtle difference, but in my case it gives > compiler warnings when I compile my stuff (my stuff is a little bit > special, I have to admit, but anyway). > > I thought that the stable API would keep exactly the same across > releases - is this expectation wrong or is this a bug? I think you're confusing API with ABI. PEP 384 describes a stable ABI. This means that an extension module can be *binary* compatible with several different Python versions without needing to be recompiled for each particular CPython release. I don't think the change you refer to breaks binary compatibility. Oscar ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Simplify and unify SSL verification
Hi, I'd like to simplify, unify and tighten SSL handling and verification code in Python 3.5. Therefore I propose to deprecate some features for Python 3.4. SSLContext objects are going to be the central instance for configuration. In order to archive the goal I propose to - deprecate the key_file, cert_file and check_hostname arguments in various in favor of context. Python 3.5 will no longer have these arguments in http.client, smtplib etc. - deprecate implicit verify_mode=CERT_NONE. Python 3.5 will default to CERT_REQUIRED. - enforce hostname matching in CERT_OPTIONAL or CERT_REQUIRED mode by default in Python 3.5. - add ssl.get_default_context() to acquire a default SSLContext object if the context argument is None. (Python 3.4) - add check_cert option to SSLContext and a check_cert() function to SSLSocket in order to unify certificate validation and hostname matching. (Python 3.4) The SSLContext's check_cert option will support four values: None (default) use match_hostname() if verify_mode is CERT_OPTIONAL or CERT_REQUIRED True always use match_hostname() False never use match_hostname() callable function: call func(sslsock, hostname, initiator, **kwargs) The initiator argument is the object that has initiated the SSL connection, e.g. http.client.HTTPSConnection object. sslsock.context points to its SSLContext object A connect method may look like this: def connect(self): hostname = self.hostname s = socket.create_connection((hostname, self.port)) sock = self.context.wrap_socket(sock, server_hostname=hostname) sock.check_cert(hostname, initiator=self) return sock The check_cert option for SSLContext makes it easy to customize cert handling. Application developers just need to configure a SSLContext object in order to customize SSL cert verification and matching. In Python 3.4 it will be possible to get the full peer chain, cert information in CERT_NONE verification mode and SPKI data for SSL pinning. A custom check_cert callback can be used to validate self-signed certs or test for pinned certificates. The proposal does NOT address the CA certificates difficulty. It's a different story that needs to be addressed for all platforms separately. Most Linux distributions, (Net|Free|Open) BSD and Mac Ports come with SSL certs anyway, although some do not correctly handle the certs' purposes. I have working code for Windows' cert system store that will land in 3.4. Native Mac OS X hasn't been addressed yet. AIX, HP-UX, Solaris etc. don't come with CA certs. Christian ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 384 (stable api) question
Am 07.11.13 13:44, schrieb Thomas Heller: > I thought that the stable API would keep exactly the same across > releases - is this expectation wrong or is this a bug? Oscar is right - this change doesn't affect the ABI, just the API. That said, please file an issue reporting what change you see in your compiler warnings. 3.4 is not released yet, perhaps there is something that can be done. Regards, Martin ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Avoid formatting an error message on attribute error
On Thu, Nov 7, 2013 at 7:41 AM, Nick Coghlan wrote: > > On 7 Nov 2013 21:34, "Victor Stinner" wrote: > > > > 2013/11/7 Steven D'Aprano : > > > My initial instinct here was to say that sounded like premature > > > optimization, but to my surprise the overhead of generating the error > > > message is actually significant -- at least from pure Python 3.3 code. > > > > I ran a quick and dirty benchmark by replacing the error message with > None. > > > > Original: > > > > $ ./python -m timeit 'hasattr(1, "y")' > > 100 loops, best of 3: 0.354 usec per loop > > $ ./python -m timeit -s 's=1' 'try: s.y' 'except AttributeError: pass' > > 100 loops, best of 3: 0.471 usec per loop > > > > Patched: > > > > $ ./python -m timeit 'hasattr(1, "y")' > > 1000 loops, best of 3: 0.106 usec per loop > > $ ./python -m timeit -s 's=1' 'try: s.y' 'except AttributeError: pass' > > 1000 loops, best of 3: 0.191 usec per loop > > > > hasattr() is 3.3x faster and try/except is 2.4x faster on such micro > benchmark. > > > > > Given that, I wonder whether it would be worth coming up with a more > > > general solution to the question of lazily generating error messages > > > rather than changing AttributeError specifically. > > > > My first question is about keeping strong references to objects (type > > object for AttributeError). Is it an issue? If it is an issue, it's > > maybe better to not modify the code :-) > > > > > > Yes, the lazy formatting idea can be applied to various other > > exceptions. For example, TypeError message is usually build using > > PyErr_Format() to mention the name of the invalid type. Example: > > > > PyErr_Format(PyExc_TypeError, "exec() arg 2 must be a dict, not > %.100s", > > globals->ob_type->tp_name); > > > > But it's not easy to store arbitary C types for PyUnicode_FromFormat() > > parameters. Argument types can be char*, Py_ssize_t, PyObject*, int, > > etc. > > > > I proposed to modify (first/only) AttributeError, because it is more > > common to ignore the AttributeError than other errors like TypeError. > > (TypeError or UnicodeDecodeError are usually unexpected.) > > > > >> It would be nice to only format the message on demand. The > > >> AttributeError would keep a reference to the type. > > > > > > Since only the type name is used, why keep a reference to the type > > > instead of just type.__name__? > > > > In the C language, type.__name__ does not exist, it's a char* object. > > If the type object is destroyed, type->tp_name becomes an invalid > > pointer. So AttributeError should keep a reference on the type object. > > > > >> AttributeError.args would be (type, attr) instead of (message,). > > >> ImportError was also modified to add a new "name "attribute". > > The existing signature continued to be supported, though. > > > > > > > I don't like changing the signature of AttributeError. I've got code > > > that raises AttributeError explicitly. > > > > The constructor may support different signature for backward > > compatibility: AttributeError(message: str) and AttributeError(type: > > type, attr: str). > > > > I'm asking if anyone relies on AttributeError.args attribute. > > The bigger problem is you can't change the constructor signature in a > backwards incompatible way. You would need a new class method as an > alternative constructor instead, or else use optional parameters. > The optional parameter approach is the one ImportError took for introducing its `name` and `path` attributes, so there is precedent. Currently it doesn't use a default exception message for backwards-compatibility, but adding one wouldn't be difficult technically. In the case of AttributeError, what you could do is follow the suggestion I made in http://bugs.python.org/issue18156 and add an `attr` keyword-only argument and correpsonding attribute to AttributeError. If you then add whatever other keyword arguments are needed to generate a good error message (`object` so you have the target, or at least get the string name to prevent gc issues for large objects?) you could construct the message lazily in the __str__ method very easily while also doing away with inconsistencies in the message which should make doctest users happy. Lazy message creation through __str__ does leave the message out of `args`, though. In a perfect world (Python 4 maybe?) BaseException would take a single argument which would be an optional message, `args` wouldn't exist, and people called `str(exc)` to get the message for the exception. That would allow subclasses to expand the API with keyword-only arguments to carry extra info and have reasonable default messages that were built on-demand when __str__ was called. It would also keep `args` from just being a dumping ground of stuff that has no structure except by calling convention (which is not how to do an API; explicit > implicit and all). IOW the original dream of PEP 352 ( http://python.org/dev/peps/pep-0352/#retracted-ideas). ___
Re: [Python-Dev] Avoid formatting an error message on attribute error
On Thu, Nov 7, 2013 at 11:44 AM, Brett Cannon wrote: > Lazy message creation through > __str__ does leave the message out of `args`, though. If that's an issue you could make args a (settable) property that dynmically returns str(self) if appropriate: @property def args(self): actual = super().args if actual or self.name is None: return actual return (str(self),) > > In a perfect world (Python 4 maybe?) BaseException would take a single > argument which would be an optional message, `args` wouldn't exist, and > people called `str(exc)` to get the message for the exception. That would > allow subclasses to expand the API with keyword-only arguments to carry > extra info and have reasonable default messages that were built on-demand > when __str__ was called. It would also keep `args` from just being a dumping > ground of stuff that has no structure except by calling convention (which is > not how to do an API; explicit > implicit and all). IOW the original dream > of PEP 352 (http://python.org/dev/peps/pep-0352/#retracted-ideas). This reminds me that I need to revisit that idea of reimplementing all the builtin exceptions in pure Python. :) -eric ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Avoid formatting an error message on attribute error
On Thu, Nov 7, 2013 at 2:20 PM, Eric Snow wrote: > On Thu, Nov 7, 2013 at 11:44 AM, Brett Cannon wrote: > > Lazy message creation through > > __str__ does leave the message out of `args`, though. > > If that's an issue you could make args a (settable) property that > dynmically returns str(self) if appropriate: > > @property > def args(self): > actual = super().args > if actual or self.name is None: > return actual > return (str(self),) > Good point. That would solve that backwards-compatibility issue and allow the raising of DeprecationWarning. > > > > > In a perfect world (Python 4 maybe?) BaseException would take a single > > argument which would be an optional message, `args` wouldn't exist, and > > people called `str(exc)` to get the message for the exception. That would > > allow subclasses to expand the API with keyword-only arguments to carry > > extra info and have reasonable default messages that were built on-demand > > when __str__ was called. It would also keep `args` from just being a > dumping > > ground of stuff that has no structure except by calling convention > (which is > > not how to do an API; explicit > implicit and all). IOW the original > dream > > of PEP 352 (http://python.org/dev/peps/pep-0352/#retracted-ideas). > > This reminds me that I need to revisit that idea of reimplementing all > the builtin exceptions in pure Python. :) > Ah, that idea. =) Definitely a question of performance. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Simplify and unify SSL verification
Le 07/11/2013 19:13, Christian Heimes a écrit : Hi, I'd like to simplify, unify and tighten SSL handling and verification code in Python 3.5. Therefore I propose to deprecate some features for Python 3.4. SSLContext objects are going to be the central instance for configuration. In order to archive the goal I propose to - deprecate the key_file, cert_file and check_hostname arguments in various in favor of context. Python 3.5 will no longer have these arguments in http.client, smtplib etc. I'm in favour but I think 3.5 is too early. Keep in mind SSLContext is quite young. - deprecate implicit verify_mode=CERT_NONE. Python 3.5 will default to CERT_REQUIRED. -0.9. This breaks compatibility and doesn't achieve anything, since there's no reliable story for CA certs. - enforce hostname matching in CERT_OPTIONAL or CERT_REQUIRED mode by default in Python 3.5. Service matching is an application protocol level thing, it isn't strictly part of SSL. I'd feel warmer if you removed the "by default" part. - add ssl.get_default_context() to acquire a default SSLContext object if the context argument is None. (Python 3.4) I think I'm against it, for two reasons: 1. It will in the long term create compatibility and/or security issues (we'll have to keep fragile legacy settings if we want to avoid breaking existing uses of the default context) 2. SSLContexts are mutable, which means some code can affect other code's behaviour without even knowing. It's a recipe for subtle bugs and security issues. Applications and/or higher-level libraries should be smart enough to choose their own security preferences, and it's the better place to do so anyway. - add check_cert option to SSLContext and a check_cert() function to SSLSocket in order to unify certificate validation and hostname matching. (Python 3.4) The check_cert() function wouldn't achieve anything besides calling match_hostname(), right? I'm against calling it check_cert(), it's too generic and only induces confusion. The SSLContext's check_cert option will support four values: None (default) use match_hostname() if verify_mode is CERT_OPTIONAL or CERT_REQUIRED True always use match_hostname() False never use match_hostname() What is the hostname that the cert is matched against? callable function: call func(sslsock, hostname, initiator, **kwargs) The initiator argument is the object that has initiated the SSL connection, e.g. http.client.HTTPSConnection object. sslsock.context points to its SSLContext object Where does the "hostname" come from? And why is there an "initiator" object? Personally I prefer to avoid passing opaque user data, since the caller can use a closure anyway. And what are the **kwargs? A connect method may look like this: def connect(self): hostname = self.hostname s = socket.create_connection((hostname, self.port)) sock = self.context.wrap_socket(sock, server_hostname=hostname) sock.check_cert(hostname, initiator=self) So what does this bring compared to the statu quo? I thought at least sock.check_cert() would be called automatically, but if you have to call it yourself it starts looking pointless, IMO. Regards Antoine. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython: Fix _Py_normalize_encoding(): ensure that buffer is big enough to store "utf-8"
2013/11/7 victor.stinner :
> http://hg.python.org/cpython/rev/99afa4c74436
> changeset: 86995:99afa4c74436
> user:Victor Stinner
> date:Thu Nov 07 13:33:36 2013 +0100
> summary:
> Fix _Py_normalize_encoding(): ensure that buffer is big enough to store
> "utf-8"
> if the input string is NULL
>
> files:
> Objects/unicodeobject.c | 2 ++
> 1 files changed, 2 insertions(+), 0 deletions(-)
>
>
> diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
> --- a/Objects/unicodeobject.c
> +++ b/Objects/unicodeobject.c
> @@ -2983,6 +2983,8 @@
> char *l_end;
>
> if (encoding == NULL) {
> +if (lower_len < 6)
How about doing something like strlen("utf-8") rather than hardcoding that?
--
Regards,
Benjamin
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 384 (stable api) question
On 8 Nov 2013 04:42, Martin v. Löwis wrote: > > Am 07.11.13 13:44, schrieb Thomas Heller: > > > I thought that the stable API would keep exactly the same across > > releases - is this expectation wrong or is this a bug? > > Oscar is right - this change doesn't affect the ABI, just the API. > > That said, please file an issue reporting what change you see in > your compiler warnings. 3.4 is not released yet, perhaps there is > something that can be done. Thanks for bringing this up, Thomas. I'd spotted warnings from that change in CPython's own build process on Fedora, but not followed it up yet. At a very quick glance, the Fedora issue appeared to be a platform header that publishes a parameter as "char *", but CPython is now calling it with a "const char *" pointer. (I agree with the change to the API signatures in principle, though, since it endeavours to accurately indicate which string inputs CPython may modify and which we guarantee to leave alone. There may just be some specific cases where we can't actually make that guarantee because the underlying platform APIs aren't consistent in their declarations) Cheers, Nick. > > Regards, > Martin > > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython: Fix _Py_normalize_encoding(): ensure that buffer is big enough to store "utf-8"
2013/11/7 Benjamin Peterson :
> 2013/11/7 victor.stinner :
>> http://hg.python.org/cpython/rev/99afa4c74436
>> changeset: 86995:99afa4c74436
>> user:Victor Stinner
>> date:Thu Nov 07 13:33:36 2013 +0100
>> summary:
>> Fix _Py_normalize_encoding(): ensure that buffer is big enough to store
>> "utf-8"
>> if the input string is NULL
>>
>> files:
>> Objects/unicodeobject.c | 2 ++
>> 1 files changed, 2 insertions(+), 0 deletions(-)
>>
>>
>> diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
>> --- a/Objects/unicodeobject.c
>> +++ b/Objects/unicodeobject.c
>> @@ -2983,6 +2983,8 @@
>> char *l_end;
>>
>> if (encoding == NULL) {
>> +if (lower_len < 6)
>
> How about doing something like strlen("utf-8") rather than hardcoding that?
Full code:
if (encoding == NULL) {
if (lower_len < 6)
return 0;
strcpy(lower, "utf-8");
return 1;
}
On my opinion, it is easy to guess that 6 is len("utf-8") + 1 byte for NUL.
Calling strlen() at runtime may slow-down a function in the fast-path
of PyUnicode_Decode() and PyUnicode_AsEncodedString() which are
important functions. I know that some developers can execute strlen()
during compilation, but I don't see the need of replacing 6 with
strlen("utf-8")+1.
Victor
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Simplify and unify SSL verification
Am 07.11.2013 21:45, schrieb Antoine Pitrou:
> I'm in favour but I think 3.5 is too early. Keep in mind SSLContext is
> quite young.
It's available since 3.3
>> - deprecate implicit verify_mode=CERT_NONE. Python 3.5 will default
>>to CERT_REQUIRED.
>
> -0.9. This breaks compatibility and doesn't achieve anything, since
> there's no reliable story for CA certs.
I'd like to move to "secure by default". The CA cert situation is solved
on most platforms.
> Service matching is an application protocol level thing, it isn't
> strictly part of SSL. I'd feel warmer if you removed the "by default" part.
Hostname matching is done through an explicit call to
SSLSocket.check_cert(). All Python stdlib libraries like smtplib, ftplib
etc. are going to verify the hostname by default. 3rd party libraries
have to call check_cert() explicitly.
>> - add ssl.get_default_context() to acquire a default SSLContext object
>>if the context argument is None. (Python 3.4)
>
> I think I'm against it, for two reasons:
>
> 1. It will in the long term create compatibility and/or security issues
> (we'll have to keep fragile legacy settings if we want to avoid breaking
> existing uses of the default context)
>
> 2. SSLContexts are mutable, which means some code can affect other
> code's behaviour without even knowing. It's a recipe for subtle bugs and
> security issues.
>
> Applications and/or higher-level libraries should be smart enough to
> choose their own security preferences, and it's the better place to do
> so anyway.
You misunderstood me. I'm not proposing a global SSLContext object but a
factory function that creates a context for Python stdlib modules. Right
now every urllib, http.client, nntplib, asyncio, ftplib, poplib and
imaplib have duplicated code. I'd like to have ONE function that creates
and configures a SSLContext object with sensible default values for
Python stdlib.
3rd party apps are free to create their own SSLContext object.
> The check_cert() function wouldn't achieve anything besides calling
> match_hostname(), right?
It does more: http://pastebin.com/gKi7N2WM
> I'm against calling it check_cert(), it's too generic and only induces
> confusion.
Do you have an idea for a better name?
>
>> The SSLContext's check_cert option will support four values:
>>
>> None (default)
>>use match_hostname() if verify_mode is CERT_OPTIONAL or
>>CERT_REQUIRED
>> True
>>always use match_hostname()
>> False
>>never use match_hostname()
>
> What is the hostname that the cert is matched against?
The library code must provide as argument to check_cert or it's taken
from server_hostname if no hostname is provided.
> And why is there an "initiator" object? Personally I prefer to avoid
> passing opaque user data, since the caller can use a closure anyway.
> And what are the **kwargs?
No, they can't use a closure. The callback function is part of the
SSLContext object. The context object can be used by multiple SSLSocket
objects. The **kwargs make it possible to pass data from the caller of
check_cert() to the callback function of the SSLContext instance.
> So what does this bring compared to the statu quo? I thought at least
> sock.check_cert() would be called automatically, but if you have to call
> it yourself it starts looking pointless, IMO.
As you have said earlier, cert matching is not strictly a SSL connection
thing. With the check_cert feature you can do stuff like validation of
self-signed certs with known hash sums:
def check_cert_cb(sslsock, hostname, initiator, **kwargs):
# real code would use SPKI
certdigest = sha1(sslsock.getpeercert(True)).digest()
if hostname == "my.host.name" and certdigest == b"abcdef...":
return True
do_other_check(sslsock, hostname)
ctx = SSLContext(PROTOCOL_TLSv1, check_cert_cb)
ctx.verify_mode = CERT_NONE
httpscon = http.client.HTTPSConnection("my.host.name", 443, context=ctx)
httpscon = http.client.HTTPSConnection("my.host.name", 443, context=ctx)
httpscon = http.client.HTTPSConnection("my.host.name", 443, context=ctx)
Christian
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Simplify and unify SSL verification
On Nov 7, 2013, at 4:42 PM, Christian Heimes wrote: >>> >>> - deprecate implicit verify_mode=CERT_NONE. Python 3.5 will default >>> to CERT_REQUIRED. >> >> -0.9. This breaks compatibility and doesn't achieve anything, since >> there's no reliable story for CA certs. > > I'd like to move to "secure by default". The CA cert situation is solved > on most platforms. Please Yes, secure by default +1000 - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython: Fix _Py_normalize_encoding(): ensure that buffer is big enough to store "utf-8"
On 11/7/2013 4:38 PM, Victor Stinner wrote:
> 2013/11/7 Benjamin Peterson :
>> 2013/11/7 victor.stinner :
>>> http://hg.python.org/cpython/rev/99afa4c74436
>>> changeset: 86995:99afa4c74436
>>> user:Victor Stinner
>>> date:Thu Nov 07 13:33:36 2013 +0100
>>> summary:
>>> Fix _Py_normalize_encoding(): ensure that buffer is big enough to store
>>> "utf-8"
>>> if the input string is NULL
>>>
>>> files:
>>> Objects/unicodeobject.c | 2 ++
>>> 1 files changed, 2 insertions(+), 0 deletions(-)
>>>
>>>
>>> diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
>>> --- a/Objects/unicodeobject.c
>>> +++ b/Objects/unicodeobject.c
>>> @@ -2983,6 +2983,8 @@
>>> char *l_end;
>>>
>>> if (encoding == NULL) {
>>> +if (lower_len < 6)
>>
>> How about doing something like strlen("utf-8") rather than hardcoding that?
>
> Full code:
>
> if (encoding == NULL) {
> if (lower_len < 6)
> return 0;
> strcpy(lower, "utf-8");
> return 1;
> }
>
> On my opinion, it is easy to guess that 6 is len("utf-8") + 1 byte for NUL.
>
> Calling strlen() at runtime may slow-down a function in the fast-path
> of PyUnicode_Decode() and PyUnicode_AsEncodedString() which are
> important functions. I know that some developers can execute strlen()
> during compilation, but I don't see the need of replacing 6 with
> strlen("utf-8")+1.
Then how about at least a comment about how 6 is derived?
if (lower_len < 6) /* 6 == strlen("utf-8") + 1 */
return 0;
Eric.
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Avoid formatting an error message on attribute error
On 8 Nov 2013 04:44, "Brett Cannon" wrote: > On Thu, Nov 7, 2013 at 7:41 AM, Nick Coghlan wrote: >> >> The bigger problem is you can't change the constructor signature in a backwards incompatible way. You would need a new class method as an alternative constructor instead, or else use optional parameters. > > > The optional parameter approach is the one ImportError took for introducing its `name` and `path` attributes, so there is precedent. Currently it doesn't use a default exception message for backwards-compatibility, but adding one wouldn't be difficult technically. > > In the case of AttributeError, what you could do is follow the suggestion I made in http://bugs.python.org/issue18156 and add an `attr` keyword-only argument and correpsonding attribute to AttributeError. If you then add whatever other keyword arguments are needed to generate a good error message (`object` so you have the target, or at least get the string name to prevent gc issues for large objects?) you could construct the message lazily in the __str__ method very easily while also doing away with inconsistencies in the message which should make doctest users happy. Lazy message creation through __str__ does leave the message out of `args`, though. > > In a perfect world (Python 4 maybe?) BaseException would take a single argument which would be an optional message, `args` wouldn't exist, and people called `str(exc)` to get the message for the exception. That would allow subclasses to expand the API with keyword-only arguments to carry extra info and have reasonable default messages that were built on-demand when __str__ was called. It would also keep `args` from just being a dumping ground of stuff that has no structure except by calling convention (which is not how to do an API; explicit > implicit and all). IOW the original dream of PEP 352 ( http://python.org/dev/peps/pep-0352/#retracted-ideas). A related problem (that rich exceptions cause, rather than solve) is one I encountered recently in trying to improve the error messages thrown by the codec machinery to ensure they mention the specific codec operation that failed: you can't easily create a copy of a rich exception with all the same attributes, but a different error message (at least through the C API - copy.copy may work at the Python level, but I don't want to make the codec infrastructure depend on the copy module). Rich exception types that require more than one parameter are also incompatible with PyErr_Format in general. For the current patch [1], I've "solved" the problem by whitelisting a common set of types that the chaining code knows how to handle, but it would be good to have some kind of BaseException.replace() API that did the heavy lifting and returned None rather than throwing an exception if replacement wasn't supported (since blindly bitwise copying instance objects isn't safe for subclasses that add extra C level pointers and throwing an exception would be really annoying for exception chaining code that wants to reraise the original exception if replacement isn't possible). Cheers, Nick. [1] http://bugs.python.org/issue17828 ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython: Fix _Py_normalize_encoding(): ensure that buffer is big enough to store "utf-8"
2013/11/7 Eric V. Smith :
> Then how about at least a comment about how 6 is derived?
>
> if (lower_len < 6) /* 6 == strlen("utf-8") + 1 */
> return 0;
Ok, I added the comment in changeset 9c929b9e0a2a.
Victor
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython: Fix _Py_normalize_encoding(): ensure that buffer is big enough to store "utf-8"
On 11/7/2013 5:13 PM, Victor Stinner wrote:
> 2013/11/7 Eric V. Smith :
>> Then how about at least a comment about how 6 is derived?
>>
>> if (lower_len < 6) /* 6 == strlen("utf-8") + 1 */
>> return 0;
>
> Ok, I added the comment in changeset 9c929b9e0a2a.
Thanks!
Eric.
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Simplify and unify SSL verification
On 8 Nov 2013 04:18, "Christian Heimes" wrote: > > Hi, > > I'd like to simplify, unify and tighten SSL handling and verification > code in Python 3.5. Therefore I propose to deprecate some features for > Python 3.4. SSLContext objects are going to be the central instance for > configuration. > > In order to archive the goal I propose to > > - deprecate the key_file, cert_file and check_hostname arguments in > various in favor of context. Python 3.5 will no longer have these > arguments in http.client, smtplib etc. > > - deprecate implicit verify_mode=CERT_NONE. Python 3.5 will default > to CERT_REQUIRED. > > - enforce hostname matching in CERT_OPTIONAL or CERT_REQUIRED mode > by default in Python 3.5. > > - add ssl.get_default_context() to acquire a default SSLContext object > if the context argument is None. (Python 3.4) Change this to create_default_context() (to make it clearer there is no global context object) and I'm generally +1. > - add check_cert option to SSLContext and a check_cert() function to > SSLSocket in order to unify certificate validation and hostname > matching. (Python 3.4) > > > The SSLContext's check_cert option will support four values: I suggest just making this a "verifier" argument that's always a callable. > None (default) > use match_hostname() if verify_mode is CERT_OPTIONAL or > CERT_REQUIRED > True > always use match_hostname() > False > never use match_hostname() > callable function: > call func(sslsock, hostname, initiator, **kwargs) This overlaps confusingly with "verify_mode". Perhaps just offer a module level "verify_hostname" API that adapts between the verifier signature and match_hostname, say that is the default verifier, and that the verifier is always called. Also, how does the verifier know the verify_mode? Even if that info is set on the wrapped socket, it may be better to make it an optional argument to verifiers as well, to make it clearer that a verify mode of CERT_NONE may mean verification doesn't actually check anything. For the new method API (name suggestion: verify_cert), I believe that should default to CERT_REQUIRED, regardless of the mode configured on the wrapped socket. > The initiator argument is the object that has initiated the > SSL connection, e.g. http.client.HTTPSConnection object. > sslsock.context points to its SSLContext object > > > A connect method may look like this: > > def connect(self): > hostname = self.hostname > s = socket.create_connection((hostname, self.port)) > sock = self.context.wrap_socket(sock, > server_hostname=hostname) > sock.check_cert(hostname, initiator=self) > return sock > > > The check_cert option for SSLContext makes it easy to customize cert > handling. Application developers just need to configure a SSLContext > object in order to customize SSL cert verification and matching. In > Python 3.4 it will be possible to get the full peer chain, cert > information in CERT_NONE verification mode and SPKI data for SSL > pinning. A custom check_cert callback can be used to validate > self-signed certs or test for pinned certificates. > > > The proposal does NOT address the CA certificates difficulty. It's a > different story that needs to be addressed for all platforms separately. > Most Linux distributions, (Net|Free|Open) BSD and Mac Ports come with > SSL certs anyway, although some do not correctly handle the certs' > purposes. I have working code for Windows' cert system store that will > land in 3.4. Native Mac OS X hasn't been addressed yet. AIX, HP-UX, > Solaris etc. don't come with CA certs. > > Christian > > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython: Fix _Py_normalize_encoding(): ensure that buffer is big enough to store "utf-8"
2013/11/7 Victor Stinner :
> Calling strlen() at runtime may slow-down a function in the fast-path
> of PyUnicode_Decode() and PyUnicode_AsEncodedString() which are
> important functions. I know that some developers can execute strlen()
> during compilation, but I don't see the need of replacing 6 with
> strlen("utf-8")+1.
GCC will fold constant strlen calls. Another option would be to use sizeof().
--
Regards,
Benjamin
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Simplify and unify SSL verification
On Nov 07, 2013, at 10:42 PM, Christian Heimes wrote: >You misunderstood me. I'm not proposing a global SSLContext object but a >factory function that creates a context for Python stdlib modules. Right >now every urllib, http.client, nntplib, asyncio, ftplib, poplib and >imaplib have duplicated code. I'd like to have ONE function that creates >and configures a SSLContext object with sensible default values for >Python stdlib. I'm sure you're considering this, but I want to explicitly preserve the ability to register self-signed certificates. It's often necessary in practice, but very useful for testing purposes. ssl.SSLContext.load_cert_chain() is the way to do this, but will this be exposed in your proposed factory function? If not, then I think it's critically important that whatever API is exposed in the client code not hide the SSLContext object, such that clients of the client code can load up those self-signed certificates after the context has been created. -Barry ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython: Issue #19512: Add PyRun_InteractiveOneObject() function
New C APIs should either be documented or have an underscore prefix. Also, if they're part of the stable ABI, they need a version guard. Wishlist item: an automated ABI checker that can diff the exported symbols against a reference list (Could ctypes or cffi be used for that?) As long as this kind of thing involves manual review, we're going to keep making similar mistakes :P That said, a potentially simpler first step would be a list of common mistakes to check for/questions to ask during reviews as part of the devguide. Cheers, Nick. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython: Issue #19512: Add a new _PyDict_DelItemId() function, similar to
On 7 Nov 2013 04:06, "victor.stinner" wrote:
>
> http://hg.python.org/cpython/rev/69071054b42f
> changeset: 86968:69071054b42f
> user:Victor Stinner
> date:Wed Nov 06 18:58:22 2013 +0100
> summary:
> Issue #19512: Add a new _PyDict_DelItemId() function, similar to
> PyDict_DelItemString() but using an identifier for the key
>
> files:
> Include/dictobject.h | 1 +
> Objects/dictobject.c | 9 +
> 2 files changed, 10 insertions(+), 0 deletions(-)
>
>
> diff --git a/Include/dictobject.h b/Include/dictobject.h
> --- a/Include/dictobject.h
> +++ b/Include/dictobject.h
> @@ -109,6 +109,7 @@
> PyAPI_FUNC(int) PyDict_SetItemString(PyObject *dp, const char *key,
PyObject *item);
> PyAPI_FUNC(int) _PyDict_SetItemId(PyObject *dp, struct _Py_Identifier
*key, PyObject *item);
> PyAPI_FUNC(int) PyDict_DelItemString(PyObject *dp, const char *key);
> +PyAPI_FUNC(int) _PyDict_DelItemId(PyObject *mp, struct _Py_Identifier
*key);
As a private API, this shouldn't be part of the stable ABI.
Cheers,
Nick.
>
> #ifndef Py_LIMITED_API
> int _PyObjectDict_SetItem(PyTypeObject *tp, PyObject **dictptr, PyObject
*name, PyObject *value);
> diff --git a/Objects/dictobject.c b/Objects/dictobject.c
> --- a/Objects/dictobject.c
> +++ b/Objects/dictobject.c
> @@ -2736,6 +2736,15 @@
> }
>
> int
> +_PyDict_DelItemId(PyObject *v, _Py_Identifier *key)
> +{
> +PyObject *kv = _PyUnicode_FromId(key); /* borrowed */
> +if (kv == NULL)
> +return -1;
> +return PyDict_DelItem(v, kv);
> +}
> +
> +int
> PyDict_DelItemString(PyObject *v, const char *key)
> {
> PyObject *kv;
>
> --
> Repository URL: http://hg.python.org/cpython
>
> ___
> Python-checkins mailing list
> [email protected]
> https://mail.python.org/mailman/listinfo/python-checkins
>
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Simplify and unify SSL verification
Am 07.11.2013 23:25, schrieb Nick Coghlan: > Change this to create_default_context() (to make it clearer there is no > global context object) and I'm generally +1. Good idea! > This overlaps confusingly with "verify_mode". Perhaps just offer a > module level "verify_hostname" API that adapts between the verifier > signature and match_hostname, say that is the default verifier, and that > the verifier is always called. verify_mode = CERT_REQUIRED checks a lot of stuff like certificate chain, expiration date, cert purpose and all the other crypto + X.509 magic. It doesn't check that the cert data matches the host name, though. My proposal tries to make the SSLContext object *the* central configuration point for cert validation and give the user an API for a callback that e.g. can check a a self signed certificate or do cert pinning. It already holds all options (protocol, verify mode, CA certs, cert, key, cert chain, NPN protocol, ciphers ...). The context also holds information for SSL session resumption to speed up future connection. It's just for a way to configure hostname matching and post connection checks. > Also, how does the verifier know the verify_mode? Even if that info is > set on the wrapped socket, it may be better to make it an optional > argument to verifiers as well, to make it clearer that a verify mode of > CERT_NONE may mean verification doesn't actually check anything. It involves some layers of indirection. Since Python 3.2 all SSL sockets are created from a SSLContext object. Even SSLSocket.__init__() and ssl.wrap_socket() create a SSLContext object first. The SSLSocket instance has a reference to its SSLContext in SSLSocket.context. Internally SSLSocket.check_cert(hostname, **kwargs) calls self.context.check_cert(self, hostname, **kwargs) of its SSLContext. SSLSocket.check_cert() has also an option to shut down the SSL connection gracefully when an exception occurs. > For the new method API (name suggestion: verify_cert), I believe that > should default to CERT_REQUIRED, regardless of the mode configured on > the wrapped socket. I don't want to create more confusion between verify_mode and the new feature, so I didn't use the term "verify" in the method name. Do you have a good idea for a better name that does not contain verify? Christian ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython: Issue #19512: Add PyRun_InteractiveOneObject() function
Hi, [PyRun_InteractiveOneObject()] 2013/11/8 Nick Coghlan : > New C APIs should either be documented or have an underscore prefix. I created the issue #19518 to add documentation (but also to add even more functions :-)). > Also, if they're part of the stable ABI, they need a version guard. As most PyRun_xxx() functions, the function is declared in a "#ifndef Py_LIMITED_API" block. It means that it is not part of the stable ABI, is that correct? > Wishlist item: an automated ABI checker that can diff the exported symbols > against a reference list (Could ctypes or cffi be used for that?) Yeah, it would be nice to have a tool to list changes on the stable ABI :-) * * * For a previous issue, I also added various other functions like PyParser_ASTFromStringObject() or PyParser_ASTFromFileObject(). As PyRun_InteractiveOneObject(), they are declared in a "#ifndef Py_LIMITED_API" block in pythonrun.h. These new functions are not documented, because PyParser_AST*() functions are not documented. http://bugs.python.org/issue11619 http://hg.python.org/cpython/rev/df2fdd42b375 The patch parser_unicode.patch was attached to the issue during 2 years. First I didn't want to commit it becaues it added too many new functions. But I pushed by change because some Windows users like using the full Unicode range in the name of their scripts and modules! If it's not too late, I would appreciate a review of the stable ABI of this changeset. Victor ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython: Issue #19512: Add a new _PyDict_DelItemId() function, similar to
2013/11/8 Nick Coghlan : >> http://hg.python.org/cpython/rev/69071054b42f >> changeset: 86968:69071054b42f >> user:Victor Stinner >> date:Wed Nov 06 18:58:22 2013 +0100 >> summary: >> Issue #19512: Add a new _PyDict_DelItemId() function, similar to >> PyDict_DelItemString() but using an identifier for the key >> ... > > As a private API, this shouldn't be part of the stable ABI. When I don't know if a function should be made public or not, I compare with other functions. In Python 3.3, _PyDict_GetItemIdWithError(), _PyDict_GetItemId() and _PyDict_SetItemId() are part of the stable ABI if I read correctly dictobject.h. _PyObject_GetAttrId() is also part of the stable ABI. Was it a mistake, or did I misunderstand how stable functions are declared? Victor ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython: Issue #19512: Add PyRun_InteractiveOneObject() function
On 8 Nov 2013 09:45, "Victor Stinner" wrote: > > Hi, > > [PyRun_InteractiveOneObject()] > > 2013/11/8 Nick Coghlan : > > New C APIs should either be documented or have an underscore prefix. > > I created the issue #19518 to add documentation (but also to add even > more functions :-)). > > > Also, if they're part of the stable ABI, they need a version guard. > > As most PyRun_xxx() functions, the function is declared in a "#ifndef > Py_LIMITED_API" block. It means that it is not part of the stable ABI, > is that correct? Yeah, that's fine - I'm just replying on my phone at the moment, so checking the broader file context is a pain :) Cheers, Nick. > > > Wishlist item: an automated ABI checker that can diff the exported symbols > > against a reference list (Could ctypes or cffi be used for that?) > > Yeah, it would be nice to have a tool to list changes on the stable ABI :-) > > * * * > > For a previous issue, I also added various other functions like > PyParser_ASTFromStringObject() or PyParser_ASTFromFileObject(). As > PyRun_InteractiveOneObject(), they are declared in a "#ifndef > Py_LIMITED_API" block in pythonrun.h. These new functions are not > documented, because PyParser_AST*() functions are not documented. > > http://bugs.python.org/issue11619 > http://hg.python.org/cpython/rev/df2fdd42b375 > > The patch parser_unicode.patch was attached to the issue during 2 > years. First I didn't want to commit it becaues it added too many new > functions. But I pushed by change because some Windows users like > using the full Unicode range in the name of their scripts and modules! > > If it's not too late, I would appreciate a review of the stable ABI of > this changeset. > > Victor > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Simplify and unify SSL verification
Am 08.11.2013 00:09, schrieb Barry Warsaw: > I'm sure you're considering this, but I want to explicitly preserve the > ability to register self-signed certificates. It's often necessary in > practice, but very useful for testing purposes. > > ssl.SSLContext.load_cert_chain() is the way to do this, but will this be > exposed in your proposed factory function? If not, then I think it's > critically important that whatever API is exposed in the client code not hide > the SSLContext object, such that clients of the client code can load up those > self-signed certificates after the context has been created. If you want full control over the context then you can still create your own context object. Nobody is going to stop you from that. The factory function removes code duplication. Right now 6 modules have the same code for PROTOCOL_SSLv23 with OP_NO_SSLv2. Old code class HTTPSConnection: def __init__(self, hostname, port, key_file=None, cert_file=None, context=None): if context is None: # Some reasonable defaults context = ssl.SSLContext(ssl.PROTOCOL_SSLv23) context.options |= ssl.OP_NO_SSLv2 if key_file or cert_file: context.load_cert_chain(cert_file, key_file) New code def create_default_context(protocol=None): if protocol is None: context = SSLContext(PROTOCOL_SSLv23) context.options |= OP_NO_SSLv2 else: context = SSLContext(protocol) return context class HTTPSConnection: def __init__(self, hostname, port, context=None): if context is None: context = ssl.create_default_context() self.context = context If you want full control barrys_special_context = ssl.SSLContext(ssl.PROTOCOL_TLSv1) barrys_special_context.load_cert_chain(cert_file, key_file) con = HTTPSConnection(host, port, barrys_special_context) With my proposed new option for SSLContext() you also gain full control over hostname matching and extra cert checks. Super Barry power! :) Christian ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython: Issue #19512: Add PyRun_InteractiveOneObject() function
2013/11/8 Nick Coghlan : >> [PyRun_InteractiveOneObject()] >> (...) >> > Also, if they're part of the stable ABI, they need a version guard. >> >> As most PyRun_xxx() functions, the function is declared in a "#ifndef >> Py_LIMITED_API" block. It means that it is not part of the stable ABI, >> is that correct? > > Yeah, that's fine - I'm just replying on my phone at the moment, so checking > the broader file context is a pain :) About the 72523 functions PyRun_xxx(), I don't understand something. The PyRun_FileEx() is documented in the Python "very high" C API to use Python. But this function is not part of the stable ABI. So there is no "very high" function in the stable ABI to use Python? Another question: it's not documented if a function is part or not part of the stable ABI. So as an user of the API, it is hard to check if a function is part of the stable ABI or not. Victor ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython: Issue #19512: Add PyRun_InteractiveOneObject() function
2013/11/8 Victor Stinner : > Another question: it's not documented if a function is part or not > part of the stable ABI. So as an user of the API, it is hard to check > if a function is part of the stable ABI or not. A solution for that it maybe to split the documentation of the C API in two parts: * Stable C ABI * Private C ABI If it's stupid, at least functions of the stable ABI should be marked in the documentation. Victor ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Simplify and unify SSL verification
Somehow your mail didn't end up on Python-dev Am 08.11.2013 00:38, schrieb Nick Coghlan: > In that case, it sounds like you need *two* new options rather than > one. "verify_hostname", with the None/True/False behaviour and a > separate postverify hook. Mmmh, yes, you are making an intriguing point. Two different options are easier to understand and more powerful. > It contains the word verify, but if I'm correct in thinking you > intend for the new callback to be invoked only if the checks > specified by verify_mode pass, then I would suggest "postverify", > and skip adding the separate method. The tests specified by verify_mode are done by OpenSSL during the protocol handshake. The SSLSocket object has no peer, peer cert and transport information before the hand shake is done. So yes, these checks are always done before Python can match the hostname of the peer's cert and before the postverify hook can run. OpenSSL has a verify callback hook that is called for each certificate in the trust chain starting with the peer cert up to a root cert. This callback is too low level and too complex to be useful for the majority of users. Python would also have to gain wrappers for X509_STORE and X509_STORE_CTX objects... You don't want to know the difference :) ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython: Issue #19512: Add PyRun_InteractiveOneObject() function
On Thu, Nov 7, 2013 at 4:55 PM, Victor Stinner wrote: > About the 72523 functions PyRun_xxx(), I don't understand something. > The PyRun_FileEx() is documented in the Python "very high" C API to > use Python. But this function is not part of the stable ABI. So there > is no "very high" function in the stable ABI to use Python? My understanding is that if you are using the PyRun_* API (embedding?), you aren't concerned with maintaining binary compatibility across Python versions. If you are writing an extension module, you probably aren't using PyRun_*. > > Another question: it's not documented if a function is part or not > part of the stable ABI. So as an user of the API, it is hard to check > if a function is part of the stable ABI or not. The best we have is probably PEP 384. I've been meaning to work on the C-API docs for a while and add a concise reference page that would include a summary of the stable API. Alas, other things have taken precedence. -eric ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Simplify and unify SSL verification
Le 07/11/2013 22:42, Christian Heimes a écrit : Am 07.11.2013 21:45, schrieb Antoine Pitrou: I'm in favour but I think 3.5 is too early. Keep in mind SSLContext is quite young. It's available since 3.3 3.2 actually, while many code bases are still 2.x-compatible. There's no need to make migration more annoying right now. - deprecate implicit verify_mode=CERT_NONE. Python 3.5 will default to CERT_REQUIRED. -0.9. This breaks compatibility and doesn't achieve anything, since there's no reliable story for CA certs. I'd like to move to "secure by default". The CA cert situation is solved on most platforms. If it's not solved on *all* platforms then you're introducing a nasty discrepancy between platforms. That said, "secure by default" could be a property of the "default context factory" (see below). - add ssl.get_default_context() to acquire a default SSLContext object if the context argument is None. (Python 3.4) I think I'm against it, for two reasons: 1. It will in the long term create compatibility and/or security issues (we'll have to keep fragile legacy settings if we want to avoid breaking existing uses of the default context) 2. SSLContexts are mutable, which means some code can affect other code's behaviour without even knowing. It's a recipe for subtle bugs and security issues. Applications and/or higher-level libraries should be smart enough to choose their own security preferences, and it's the better place to do so anyway. You misunderstood me. I'm not proposing a global SSLContext object but a factory function that creates a context for Python stdlib modules. Right now every urllib, http.client, nntplib, asyncio, ftplib, poplib and imaplib have duplicated code. I'd like to have ONE function that creates and configures a SSLContext object with sensible default values for Python stdlib. Ok, so what about the long term compatibility issues? I'd be ok if you changed the name as suggested by Nick *and* if it was explicit that the security settings for this context can change between releases, therefore working code can break due to upgraded security standards. I'm against calling it check_cert(), it's too generic and only induces confusion. Do you have an idea for a better name? match_service()? match_cert()? But, really, it strikes me that adding this method to SSLSocket may make things more confusing. The docs will have to be clear that certificate *validation* and certificate *matching* are two different things (that is, you can't pass CERT_NONE and then expect calling match_cert() is sufficient). And why is there an "initiator" object? Personally I prefer to avoid passing opaque user data, since the caller can use a closure anyway. And what are the **kwargs? No, they can't use a closure. The callback function is part of the SSLContext object. > The context object can be used by multiple SSLSocket objects. So what? If you want to use multiple callbacks or closures, you can create multiple context objects. > The **kwargs make it possible to pass data from the caller of check_cert() to the callback function of the SSLContext instance. Well, I think such explicit "user data" needn't exist in Python. So what does this bring compared to the statu quo? I thought at least sock.check_cert() would be called automatically, but if you have to call it yourself it starts looking pointless, IMO. As you have said earlier, cert matching is not strictly a SSL connection thing. With the check_cert feature you can do stuff like validation of self-signed certs with known hash sums: Ok, so you can pass that context to e.g. httplib, right? I'm convinced, except on the "user data" part :) Regards Antoine. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
