Re: [Python-Dev] Python stdlib ssl.SSLContext is missing mode setting ability
> On 19 Nov 2015, at 03:53, Ben Bangert wrote: > > In Python 2 and 3, the ssl module's SSLContext object has a way to set > SSL options, but not to set SSL modes. > > The set_mode command and some of the available modes: > https://www.openssl.org/docs/manmaster/ssl/SSL_CTX_set_mode.html > > The most critical mode is SSL_MODE_RELEASE_BUFFERS, which can drop the > SSL overhead *per connection* from around 25kb to ~7kb. The pyopenssl > library allows the setting of SSLContext modes, it seems very odd that > the Python 2/3 ssl modules do not. Though I could understand that > perhaps not all SSL libraries Python might build against would have > this mode thing available. > Ben, Do we need the ability to set arbitrary modes? Most of the modes mentioned in the OpenSSL documentation are things we actively don’t want the user to set because stuff will randomly break. With that in mind, and with the fact that SSL_MODE_RELEASE_BUFFERS is so obviously better than the standard, should we just instead have the ssl module automatically set SSL_MODE_RELEASE_BUFFERS unconditionally? If so, I’m happy to submit a bug/patch to get that to happen. Cory signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Reading Python source file
On 18 November 2015 at 15:57, Hrvoje Niksic wrote: > If this never really worked in Python, feel free to drop the issue. I may be > misremembering the language in which scripts I saw using this techniques > years ago were written - most likely sh or Perl. It was Perl. In the past I've tried to emulate this in Python and it's not been very successful, so I'd say it's not something to worry about here. Paul ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Reading Python source file
On 17.11.2015 16:22, Guido van Rossum wrote: > On Tue, Nov 17, 2015 at 1:59 AM, M.-A. Lemburg wrote: >>> [moving from read source line by line to reading all in one go] >> We use the same simplification in eGenix PyRun's emulation of >> the Python command line interface and it has so far not >> caused any problems. > > Curious how you do it? I'd actually be quite disappointed if the > amount of parsing done by the standard REPL went down. Oh, that's easy: elif sys.argv[0] == '-' and not (pyrun_as_string or pyrun_as_module): # Read the script from stdin pyrun_as_string = True pyrun_script = sys.stdin.read() and then, later on: # Run the script try: pyrun_execute_script(pyrun_script, mode) except Exception as reason: if pyrun_interactive: import traceback traceback.print_exc() pyrun_prompt(banner='') else: raise else: # Enter interactive mode, in case wanted if pyrun_interactive: pyrun_prompt() The REPL is not affected by this, since we use the standard code.interact() for the prompt. This reads the entry line by line, joins the lines and tries to compile the entry every time it receives a new line until it succeeds or fails. Serhiy's proposed change should not affect this mode of operation. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Nov 19 2015) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ 2015-10-23: Released mxODBC Connect 2.1.5 ... http://egenix.com/go85 ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python stdlib ssl.SSLContext is missing mode setting ability
On 19.11.2015 09:14, Cory Benfield wrote: > >> On 19 Nov 2015, at 03:53, Ben Bangert wrote: >> >> In Python 2 and 3, the ssl module's SSLContext object has a way to set >> SSL options, but not to set SSL modes. >> >> The set_mode command and some of the available modes: >> https://www.openssl.org/docs/manmaster/ssl/SSL_CTX_set_mode.html >> >> The most critical mode is SSL_MODE_RELEASE_BUFFERS, which can drop the >> SSL overhead *per connection* from around 25kb to ~7kb. The pyopenssl >> library allows the setting of SSLContext modes, it seems very odd that >> the Python 2/3 ssl modules do not. Though I could understand that >> perhaps not all SSL libraries Python might build against would have >> this mode thing available. >> > > Ben, > > Do we need the ability to set arbitrary modes? Most of the modes mentioned in > the OpenSSL documentation are things we actively don’t want the user to set > because stuff will randomly break. With that in mind, and with the fact that > SSL_MODE_RELEASE_BUFFERS is so obviously better than the standard, should we > just instead have the ssl module automatically set SSL_MODE_RELEASE_BUFFERS > unconditionally? > > If so, I’m happy to submit a bug/patch to get that to happen. The mode should only be enabled for OpenSSL versions which are not affected by this vulnerability: https://www.rapid7.com/db/vulnerabilities/http-openssl-cve-2014-0198 Other than that it seems like a good way forward. Plenty other projects have had this enabled per default for years: http://www.dovecot.org/list/dovecot/2011-October/131381.html https://svn.boost.org/trac/boost/changeset/71706 https://community.openvpn.net/openvpn/ticket/157 -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Nov 19 2015) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ 2015-10-23: Released mxODBC Connect 2.1.5 ... http://egenix.com/go85 ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Reading Python source file
On 17.11.15 18:50, Guido van Rossum wrote:
On Tue, Nov 17, 2015 at 8:20 AM, Serhiy Storchaka wrote:
Current implementation of import system went the same way. As a result
importing the script as a module and running it with command line can have
different behaviours in corner cases.
I'm confused. *Of course* these two behaviors differ, since Python
uses a different __name__. Not sure how this relates to the REPL.
Sorry for confusing. I meant parser level. File parser has a few bugs,
that can cause that the source will be differently interpreted with file
and string parsers. For example attached script produces different
output, "ä" if executed as a script, and "À" if imported as a module.
And there is a question about the null byte. Now compile(), exec(),
eval() raises an exception if the script contains the null byte.
Formerly they accepted it, but the null byte ended the script. The
behavior of file parser is more weird. The null byte makes parser to
ignore the end of script including the newline byte [1]. E.g.
"#\0\nprint('a')" is interpreted as "#print('a')". This is different
from PyPy (and may be other implementations) that interprets the null
byte just as ordinal character.
The question is wherever we should support the null byte in Python sources.
[1] http://bugs.python.org/issue20115
#
Re: [Python-Dev] Reading Python source file
On Thu, Nov 19, 2015 at 10:51 PM, Serhiy Storchaka wrote: > http://bugs.python.org/issue20115 Interestingly, the file linked in the last comment on that issue [1] ties in with another part of this thread, regarding binary blobs in Python scripts. It uses open(sys.argv[0],'rb') to find itself, and has the binary data embedded as a comment. (Though the particular technique strikes me as fragile; to prevent newlines from messing things up, \n becomes #% and \r becomes #$, but I don't see any protection against those sequences occurring naturally in the blob. Given that it's a bz2 archive, I would expect any two-byte sequence to be capable of occurring.) To be quite honest, I wouldn't mind if Python objected to this kind of code. If I were writing it myself, I'd use a triple-quoted string containing some kind of textualized version - either the repr of the string, or some kind of base 64 or base 85 encoding. Binary data embedded literally will prevent non-ASCII file encodings. ChrisA [1] http://ftp.waf.io/pub/release/waf-1.7.16 ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance)
On 11 November 2015 at 10:47, Nick Coghlan wrote: > Our last discussion back in July seemed to show that folks either > didn't care about the question (because they're using unmodified > upstream versions so the PEP didn't affect them), or else thought the > approach described in the PEP was reasonable, so I'm hoping the > consequences of my mistake won't be too severe. > RHEL 7.2 is out today, together with details of what we've now committed to supporting for certification verification configuration in the RHEL 7 system Python: https://access.redhat.com/articles/2039753 That article also describes the PEP 476 status across the different Python builds Red Hat supports, which I've summarised below. Versions with PEP 476 implemented: * Python 3.4 SCL: cert verification always enabled by default * system Python in RHEL 7.2+: still off by default due to the compatibility break, but with PEP 493's file based configuration setting to enable it by default Versions without PEP 476 implemented: * Python 2.7 SCL * Python 3.3 SCL * system Python in RHEL 7.1 and earlier * system Python in all versions of RHEL 6 I assume that status of the Python 2.7 SCL will change at some point, but don't have an ETA or any technical details to share at this point. The system Python versions in RHEL 5 and earlier didn't include the ssl module at all (since that was only added to the standard library in Python 2.6), so they're not affected by the concerns raised in PEP 476 in the first place. Regards, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python stdlib ssl.SSLContext is missing mode setting ability
On Thu, Nov 19, 2015 at 1:58 AM, M.-A. Lemburg wrote: > On 19.11.2015 09:14, Cory Benfield wrote: >> >>> On 19 Nov 2015, at 03:53, Ben Bangert wrote: >>> >>> In Python 2 and 3, the ssl module's SSLContext object has a way to set >>> SSL options, but not to set SSL modes. >>> >>> The set_mode command and some of the available modes: >>> https://www.openssl.org/docs/manmaster/ssl/SSL_CTX_set_mode.html >>> >>> The most critical mode is SSL_MODE_RELEASE_BUFFERS, which can drop the >>> SSL overhead *per connection* from around 25kb to ~7kb. The pyopenssl >>> library allows the setting of SSLContext modes, it seems very odd that >>> the Python 2/3 ssl modules do not. Though I could understand that >>> perhaps not all SSL libraries Python might build against would have >>> this mode thing available. >>> >> >> Ben, >> >> Do we need the ability to set arbitrary modes? Most of the modes mentioned >> in the OpenSSL documentation are things we actively don’t want the user to >> set because stuff will randomly break. With that in mind, and with the fact >> that SSL_MODE_RELEASE_BUFFERS is so obviously better than the standard, >> should we just instead have the ssl module automatically set >> SSL_MODE_RELEASE_BUFFERS unconditionally? >> >> If so, I’m happy to submit a bug/patch to get that to happen. > > The mode should only be enabled for OpenSSL versions which are > not affected by this vulnerability: > > https://www.rapid7.com/db/vulnerabilities/http-openssl-cve-2014-0198 > > Other than that it seems like a good way forward. Plenty other > projects have had this enabled per default for years: I can't think of any other mode to set, setting this with the condition cited for that vulnerability looks like a good idea. Cheers, Ben ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Reading Python source file
On Thu, Nov 19, 2015 at 1:47 AM, M.-A. Lemburg wrote:
> On 17.11.2015 16:22, Guido van Rossum wrote:
>> On Tue, Nov 17, 2015 at 1:59 AM, M.-A. Lemburg wrote:
[moving from read source line by line to reading all in one go]
>>> We use the same simplification in eGenix PyRun's emulation of
>>> the Python command line interface and it has so far not
>>> caused any problems.
>>
>> Curious how you do it? I'd actually be quite disappointed if the
>> amount of parsing done by the standard REPL went down.
>
> Oh, that's easy:
>
> elif sys.argv[0] == '-' and not (pyrun_as_string or pyrun_as_module):
> # Read the script from stdin
> pyrun_as_string = True
> pyrun_script = sys.stdin.read()
>
> and then, later on:
>
> # Run the script
> try:
> pyrun_execute_script(pyrun_script, mode)
> except Exception as reason:
> if pyrun_interactive:
> import traceback
> traceback.print_exc()
> pyrun_prompt(banner='')
> else:
> raise
> else:
> # Enter interactive mode, in case wanted
> if pyrun_interactive:
> pyrun_prompt()
Yes, this makes sense.
> The REPL is not affected by this, since we use the standard
> code.interact() for the prompt. This reads the entry line
> by line, joins the lines and tries to compile the entry every
> time it receives a new line until it succeeds or fails.
Heh, I wrote code.interact() as a poor-man's substitute for what the
"real" REPL (implemented in C) does. :-) It usually ends up doing the
same thing, but I'm sure there are edge cases where the "real" REPL is
better. It doesn't re-parse after each line is read, it actually keeps
the parser state and adds new tokens read from the tty. There is even
a special grammar root symbol ('single') for this mode.
> Serhiy's proposed change should not affect this mode of
> operation.
I sure hope not.
Though there is actually one case that IIRC doesn't work today: if
sys.stdin is a stream that doesn't wrap a file descriptor. Would be
nice to make that work. (Pretty esoteric use case though.)
--
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python stdlib ssl.SSLContext is missing mode setting ability
> On 19 Nov 2015, at 15:26, Ben Bangert wrote: > > I can't think of any other mode to set, setting this with the > condition cited for that vulnerability looks like a good idea. > > Cheers, > Ben Ok, we’re agreed. The work can be tracked under Issue 25672: https://bugs.python.org/issue25672 Cory signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Reading Python source file
On Thu, Nov 19, 2015 at 3:51 AM, Serhiy Storchaka wrote:
> On 17.11.15 18:50, Guido van Rossum wrote:
>>
>> On Tue, Nov 17, 2015 at 8:20 AM, Serhiy Storchaka
>> wrote:
>>>
>>> Current implementation of import system went the same way. As a result
>>> importing the script as a module and running it with command line can
>>> have
>>> different behaviours in corner cases.
>>
>>
>> I'm confused. *Of course* these two behaviors differ, since Python
>> uses a different __name__. Not sure how this relates to the REPL.
>
> Sorry for confusing. I meant parser level. File parser has a few bugs, that
> can cause that the source will be differently interpreted with file and
> string parsers. For example attached script produces different output, "ä"
> if executed as a script, and "À" if imported as a module.
I see. Well, I trust you in this area (it's been too long since I
wrote the original code for all that :-).
> And there is a question about the null byte. Now compile(), exec(), eval()
> raises an exception if the script contains the null byte. Formerly they
> accepted it, but the null byte ended the script.
I like erroring out better.
> The behavior of file parser
> is more weird. The null byte makes parser to ignore the end of script
> including the newline byte [1]. E.g. "#\0\nprint('a')" is interpreted as
> "#print('a')". This is different from PyPy (and may be other
> implementations) that interprets the null byte just as ordinal character.
Yeah, this is just poorly written old code that didn't expect anyone
to care about null bytes. IMO here too the null byte should be an
error.
> The question is wherever we should support the null byte in Python sources.
>
> [1] http://bugs.python.org/issue20115
Good luck, and thanks for working on this!
--
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Reading Python source file
Yeah, let's kill this undefined behavior. On Thu, Nov 19, 2015 at 4:10 AM, Chris Angelico wrote: > On Thu, Nov 19, 2015 at 10:51 PM, Serhiy Storchaka > wrote: >> http://bugs.python.org/issue20115 > > Interestingly, the file linked in the last comment on that issue [1] > ties in with another part of this thread, regarding binary blobs in > Python scripts. It uses open(sys.argv[0],'rb') to find itself, and has > the binary data embedded as a comment. (Though the particular > technique strikes me as fragile; to prevent newlines from messing > things up, \n becomes #% and \r becomes #$, but I don't see any > protection against those sequences occurring naturally in the blob. > Given that it's a bz2 archive, I would expect any two-byte sequence to > be capable of occurring.) To be quite honest, I wouldn't mind if > Python objected to this kind of code. If I were writing it myself, I'd > use a triple-quoted string containing some kind of textualized version > - either the repr of the string, or some kind of base 64 or base 85 > encoding. Binary data embedded literally will prevent non-ASCII file > encodings. > > ChrisA > > [1] http://ftp.waf.io/pub/release/waf-1.7.16 > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Improving the reading part of REPL
It seems that there will be some refactoring of the tokenizer code. Regarding this, I'd like to recall my proposal on readline hooks. It would be nice if char* based PyOS_Readline API was replaced by a Python str based hook customizable by Python code. I propose to add function sys.readlinehook accepting optional prompt and returning a line read interactively from a user. There would also be sys.__readlinehook__ containing the original value of sys.readlinehook (similarly to sys.(__)displayhook(__), sys.(__)excepthook(__) and sys.(__)std(in/out/err)(__)). Currently, the input is read from C stdin even if sys.stdin is changed (see http://bugs.python.org/issue17620). This complicates fixing http://bugs.python.org/issue1602 – the standard sys.std* streams are not capable of communicating in Unicode with Windows console, and replacing the streams with custom ones is not enough – one has also to install a custom readline hook, which is currently complicated. And even after installing a custom readine hook one finds out that Python tokenizer cannot handle UTF-16, so he has to wrap the custom stream objects just to let their encoding attribute have a different value, because readlinehook currently returns char* rather than a Python string. For more details see the documentation of my package: https://github.com/Drekin/win-unicode-console. The pyreadline package also sets up a custom readline so it would benefit if doing so would be easier. Moreover, the two consumers of PyOS_Readline API – the input function and the tokenizer – assume a different encoding of the bytes returned by the readlinehook. Effectively, one assumes sys.stdout.encoding and the other sys.stdin.encoding, so if these two are different, there is no way to implement a correct readline hook. If sys.readlinehook was added, the builting input function would be just a thin wrapper over sys.readlinehook removing the newline character and turning no input into EOFError. I thing that the best default value for sys.readlinehook on Windows would be stdio_readline – just write the prompt to sys.stdout and read a line from sys.stdin. On Linux, the default implementation would call GNU readline if it is available and sys.stdin and sys.stdout are standard TTYs (the check present in the current implementation of the input function), and it would call stdio_readline otherwise. Regards, Adam Bartoš ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
