Re: [Python-Dev] Reading Python source file

[Ryan Gonzalez]

Well, not quite the same thing, but 
https://github.com/kirbyfan64/pfbuild/blob/master/pfbuild embeds the compressed 
version of 16k LOC. Would it be affected negatively in any way be this?

Since all the data is on one line, I'd think the old (current) parser would end 
up reading in the whole line anyway.

On November 18, 2015 9:48:41 AM CST, Guido van Rossum  wrote:
>On Wed, Nov 18, 2015 at 4:15 AM, Hrvoje Niksic 
>wrote:
>> On 11/18/2015 03:31 AM, Nick Coghlan wrote:
>>>
>>> That behaviour is then inherited at the command line by both the -m
>>> switch and the support for executing directories and zip archives.
>>> When we consider that the "-c" switch also executes an in-memory
>>> string, direct script execution is currently the odd one out in
>*not*
>>> reading the entire source file into memory first, so Serhiy's
>proposed
>>> simplification of the implementation makes sense to me.
>>
>>
>> Reading the whole script in memory will incur an overhead when
>executing
>> scripts that contain (potentially large) data embedded after the end
>of
>> script source.
>>
>> The technique of reading data from sys.argv[0] is probably obsolete
>now that
>> Python supports executing zipped archives, but it is popular in shell
>> scripting and might still be used for self-extracting scripts that
>must
>> support older Python versions. This feature doesn't affect imports
>and -c
>> which are not expected to contain non-Python data.
>
>That trick doesn't work unless the data looks like Python comments or
>data (e.g. a docstring). Python has always insisted on being able to
>parse until EOF. The only extreme case would be a small script
>followed by e.g. 4 GB of comments (where the old parser would indeed
>be more efficient). But unless you can point me to an occurrence of
>this in the wild I'm going to speculate that you just made this up
>based on the shell analogy (which isn't perfect).
>
>-- 
>--Guido van Rossum (python.org/~guido)
>___
>Python-Dev mailing list
>Python-Dev@python.org
>https://mail.python.org/mailman/listinfo/python-dev
>Unsubscribe:
>https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com

-- 
Sent from my Nexus 5 with K-9 Mail. Please excuse my brevity.___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Reading Python source file

[Guido van Rossum]

On Wed, Nov 18, 2015 at 4:15 AM, Hrvoje Niksic  wrote:
> On 11/18/2015 03:31 AM, Nick Coghlan wrote:
>>
>> That behaviour is then inherited at the command line by both the -m
>> switch and the support for executing directories and zip archives.
>> When we consider that the "-c" switch also executes an in-memory
>> string, direct script execution is currently the odd one out in *not*
>> reading the entire source file into memory first, so Serhiy's proposed
>> simplification of the implementation makes sense to me.
>
>
> Reading the whole script in memory will incur an overhead when executing
> scripts that contain (potentially large) data embedded after the end of
> script source.
>
> The technique of reading data from sys.argv[0] is probably obsolete now that
> Python supports executing zipped archives, but it is popular in shell
> scripting and might still be used for self-extracting scripts that must
> support older Python versions. This feature doesn't affect imports and -c
> which are not expected to contain non-Python data.

That trick doesn't work unless the data looks like Python comments or
data (e.g. a docstring). Python has always insisted on being able to
parse until EOF. The only extreme case would be a small script
followed by e.g. 4 GB of comments (where the old parser would indeed
be more efficient). But unless you can point me to an occurrence of
this in the wild I'm going to speculate that you just made this up
based on the shell analogy (which isn't perfect).

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Benchmark results across all major Python implementations

[R. David Murray]

On 17 Nov 2015, at 21:22, Stewart, David C  wrote:
> On 11/17/15, 10:40 AM, "Python-Dev on behalf of R. David Murray" 
>  rdmur...@bitdance.com> wrote:
>>
>> I suppose that for this to have maximum effect someone would have to
>> specifically be paying attention to performance and figuring out why
>> every (real) regression happened.  I don't suppose we have anyone in the
>> community currently who is taking on that role, though we certainly do
>> have people who are *interested* in Python performance :)
> 
> We're trying to fill that role as much as we can. When there is a
> significant (and unexplained) regression that we see, I usually ask
> our engineers to bisect it to identify the offending patch and
> root-cause it.

That's great news.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Reading Python source file

[Hrvoje Niksic]

On 11/18/2015 04:48 PM, Guido van Rossum wrote:

That trick doesn't work unless the data looks like Python comments or
data (e.g. a docstring). Python has always insisted on being able to
parse until EOF. The only extreme case would be a small script
followed by e.g. 4 GB of comments (where the old parser would indeed
be more efficient). But unless you can point me to an occurrence of
this in the wild I'm going to speculate that you just made this up
based on the shell analogy (which isn't perfect).


If this never really worked in Python, feel free to drop the issue. I 
may be misremembering the language in which scripts I saw using this 
techniques years ago were written - most likely sh or Perl.


Hrvoje

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Reading Python source file

[Hrvoje Niksic]

On 11/18/2015 03:31 AM, Nick Coghlan wrote:

That behaviour is then inherited at the command line by both the -m
switch and the support for executing directories and zip archives.
When we consider that the "-c" switch also executes an in-memory
string, direct script execution is currently the odd one out in *not*
reading the entire source file into memory first, so Serhiy's proposed
simplification of the implementation makes sense to me.


Reading the whole script in memory will incur an overhead when executing 
scripts that contain (potentially large) data embedded after the end of 
script source.


The technique of reading data from sys.argv[0] is probably obsolete now 
that Python supports executing zipped archives, but it is popular in 
shell scripting and might still be used for self-extracting scripts that 
must support older Python versions. This feature doesn't affect imports 
and -c which are not expected to contain non-Python data.


Hrvoje

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Reading Python source file

[Nick Coghlan]

On 19 November 2015 at 02:50, Ryan Gonzalez  wrote:
> Well, not quite the same thing, but
> https://github.com/kirbyfan64/pfbuild/blob/master/pfbuild embeds the
> compressed version of 16k LOC. Would it be affected negatively in any way be
> this?
>
> Since all the data is on one line, I'd think the old (current) parser would
> end up reading in the whole line anyway.

Right. The other main "embedded binary blob" case I'm familiar with is
get-pip.py, which embeds a base85 encoded copy of pip as a zip archive
in a DATA variable, and there aren't any appending tricks there either
- it's a normal variable, and the "if __name__ == '__main__':" block
is the final two lines of the file, so Python already has to read the
whole thing before it starts the process of unpacking it and executing
it.

Two things worth keeping in mind here is that if a script is embedding
enough data for reading the whole thing into memory it to pose a
problem:

* compiling and executing it is likely to pose an even bigger problem
than reading it in
* downloading it in the first place is also likely to be annoying

We couldn't make a change like this in a maintenance release, but for
a new feature release, the consistency gain is definitely worth it.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] cpython (3.4): asyncio.docs: Fix versionadded

[Martin Panter]

On 18 November 2015 at 17:45, yury.selivanov  wrote:
> https://hg.python.org/cpython/rev/b34c42e46e7b
> changeset:   99204:b34c42e46e7b
> branch:  3.4
> parent:  99201:89d66f912671
> user:Yury Selivanov 
> date:Wed Nov 18 12:44:31 2015 -0500
> summary:
>   asyncio.docs: Fix versionadded
>
> files:
>   Doc/library/asyncio-task.rst |  2 +-
>   1 files changed, 1 insertions(+), 1 deletions(-)
>
>
> diff --git a/Doc/library/asyncio-task.rst b/Doc/library/asyncio-task.rst
> --- a/Doc/library/asyncio-task.rst
> +++ b/Doc/library/asyncio-task.rst
> @@ -699,4 +699,4 @@
>Unlike the functions above, :func:`run_coroutine_threadsafe` requires 
> the
>*loop* argument to be passed explicitely.
>
> -   .. versionadded:: 3.4.4
> +   .. versionadded:: 3.4.4, 3.5.1

This doesn’t get marked up properly; have a look at
.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Python stdlib ssl.SSLContext is missing mode setting ability

[Ben Bangert]

In Python 2 and 3, the ssl module's SSLContext object has a way to set
SSL options, but not to set SSL modes.

The set_mode command and some of the available modes:
https://www.openssl.org/docs/manmaster/ssl/SSL_CTX_set_mode.html

The most critical mode is SSL_MODE_RELEASE_BUFFERS, which can drop the
SSL overhead *per connection* from around 25kb to ~7kb. The pyopenssl
library allows the setting of SSLContext modes, it seems very odd that
the Python 2/3 ssl modules do not. Though I could understand that
perhaps not all SSL libraries Python might build against would have
this mode thing available.

(BoringSSL sets this mode by default its considered such an obvious win)

If there is some way to set this I happened to miss, apologies, I only
went looking through the docs for it, not the code.

On a side-note, in my testing, Python 3.4 had about 20kb/connection of
overhead for using SSL, but Python 3.5 jumped to 30kb/connection of
SSL overhead. These numbers for SSL overhead are far too high for any
reasonable use of Python+SSL on highly concurrent systems. Test repo
for testing SSL overhead here:
https://github.com/bbangert/ssl-ram-testing/

Cheers,
Ben
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com