Re: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance)

[Nick Coghlan]

On 17 November 2015 at 20:33, Victor Stinner  wrote:
> 2015-11-17 1:00 GMT+01:00 Guido van Rossum :
>> Hm, making Christian the BDFL-delegate would mean two out of three
>> authors *and* the BDFL-delegate all working for Red Hat, which clearly
>> has a stake (and IIUC has already committed to this approach ahead of
>> PEP approval). SO then it would look like this is just rubber-stamping
>> Red Hat's internal decision process (if it's a process -- sounds more
>> like an accident :-).
>
> Can we try to get a vote from maintainers of the Python2/3 packages of
> other Linux distributions? Debian, Ubuntu, OpenSUSE, etc.?

I know Oracle were interested based on a discussion between them and a
member of Red Hat's product security team about it on oss-security,
but their devs never followed up on it upstream (even after an
explicit suggestion that they do so), so I'm interpreting that as
willingness to go along with whatever happens in RHEL.

For Debian, Ubuntu and SUSE, their original determinations for the
relevant CVE were "too intrusive to backport", so folks currently need
to upgrade to newer versions of those distros to get the improved
default behaviour:

* http://people.canonical.com/~ubuntu-security/cve/2014/CVE-2014-9365.html
* https://security-tracker.debian.org/tracker/CVE-2014-9365
* https://www.suse.com/security/cve/CVE-2014-9365.html

If having an opt-in backwards-compatible-by-default approach available
(albeit as a PEP 466+476+493 patch set in the RHEL/CentOS system
Python 2.7.5 package) prompts other distro security teams to
reconsider those initial assessments, that would be a nice outcome,
but it isn't my own main priority (so Guido makes a good point in
favouring finding a non-Red-Hatter willing to act as BDFL-Delegate)

Regards,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Reading Python source file

[Serhiy Storchaka]

On 17.11.15 17:22, Guido van Rossum wrote:

But more important is the interactive REPL, which parses your input
fully each time you hit ENTER.


Interactive REPL runs different code. It is simpler that the code for 
reading from file, because it have no care about BOM or coding cookie.



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Reading Python source file

[Serhiy Storchaka]

On 17.11.15 18:06, Guido van Rossum wrote:

OK, but what are you going to do about the interactive REPL?


Nothing (except some simplification). This is a separate branch of the code.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Reading Python source file

[Guido van Rossum]

OK, but what are you going to do about the interactive REPL?

On Tue, Nov 17, 2015 at 7:40 AM, Serhiy Storchaka  wrote:
> On 17.11.15 05:00, Guido van Rossum wrote:
>>
>> If you free the memory used for the source buffer before starting code
>> generation you should be good.
>
>
> Thank you. The buffer is freed just after the end of generating AST.
>
>
>> On Mon, Nov 16, 2015 at 5:53 PM, Serhiy Storchaka 
>> wrote:
>>>
>>> I'm working on rewriting Python tokenizer (in particular the part that
>>> reads
>>> and decodes Python source file). The code is complicated. For now there
>>> are
>>> such cases:
>>>
>>> * Reading from the string in memory.
>>> * Interactive reading from the file.
>>> * Reading from the file:
>>>- Raw reading ignoring encoding in parser generator.
>>>- Raw reading UTF-8 encoded file.
>>>- Reading and recoding to UTF-8.
>>>
>>> The file is read by the line. It makes hard to check correctness of the
>>> first line if the encoding is specified in the second line. And it makes
>>> very hard problems with null bytes and with desynchronizing buffered C
>>> and
>>> Python files. All this problems can be easily solved if read all Python
>>> source file in memory and then parse it as string. This would allow to
>>> drop
>>> a large complex and buggy part of code.
>>>
>>> Are there disadvantages in this solution? As for memory consumption, the
>>> source text itself will consume only small part of the memory consumed by
>>> AST tree and other structures. As for performance, reading and decoding
>>> all
>>> file can be faster then by the line.
>>>
>>> [1] http://bugs.python.org/issue25643
>>>
>>> ___
>>> Python-Dev mailing list
>>> Python-Dev@python.org
>>> https://mail.python.org/mailman/listinfo/python-dev
>>> Unsubscribe:
>>> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>>
>>
>>
>>
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Reading Python source file

[Serhiy Storchaka]

On 17.11.15 05:05, MRAB wrote:

As I understand it, *nix expects the shebang to be b'#!', which means
that the
first line should be ASCII-compatible (it's possible that the UTF-8 BOM
might
be present). This kind of suggests that encodings like UTF-16 would cause a
problem on such systems.

The encoding line also needs to be ASCII-compatible.

I believe that the recent thread "Support of UTF-16 and UTF-32 source
encodings" also concluded that UTF-16 and UTF-32 shouldn't be supported.

This means that you could treat the first 2 lines as though they were some
kind of extended ASCII (Latin-1?), the line ending being '\n' or '\r' or
'\r\n'.

Once you'd identify the encoding, you could decode everything (including
the
shebang line) using that encoding.


Yes, that is what I were going to implement (and already halfway here). 
My question is whether it is worth to complicate the code further to 
preserve reading by the line. In any case after reading the first line 
that doesn't contain neither coding cookie, nor non-comment tokens, we 
need to wait the second line.



(What should happen if the encoding line then decoded differently, i.e.
encoding_line.decode(encoding) != encoding_line.decode('latin-1')?)


The parser should got the line decoded with specified encoding.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Reading Python source file

[Serhiy Storchaka]

On 17.11.15 11:59, M.-A. Lemburg wrote:

I don't think these situations are all that common, though,
so reading in the full source code before compiling it
sounds like a reasonable approach.

We use the same simplification in eGenix PyRun's emulation of
the Python command line interface and it has so far not
caused any problems.


Current implementation of import system went the same way. As a result 
importing the script as a module and running it with command line can 
have different behaviours in corner cases.



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Reading Python source file

[Guido van Rossum]

On Tue, Nov 17, 2015 at 1:59 AM, M.-A. Lemburg  wrote:
> On 17.11.2015 02:53, Serhiy Storchaka wrote:
>> I'm working on rewriting Python tokenizer (in particular the part that reads 
>> and decodes Python
>> source file). The code is complicated. For now there are such cases:
>>
>> * Reading from the string in memory.
>> * Interactive reading from the file.
>> * Reading from the file:
>>   - Raw reading ignoring encoding in parser generator.
>>   - Raw reading UTF-8 encoded file.
>>   - Reading and recoding to UTF-8.
>>
>> The file is read by the line. It makes hard to check correctness of the 
>> first line if the encoding
>> is specified in the second line. And it makes very hard problems with null 
>> bytes and with
>> desynchronizing buffered C and Python files. All this problems can be easily 
>> solved if read all
>> Python source file in memory and then parse it as string. This would allow 
>> to drop a large complex
>> and buggy part of code.
>>
>> Are there disadvantages in this solution? As for memory consumption, the 
>> source text itself will
>> consume only small part of the memory consumed by AST tree and other 
>> structures. As for performance,
>> reading and decoding all file can be faster then by the line.
>
> A problem with this approach is that you can no
> longer fail early and detect indentation errors et al. while
> parsing the data (which may well come from a pipe).

Oh, this use case I had forgotten about. I don't know how common or
important it is though.

But more important is the interactive REPL, which parses your input
fully each time you hit ENTER.

> Another related problem is that you have to wait for the full
> input data before you can start compiling the code.

That's always the case -- we don't start compiling before we have the
full parse tree.

> I don't think these situations are all that common, though,
> so reading in the full source code before compiling it
> sounds like a reasonable approach.
>
> We use the same simplification in eGenix PyRun's emulation of
> the Python command line interface and it has so far not
> caused any problems.

Curious how you do it? I'd actually be quite disappointed if the
amount of parsing done by the standard REPL went down.

>> [1] http://bugs.python.org/issue25643

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Reading Python source file

[Serhiy Storchaka]

On 17.11.15 05:00, Guido van Rossum wrote:

If you free the memory used for the source buffer before starting code
generation you should be good.


Thank you. The buffer is freed just after the end of generating AST.


On Mon, Nov 16, 2015 at 5:53 PM, Serhiy Storchaka  wrote:

I'm working on rewriting Python tokenizer (in particular the part that reads
and decodes Python source file). The code is complicated. For now there are
such cases:

* Reading from the string in memory.
* Interactive reading from the file.
* Reading from the file:
   - Raw reading ignoring encoding in parser generator.
   - Raw reading UTF-8 encoded file.
   - Reading and recoding to UTF-8.

The file is read by the line. It makes hard to check correctness of the
first line if the encoding is specified in the second line. And it makes
very hard problems with null bytes and with desynchronizing buffered C and
Python files. All this problems can be easily solved if read all Python
source file in memory and then parse it as string. This would allow to drop
a large complex and buggy part of code.

Are there disadvantages in this solution? As for memory consumption, the
source text itself will consume only small part of the memory consumed by
AST tree and other structures. As for performance, reading and decoding all
file can be faster then by the line.

[1] http://bugs.python.org/issue25643

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/guido%40python.org







___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Reading Python source file

[Guido van Rossum]

On Tue, Nov 17, 2015 at 8:20 AM, Serhiy Storchaka  wrote:
> On 17.11.15 11:59, M.-A. Lemburg wrote:
>>
>> I don't think these situations are all that common, though,
>> so reading in the full source code before compiling it
>> sounds like a reasonable approach.
>>
>> We use the same simplification in eGenix PyRun's emulation of
>> the Python command line interface and it has so far not
>> caused any problems.
>
> Current implementation of import system went the same way. As a result
> importing the script as a module and running it with command line can have
> different behaviours in corner cases.

I'm confused. *Of course* these two behaviors differ, since Python
uses a different __name__. Not sure how this relates to the REPL.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Reading Python source file

[Guido van Rossum]

Oh, cool! Sorry for the disturbance.

On Tue, Nov 17, 2015 at 8:27 AM, Serhiy Storchaka  wrote:
> On 17.11.15 18:06, Guido van Rossum wrote:
>>
>> OK, but what are you going to do about the interactive REPL?
>
>
> Nothing (except some simplification). This is a separate branch of the code.
>
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance)

[Victor Stinner]

2015-11-17 1:00 GMT+01:00 Guido van Rossum :
> Hm, making Christian the BDFL-delegate would mean two out of three
> authors *and* the BDFL-delegate all working for Red Hat, which clearly
> has a stake (and IIUC has already committed to this approach ahead of
> PEP approval). SO then it would look like this is just rubber-stamping
> Red Hat's internal decision process (if it's a process -- sounds more
> like an accident :-).

Can we try to get a vote from maintainers of the Python2/3 packages of
other Linux distributions? Debian, Ubuntu, OpenSUSE, etc.?

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Benchmark results across all major Python implementations

[R. David Murray]

On Mon, 16 Nov 2015 23:37:06 +, "Stewart, David C" 
 wrote:
> Last June we started publishing a daily performance report of the latest 
> Python tip against the previous day's run and some established synch point. 
> We mail these to the community to act as a "canary in the coal mine." I wrote 
> about it at https://01.org/lp/blog/0-day-challenge-what-pulse-internet
> 
> You can see our manager-style dashboard of a couple of key workloads at 
> http://languagesperformance.intel.com/
> (I have this running constantly on a dedicated screen in my office).

Just took a look at this.  Pretty cool.  The web page is a bit confusing,
though.  It doesn't give any clue as to what is being measured by the
numbers presented...it isn't obvious whether those downward sloping
lines represent progress or regression.  Also, the intro talks about
historical data, but other than the older dates[*] in the graph there's
no access to it.  Do you have plans to provide access to the raw data?
It also doesn't show all of the test shown in the example email in your
blog post or the emails to python-checkins...do you plan to make those
graphs available in the future as well?

Also, in the emails, what is the PGO column percentage relative to?

I suppose that for this to have maximum effect someone would have to
specifically be paying attention to performance and figuring out why
every (real) regression happened.  I don't suppose we have anyone in the
community currently who is taking on that role, though we certainly do
have people who are *interested* in Python performance :)

--David

[*] Personally I'd find it easier to read those dates in MM-DD form,
but I suppose that's a US quirk, since in the US when using slashes
the month comes first...
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Benchmark results across all major Python implementations

[Stewart, David C]

+Stefan (owner of the 0-day lab)




On 11/17/15, 10:40 AM, "Python-Dev on behalf of R. David Murray" 
 wrote:

>On Mon, 16 Nov 2015 23:37:06 +, "Stewart, David C" 
> wrote:
>> Last June we started publishing a daily performance report of the latest 
>> Python tip against the previous day's run and some established synch point. 
>> We mail these to the community to act as a "canary in the coal mine." I 
>> wrote about it at https://01.org/lp/blog/0-day-challenge-what-pulse-internet
>> 
>> You can see our manager-style dashboard of a couple of key workloads at 
>> http://languagesperformance.intel.com/
>> (I have this running constantly on a dedicated screen in my office).
>
>Just took a look at this.  Pretty cool.  The web page is a bit confusing,
>though.  It doesn't give any clue as to what is being measured by the
>numbers presented...it isn't obvious whether those downward sloping
>lines represent progress or regression.  Also, the intro talks about
>historical data, but other than the older dates[*] in the graph there's
>no access to it.  Do you have plans to provide access to the raw data?
>It also doesn't show all of the test shown in the example email in your
>blog post or the emails to python-checkins...do you plan to make those
>graphs available in the future as well?

The data on this website has been normalized so "up" is "good" so far as the 
slope of the line. The daily email has a lot more detail about the hardware and 
software configuration and the versions being compared. We run workloads 
multiple times and visually show the relative standard distribution on the 
graph.

No plans to show the raw data.

I think showing multiple workloads graphically sounds useful, we should look 
into that.

>
>Also, in the emails, what is the PGO column percentage relative to?

It's the performance boost on the current rev from just using PGO. Another way 
to think about it is, this is the performance that you leave on the table by 
*not* building Cpython with PGO. For example, from last night's run, we would 
see an 18.54% boost in django_v2 by building Python using PGO.

Note: PGO is not the default way to build Python because it is relatively slow 
to compile it that way. (I think it should be the default). 

Here are the instructions for using it (thanks to Peter Wang for the 
instructions):

hg clone https://hg.python.org/cpython cpython
cd cpython
hg update 2.7
./configure
make profile-opt



>
>I suppose that for this to have maximum effect someone would have to
>specifically be paying attention to performance and figuring out why
>every (real) regression happened.  I don't suppose we have anyone in the
>community currently who is taking on that role, though we certainly do
>have people who are *interested* in Python performance :)

We're trying to fill that role as much as we can. When there is a significant 
(and unexplained) regression that we see, I usually ask our engineers to bisect 
it to identify the offending patch and root-cause it.

>
>--David
>
>[*] Personally I'd find it easier to read those dates in MM-DD form,
>but I suppose that's a US quirk, since in the US when using slashes
>the month comes first...

You and me both. As you surmised, the site was developed by our friends in 
Europe. :-)

>___
>Python-Dev mailing list
>Python-Dev@python.org
>https://mail.python.org/mailman/listinfo/python-dev
>Unsubscribe: 
>https://mail.python.org/mailman/options/python-dev/david.c.stewart%40intel.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Reading Python source file

[Guido van Rossum]

Aha, so the only code path that's being replaced is the code that
reads the script file when invoking "python FILE" or "python  wrote:
> On 18 November 2015 at 02:50, Guido van Rossum  wrote:
>> On Tue, Nov 17, 2015 at 8:20 AM, Serhiy Storchaka  
>> wrote:
>>> On 17.11.15 11:59, M.-A. Lemburg wrote:

 I don't think these situations are all that common, though,
 so reading in the full source code before compiling it
 sounds like a reasonable approach.

 We use the same simplification in eGenix PyRun's emulation of
 the Python command line interface and it has so far not
 caused any problems.
>>>
>>> Current implementation of import system went the same way. As a result
>>> importing the script as a module and running it with command line can have
>>> different behaviours in corner cases.
>>
>> I'm confused. *Of course* these two behaviors differ, since Python
>> uses a different __name__. Not sure how this relates to the REPL.
>
> I think Serhiy was referring to the fact that importlib already reads
> the entire file before compiling it - since the import abstraction
> doesn't require modules to be files on disk, it uses the get_code()
> API on the module loader internally, which is typically implemented by
> calling get_source() and then passing the result to compile().
>
> That behaviour is then inherited at the command line by both the -m
> switch and the support for executing directories and zip archives.
> When we consider that the "-c" switch also executes an in-memory
> string, direct script execution is currently the odd one out in *not*
> reading the entire source file into memory first, so Serhiy's proposed
> simplification of the implementation makes sense to me.
>
> Regards,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Reading Python source file

[Nick Coghlan]

On 18 November 2015 at 02:50, Guido van Rossum  wrote:
> On Tue, Nov 17, 2015 at 8:20 AM, Serhiy Storchaka  wrote:
>> On 17.11.15 11:59, M.-A. Lemburg wrote:
>>>
>>> I don't think these situations are all that common, though,
>>> so reading in the full source code before compiling it
>>> sounds like a reasonable approach.
>>>
>>> We use the same simplification in eGenix PyRun's emulation of
>>> the Python command line interface and it has so far not
>>> caused any problems.
>>
>> Current implementation of import system went the same way. As a result
>> importing the script as a module and running it with command line can have
>> different behaviours in corner cases.
>
> I'm confused. *Of course* these two behaviors differ, since Python
> uses a different __name__. Not sure how this relates to the REPL.

I think Serhiy was referring to the fact that importlib already reads
the entire file before compiling it - since the import abstraction
doesn't require modules to be files on disk, it uses the get_code()
API on the module loader internally, which is typically implemented by
calling get_source() and then passing the result to compile().

That behaviour is then inherited at the command line by both the -m
switch and the support for executing directories and zip archives.
When we consider that the "-c" switch also executes an in-memory
string, direct script execution is currently the odd one out in *not*
reading the entire source file into memory first, so Serhiy's proposed
simplification of the implementation makes sense to me.

Regards,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Reading Python source file

[M.-A. Lemburg]

On 17.11.2015 02:53, Serhiy Storchaka wrote:
> I'm working on rewriting Python tokenizer (in particular the part that reads 
> and decodes Python
> source file). The code is complicated. For now there are such cases:
> 
> * Reading from the string in memory.
> * Interactive reading from the file.
> * Reading from the file:
>   - Raw reading ignoring encoding in parser generator.
>   - Raw reading UTF-8 encoded file.
>   - Reading and recoding to UTF-8.
> 
> The file is read by the line. It makes hard to check correctness of the first 
> line if the encoding
> is specified in the second line. And it makes very hard problems with null 
> bytes and with
> desynchronizing buffered C and Python files. All this problems can be easily 
> solved if read all
> Python source file in memory and then parse it as string. This would allow to 
> drop a large complex
> and buggy part of code.
> 
> Are there disadvantages in this solution? As for memory consumption, the 
> source text itself will
> consume only small part of the memory consumed by AST tree and other 
> structures. As for performance,
> reading and decoding all file can be faster then by the line.

A problem with this approach is that you can no
longer fail early and detect indentation errors et al. while
parsing the data (which may well come from a pipe).

Another related problem is that you have to wait for the full
input data before you can start compiling the code.

I don't think these situations are all that common, though,
so reading in the full source code before compiling it
sounds like a reasonable approach.

We use the same simplification in eGenix PyRun's emulation of
the Python command line interface and it has so far not
caused any problems.

> [1] http://bugs.python.org/issue25643

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Nov 17 2015)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/

2015-10-23: Released mxODBC Connect 2.1.5 ... http://egenix.com/go85

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] Daily reference leaks (97e2a6810f7f): sum=10

[Victor Stinner]

Hi,

I opened the http://bugs.python.org/issue25645

It looks like a simple regression in _pickle. I wrote a patch.

Victor

2015-11-16 19:49 GMT+01:00 Brett Cannon :
> Just an FYI there seems to be a consistent, minor refcount leak found by
> test_capi that has been there for what seems like a couple of weeks.
>
> On Mon, 16 Nov 2015 at 00:42  wrote:
>>
>> results for 97e2a6810f7f on branch "default"
>> 
>>
>> test_asyncio leaked [0, 0, 3] memory blocks, sum=3
>> test_capi leaked [1, 1, 1] references, sum=3
>> test_functools leaked [0, 2, 2] memory blocks, sum=4
>>
>>
>> Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R',
>> '3:3:/home/psf-users/antoine/refleaks/reflogBLsY2a', '--timeout', '7200']
>> ___
>> Python-checkins mailing list
>> python-check...@python.org
>> https://mail.python.org/mailman/listinfo/python-checkins
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Benchmark results across all major Python implementations

[Popa, Stefan A]

Hi Python community,

Thank you for your feedback! We will look into this and come up with an e-mail 
format proposal in the following days.

Best regards,

--
Stefan A. POPA
Software Engineering Manager
System Technologies and Optimization Division
Software Services Group, Intel Romania

> On 17 Nov 2015, at 21:22, Stewart, David C  wrote:
> 
> +Stefan (owner of the 0-day lab)
> 
> 
> 
> 
>> On 11/17/15, 10:40 AM, "Python-Dev on behalf of R. David Murray" 
>> > rdmur...@bitdance.com> wrote:
>> 
>>> On Mon, 16 Nov 2015 23:37:06 +, "Stewart, David C" 
>>>  wrote:
>>> Last June we started publishing a daily performance report of the latest 
>>> Python tip against the previous day's run and some established synch point. 
>>> We mail these to the community to act as a "canary in the coal mine." I 
>>> wrote about it at https://01.org/lp/blog/0-day-challenge-what-pulse-internet
>>> 
>>> You can see our manager-style dashboard of a couple of key workloads at 
>>> http://languagesperformance.intel.com/
>>> (I have this running constantly on a dedicated screen in my office).
>> 
>> Just took a look at this.  Pretty cool.  The web page is a bit confusing,
>> though.  It doesn't give any clue as to what is being measured by the
>> numbers presented...it isn't obvious whether those downward sloping
>> lines represent progress or regression.  Also, the intro talks about
>> historical data, but other than the older dates[*] in the graph there's
>> no access to it.  Do you have plans to provide access to the raw data?
>> It also doesn't show all of the test shown in the example email in your
>> blog post or the emails to python-checkins...do you plan to make those
>> graphs available in the future as well?
> 
> The data on this website has been normalized so "up" is "good" so far as the 
> slope of the line. The daily email has a lot more detail about the hardware 
> and software configuration and the versions being compared. We run workloads 
> multiple times and visually show the relative standard distribution on the 
> graph.
> 
> No plans to show the raw data.
> 
> I think showing multiple workloads graphically sounds useful, we should look 
> into that.
> 
>> 
>> Also, in the emails, what is the PGO column percentage relative to?
> 
> It's the performance boost on the current rev from just using PGO. Another 
> way to think about it is, this is the performance that you leave on the table 
> by *not* building Cpython with PGO. For example, from last night's run, we 
> would see an 18.54% boost in django_v2 by building Python using PGO.
> 
> Note: PGO is not the default way to build Python because it is relatively 
> slow to compile it that way. (I think it should be the default). 
> 
> Here are the instructions for using it (thanks to Peter Wang for the 
> instructions):
> 
> hg clone https://hg.python.org/cpython cpython
> cd cpython
> hg update 2.7
> ./configure
> make profile-opt
> 
> 
> 
>> 
>> I suppose that for this to have maximum effect someone would have to
>> specifically be paying attention to performance and figuring out why
>> every (real) regression happened.  I don't suppose we have anyone in the
>> community currently who is taking on that role, though we certainly do
>> have people who are *interested* in Python performance :)
> 
> We're trying to fill that role as much as we can. When there is a significant 
> (and unexplained) regression that we see, I usually ask our engineers to 
> bisect it to identify the offending patch and root-cause it.
> 
>> 
>> --David
>> 
>> [*] Personally I'd find it easier to read those dates in MM-DD form,
>> but I suppose that's a US quirk, since in the US when using slashes
>> the month comes first...
> 
> You and me both. As you surmised, the site was developed by our friends in 
> Europe. :-)
> 
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: 
>> https://mail.python.org/mailman/options/python-dev/david.c.stewart%40intel.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Benchmark results across all major Python implementations

[Stephen J. Turnbull]

Stewart, David C writes:

 > Note: PGO is not the default way to build Python because it is
 > relatively slow to compile it that way. (I think it should be the
 > default).

+1

Slow-build-fast-run should be the default if you're sure the
optimization works.  Only developers are likely to run a given build
few enough times to save seconds, and most people are like to turn to
some other task as soon as they type "make".

It's a slightly different use case, but in XEmacs we have a
--quick-build configure option which means that the "usual targets"
don't rebuild a bunch of auxiliary targets (mostly documentation and
development infrastructure such as xref caches).  Never heard a
complaint about that either from the developers (who learned to use
--quick-build easily enough) or the beta testers (who do remark on
long build times, but only once a week or so for most of them).

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com