Re: [Python-Dev] Python Language Reference has no mention of list comÃprehensions

[Nick Coghlan]

On 4 December 2015 at 12:48, Andrew Barnert via Python-Dev
 wrote:
> On Dec 3, 2015, at 17:25, Steven D'Aprano  wrote:
>> On Thu, Dec 03, 2015 at 09:25:53AM -0800, Andrew Barnert via Python-Dev 
>> wrote:
>>> I've seen people saying that before, but I don't know where they get
>>> that. It's certainly not the way, say, C++ or JavaScript use the term.

I'm one of the folks that use it that way, but I learned that
terminology *from* the Python language reference.

>> I wouldn't take either of those two languages as examples of best
>> practices in language design :-)
>
> No, but they seem to be the languages (along with C and Java) that people 
> usually appeal to.
>
> You also found "literal" used the same way as JavaScript in Ruby, one of 
> three languages in your quick survey. It's also used similarly in ML and 
> Haskell. In Lisp, it has a completely different meaning (a quoted list).
>
> But as I said before, we can't use the word "literal" to contrast with 
> comprehensions, because a large segment of the Python community (including 
> you) would find that use of the word confusing and/or annoying because you 
> intuitively think of the C/FORTRAN/etc. definition rather than the 
> C++/Ruby/JS/Haskell definition. It doesn't matter whether that's a peculiar 
> quirk of the Python community or not, whether there's a good reason for it or 
> not, etc.; all that matters is that it's true.

Even though it's true, I'm not sure it's sufficient to rule out a
switch to "there are two kinds of literal" as the preferred
terminology.

The recent case that comes to mind is the new format string literals -
those can include arbitrary subexpressions, like container displays
and comprehensions, but the conclusion from the PEP 498 discussion was
that it makes the most sense to still consider them a kind of string
literal.

There's also a relatively straightforward way of defining the key
semantic different between a literal and a normal constructor call:
with a literal, there's no way to override the type of the resulting
object, while a constructor call can be monkeypatched like any other
callable.

The distinction that arises for containers is then the one that Chris
Angelico pointed out: a container literal may have constant content,
*or* it may have dynamic content. If we switched from calling things
"displays" to calling them "dynamic literals", I'd be surprised if too
many folks that were able to figure out what "display" meant struggled
to make the transition.

Summarising that idea:

* literals: any of the dedicated expressions that produce an instance
of a builtin type
* constant literal: literals that produce a constant object that can
be cached in the bytecode
* dynamic literal: literals containing dynamic subexpressions that
can't be pre-calculated
* display: legacy term for a dynamic literal (originally inherited from ABC)
* comprehension: a dynamic literal that creates a new container from
an existing iterable
* lexical literal: constant literals and dynamic string literals [1]

The ast.literal_eval() docs would need a slight adjustment to refer to
"literals (excluding container comprehensions and generator
expressions)", rather than the current "literals and container
displays".

Regards,
Nick.

[1] https://docs.python.org/dev/reference/lexical_analysis.html#literals

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python Language Reference has no mention of list comÃprehensions

[R. David Murray]

On Fri, 04 Dec 2015 18:38:03 +1000, Nick Coghlan  wrote:
> Summarising that idea:
> 
> * literals: any of the dedicated expressions that produce an instance
> of a builtin type
> * constant literal: literals that produce a constant object that can
> be cached in the bytecode
> * dynamic literal: literals containing dynamic subexpressions that
> can't be pre-calculated
> * display: legacy term for a dynamic literal (originally inherited from ABC)
> * comprehension: a dynamic literal that creates a new container from
> an existing iterable
> * lexical literal: constant literals and dynamic string literals [1]
> 
> The ast.literal_eval() docs would need a slight adjustment to refer to
> "literals (excluding container comprehensions and generator
> expressions)", rather than the current "literals and container
> displays".

Except that that isn't accurate either:

>>> import ast
>>> ast.literal_eval('[1, id(1)]')
Traceback (most recent call last):
  File "", line 1, in 
  File "/home/rdmurray/python/p36/Lib/ast.py", line 84, in literal_eval
return _convert(node_or_string)
  File "/home/rdmurray/python/p36/Lib/ast.py", line 57, in _convert
return list(map(_convert, node.elts))
  File "/home/rdmurray/python/p36/Lib/ast.py", line 83, in _convert
raise ValueError('malformed node or string: ' + repr(node))
ValueError: malformed node or string: <_ast.Call object at 0xb73633ec>

So it's really container displays consisting of literals, which we could
call a "literal container display".

I think the intuitive notion of "literal" is "the value is literally
what is written here".  Which is a redundant statement; 'as written' is,
after all, what literally means when used correctly :).  That makes it
a language-agnostic concept if I'm correct.

I think we will find that f strings are called f expressions, not f literals.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Third milestone of FAT Python

[Victor Stinner]

Hi,

I implemented 3 new optimizations in FAT Python: loop unrolling, constant
folding and copy builtin functions to constants. In the previous thread,
Terry Reedy asked me if the test suite is complete enough to ensure that
FAT Python doesn't break the Python semantic. I can now say that the
second milestone didn't optimize enough code to see bugs, new
optimizations helped me to find a *lot* of bugs which are now all fixed.

The full Python test suite pass with all optimizations enabled. Only two
tests are skipped in FAT Python: test_dynamic and mock tests of
test_unittest. test_dynamic checks that it's possible to replace builtin
functions in a function and then use the replaced builtins from the same
function. FAT Python currently doesn't support the specific case of this
test. The mock tests of test_unittest does something similar, I'm more
concerned by these failures.

This email is an updated version on my previous blog article:
https://haypo.github.io/fat-python-status-nov26-2015.html

Since I wrote this blog article, I implemented the constant folding
optimization and I fixed the two major bugs mentioned in the article
(line number and exec(code, dict)).


Documentation
=

I combined the documentation of (my) various optimizations projects into
a single documentation:
http://faster-cpython.readthedocs.org/

The FAT Python project has its own page:
http://faster-cpython.readthedocs.org/fat_python.html


Constant folding


This optimization propagates constant values of variables. Example:

def func()
x = 1
y = x
return y

Constant folding:

def func()
x = 1
y = 1
return 1

This optimization alone is not really exciting. It will more used later
when the optimizer will implement peephole optimizations (ex: a+b) and
remove dead code. For example, it will be possible to remove code
specific to a platform
(ex: 'if sys.platform.startswith("freebsd"): ...').

Later, removal of unused local variables will be implemented to simplify
the code even more. The previous example will be simplified to:

def func():
return 1


Loop unrolling optimization
===

The optimization generates assignement statements (for the loop index
variable) and duplicates the loop body to reduce the cost of loops. Example:

def func():
for i in range(2):
print(i)

Loop unrolled:

def func():
i = 0
print(i)

i = 1
print(i)

If the iterator uses the builtin range function, two guards are
required on builtins and globals namespaces.

The optimization also handles tuple iterator
(ex: "for i in (1, 2, 3): ..."). No guard is needed in this case (the
code is always optimized).

Loop unrolling combines well with constant folding. The previous example
is simplified to:

def func():
i = 0
print(0)

i = 1
print(1)

Again, with a future removal of unused local variables optimization, the
previous example will be simplified to:

def func():
print(0)
print(1)


Copy builtins to constants optimization
===

This optimization is currently disabled by default. (Well, in practice,
it's enabled by the site module to test it and detect code which doesn't
work with it.)

The LOAD_GLOBAL instruction is used to load a builtin function. The
instruction requires two dictionary lookup: one in the globals namespace
(which almost always fail) and then in the builtins namespace.

It's rare to replace builtins, so the idea here is to replace the dynamic
LOAD_GLOBAL instruction with a static LOAD_CONST instruction which
loads the function from a C array, a fast O(1) access.

It is not possible to inject a builtin function during the compilation.
Python code objects are serialized by the marshal module which only
support simple types like integers, strings and tuples, not functions.
The trick is to modify the constants at runtime when the module is
loaded. I added a new patch_constants() method to functions.

Example:

def log(message):
print(message)

This function is specialized to:

def log(message):
'LOAD_GLOBAL print'(message)
log.patch_constants({'LOAD_GLOBAL print': print})

The specialized bytecode uses two guards on builtins and globals
namespaces to disable the optimization if the builtin function is
replaced.

This optimization doesn't support the case when builtins are modified
while the function is executed. I think that it will be safer to disable
the optimization by default.

Later, we can enhance the optimization to enable it if the function
cannot modify builtins and if it only calls funcions which cannot modify
builtins. I bet that the optimization will be safe with these additional checks.


Changes on builtin guards
=

When a guard is used on a builtin function, the specialization of the
function is now ignored if the builtin was 

[Python-Dev] Summary of Python tracker Issues

[Python tracker]

ACTIVITY SUMMARY (2015-11-27 - 2015-12-04)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open5289 (+26)
  closed 32266 (+28)
  total  37555 (+54)

Open issues with patches: 2332 


Issues opened (37)
==

#25744: Reference leaks in test_collections
http://bugs.python.org/issue25744  opened by serhiy.storchaka

#25745: Reference leaks in test_curses
http://bugs.python.org/issue25745  opened by serhiy.storchaka

#25746: test_unittest failure in leaks searching mode
http://bugs.python.org/issue25746  opened by serhiy.storchaka

#25747: test_idle failure in leaks searching mode
http://bugs.python.org/issue25747  opened by serhiy.storchaka

#25749: asyncio.Server class documented but not exported
http://bugs.python.org/issue25749  opened by Ron Frederick

#25750: tp_descr_get(self, obj, type) is called without owning a refer
http://bugs.python.org/issue25750  opened by jdemeyer

#25752: asyncio.readline - add customizable line separator
http://bugs.python.org/issue25752  opened by mmarkk

#25753: Reference leaks in test_smtplib
http://bugs.python.org/issue25753  opened by serhiy.storchaka

#25755: Test test_property failed if run twice
http://bugs.python.org/issue25755  opened by serhiy.storchaka

#25757: Subclasses of property lose docstring
http://bugs.python.org/issue25757  opened by torsten

#25758: ensurepip/venv broken on Windows if path includes unicode
http://bugs.python.org/issue25758  opened by Dima.Tisnek

#25759: Python 2.7.11rc1 not building with Visual Studio 2015
http://bugs.python.org/issue25759  opened by kovidgoyal

#25761: Improve unpickling errors handling
http://bugs.python.org/issue25761  opened by serhiy.storchaka

#25764: PyObject_Call() is called with an exception set in subprocess
http://bugs.python.org/issue25764  opened by serhiy.storchaka

#25765: Installation error
http://bugs.python.org/issue25765  opened by ayushmaan121

#25766: __bytes__ doesn't work for str subclasses
http://bugs.python.org/issue25766  opened by serhiy.storchaka

#25768: compileall functions do not document return values
http://bugs.python.org/issue25768  opened by Nicholas Chammas

#25769: Crash due to using weakref referent without acquiring a strong
http://bugs.python.org/issue25769  opened by ldeller

#25770: expose name, args, and kwargs from methodcaller
http://bugs.python.org/issue25770  opened by ll

#25771: importlib: '.submodule' is not a relative name (no leading dot
http://bugs.python.org/issue25771  opened by martin.panter

#25773: Deprecate deleting with PyObject_SetAttr, PyObject_SetAttrStri
http://bugs.python.org/issue25773  opened by serhiy.storchaka

#25774: [benchmarks] Adjust to allow uploading benchmark data to codes
http://bugs.python.org/issue25774  opened by zach.ware

#25776: More compact pickle of iterators etc
http://bugs.python.org/issue25776  opened by serhiy.storchaka

#25777: Misleading descriptions in docs about invoking descriptors.
http://bugs.python.org/issue25777  opened by Juchen Zeng

#25778: winreg.EnumValue does not truncate strings correctly
http://bugs.python.org/issue25778  opened by anshul6

#25780: Add support for CAN_RAW_JOIN_FILTERS
http://bugs.python.org/issue25780  opened by rumpelsepp

#25782: CPython hangs on error __context__ set to the error itself
http://bugs.python.org/issue25782  opened by yselivanov

#25783: test_traceback.test_walk_stack() fails when run directly (with
http://bugs.python.org/issue25783  opened by haypo

#25785: TimedRotatingFileHandler missing rotations
http://bugs.python.org/issue25785  opened by felipecruz

#25786: contextlib.ExitStack introduces a cycle in exception __context
http://bugs.python.org/issue25786  opened by yselivanov

#25787: Add an explanation what happens with subprocess parent and chi
http://bugs.python.org/issue25787  opened by krichter

#25788: fileinput.hook_encoded has no way to pass arguments to codecs
http://bugs.python.org/issue25788  opened by lac

#25789: py launcher stderr is not piped to subprocess.Popen.stderr
http://bugs.python.org/issue25789  opened by wolma

#25791: Raise an ImportWarning when __spec__.parent/__package__ isn't 
http://bugs.python.org/issue25791  opened by brett.cannon

#25794: __setattr__ does not always overload operators
http://bugs.python.org/issue25794  opened by Dominik Schmid

#25795: test_fork1 cannot be run directly: ./pyhon Lib/test/test_fork1
http://bugs.python.org/issue25795  opened by haypo

#25796: Running test_multiprocessing_spawn is slow (more than 8 minute
http://bugs.python.org/issue25796  opened by haypo



Most recent 15 issues with no replies (15)
==

#25795: test_fork1 cannot be run directly: ./pyhon Lib/test/test_fork1
http://bugs.python.org/issue25795

#25791: Raise an ImportWarning when __spec__.parent/__package__ isn't 
http://bugs.python.org/issue25791

#25785: 

Re: [Python-Dev] Third milestone of FAT Python

[MRAB]

On 2015-12-04 19:22, Isaac Morland wrote:

On Fri, 4 Dec 2015, MRAB wrote:

> Constant folding is when, say, "1 + 2" replaced by "2".

Isn't that called backspacing?  ;-)


Oops! I meant "1 + 1", of course. Or "3". Either would work. :-)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python Language Reference has no mention of list comÃprehensions

[Eric Fahlgren]

David R. Murray wrote: 
> I think the intuitive notion of "literal" is "the value is literally what is 
> written
> here".  Which is a redundant statement; 'as written' is, after all, what 
> literally
> means when used correctly :).  That makes it a language-agnostic concept if 
> I'm
> correct.

So { x : 1 } is not literally a literal, it's figuratively a literal, or more 
simply a figurative.

Eric

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Third milestone of FAT Python

[Victor Stinner]

2015-12-04 20:16 GMT+01:00 MRAB :
> On 2015-12-04 12:49, Victor Stinner wrote:
> [snip]
>
> I don't think that's constant folding, but constant _propagation_.
>
> Constant folding is when, say, "1 + 2" replaced by "2".

Oh, you're right. I update the documentation. To avoid confusion, I
just implemented constant folding as well :-D

https://faster-cpython.readthedocs.org/fat_python.html#constant-folding

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python Language Reference has no mention of list comÃprehensions

[Andrew Barnert via Python-Dev]

On Dec 4, 2015, at 00:38, Nick Coghlan  wrote:
> 
> On 4 December 2015 at 12:48, Andrew Barnert via Python-Dev
>  wrote:
>> On Dec 3, 2015, at 17:25, Steven D'Aprano  wrote:
 On Thu, Dec 03, 2015 at 09:25:53AM -0800, Andrew Barnert via Python-Dev 
 wrote:
 I've seen people saying that before, but I don't know where they get
 that. It's certainly not the way, say, C++ or JavaScript use the term.
> 
> I'm one of the folks that use it that way, but I learned that
> terminology *from* the Python language reference.

If that's the usual case, then isn't it almost certainly more true for 
"display" than for "literal"? I doubt most Python users came in with a 
pre-existing notion of "display" from another language, or from programming in 
general--or, if they did, it's probably one of the senses that's irrelevant 
enough to not confuse anyone (like a repr, or a string formatting template). So 
if you want to redefine one of our terms to allow a new distinction, why not 
that one?

More importantly, as I said in my other message: do we actually need to be able 
to make this distinction? The problem this thread set out to solve is that 
"comprehension" doesn't have a docs section because it's just a subset of 
displays, so you can't search for it. Making it a subset of dynamic literals, 
which is a subset of literals, seems like it gets us farther from a solution. 
Right now, we could easily change the section title to "list displays 
(including comprehensions)" and we're done.

>>> I wouldn't take either of those two languages as examples of best
>>> practices in language design :-)
>> 
>> No, but they seem to be the languages (along with C and Java) that people 
>> usually appeal to.
>> 
>> You also found "literal" used the same way as JavaScript in Ruby, one of 
>> three languages in your quick survey. It's also used similarly in ML and 
>> Haskell. In Lisp, it has a completely different meaning (a quoted list).
>> 
>> But as I said before, we can't use the word "literal" to contrast with 
>> comprehensions, because a large segment of the Python community (including 
>> you) would find that use of the word confusing and/or annoying because you 
>> intuitively think of the C/FORTRAN/etc. definition rather than the 
>> C++/Ruby/JS/Haskell definition. It doesn't matter whether that's a peculiar 
>> quirk of the Python community or not, whether there's a good reason for it 
>> or not, etc.; all that matters is that it's true.
> 
> Even though it's true, I'm not sure it's sufficient to rule out a
> switch to "there are two kinds of literal" as the preferred
> terminology.
> 
> The recent case that comes to mind is the new format string literals -
> those can include arbitrary subexpressions, like container displays
> and comprehensions, but the conclusion from the PEP 498 discussion was
> that it makes the most sense to still consider them a kind of string
> literal.
> 
> There's also a relatively straightforward way of defining the key
> semantic different between a literal and a normal constructor call:
> with a literal, there's no way to override the type of the resulting
> object, while a constructor call can be monkeypatched like any other
> callable.

Is that an important distinction to anyone but people who write Python 
implementations? If some library I'm using chooses to monkeypatch or shadow a 
type name, the objects are still going to quack the way I expect (otherwise, 
I'm going to stop using that library pretty quickly). And meanwhile, why do I 
need to distinguish between libraries that monkeypatch the stdlib for me and 
libraries that install an import hook to patch my code?

It's certainly not meaningless or completely useless (e.g., the discussion 
about whether f-strings are literals would have been shorter, and had more of a 
point), but it doesn't seem useful enough to be worth redefining existing 
terminology.

> The distinction that arises for containers is then the one that Chris
> Angelico pointed out: a container literal may have constant content,
> *or* it may have dynamic content.

Well, yes, but, again, both forms of container literal can have dynamic 
content: [f(), g()] is just as dynamic as [x() for x in (f, g)]. So we still 
don't have the contrast we were looking for.

Also, [1, 2] is literal, and not dynamic, but it's not a constant value, so 
calling it a constant literal seems likely to be more confusing than helpful.

One more thing: we don't have to worry about whether def and class are literal 
constructs because they're not expressions, but what about lambda? It fits your 
definition of literal. JS calls them literals (Ruby is a bit confusing because 
it splits the notion of function into three entirely independent things), as do 
some functional languages. And this means you can write a function-table dict 
that's all literals. As for whether it's constant--the code object obviously 
can be 

Re: [Python-Dev] Third milestone of FAT Python

[MRAB]

On 2015-12-04 12:49, Victor Stinner wrote:
[snip]


Constant folding


This optimization propagates constant values of variables. Example:

 def func()
 x = 1
 y = x
 return y

Constant folding:

 def func()
 x = 1
 y = 1
 return 1


[snip]

I don't think that's constant folding, but constant _propagation_.

Constant folding is when, say, "1 + 2" replaced by "2".

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Third milestone of FAT Python

[Isaac Morland]

On Fri, 4 Dec 2015, MRAB wrote:


Constant folding is when, say, "1 + 2" replaced by "2".


Isn't that called backspacing?  ;-)

Isaac Morland   CSCF Web Guru
DC 2619, x36650 WWW Software Specialist
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com