Re: Looping [was Re: Python and the need for speed]

[Steve D'Aprano]

On Wed, 11 Oct 2017 10:57 pm, Stefan Ram wrote:

>  FWIW, in is book "Touch of Class" (2009) Bertrand Meyer writes:
> 
> |Such instructions are just the old goto in sheep's clothing.
> |Treat them the same way as the original:
> |
> |/Touch of Methodology/:
> | Sticking to one-entry, one-exit building blocks
> |Stay away from any "break" or similar control mechanism.

I have a great deal of respect for Meyer, but in this case I think he has it
badly wrong on both counts (`break` and "one-entry, one-exit").

Unrestricted GOTO (as in BASIC, circa 1970, where you could jump to any line
in the code, or as in assembly) is rightly considered harmful (albeit
necessary in assembly). But if Meyer means *any* sort of jump ("similar
control mechanism"), then that would rule out not just `break` and `continue`
but also:

* while loops
* for loops
* if...else
* case/switch statements
* calling subroutines (functions or procedures)
* exception handling

Remember that a function call is basically a GOTO in disguise: execution jumps
to the function, and jumps back when the function returns. In BASIC, there
was a GOSUB intermediate between a GOTO and a procedure call.

Surely Meyer doesn't mean to say we should never call functions or use
`while`, `for` or `if`. So he has to distinguish between kinds of jumps:

Good jumps (I presume):

* function calls
* if...else
* looping

Evil jumps:

* unrestricted BASIC-style GOTO/GOSUB any line number
* break/continue

Not sure:

* GOTO where there are restrictions on where you can jump
* COMEFROM

(I kid: I'm sure Meyer would oppose COMEFROM, and I expect that even
Pascal-style restricted GOTO would be on his "evil" list to avoid.)

So the question becomes, why are such harmless, simple to understand,
innocuous jumps like `break` and `continue` in the evil list, when they not
only simplify code but make it more efficient?

# with break
for i in range(2**64):
if isprime(i):
print(i, "is prime")
break

# without break
still_searching = True
for i in range(2**64):
if still_searching and isprime(i):
print(i, "is prime")
still_searching = False

# without break, attempt number 2
still_searching = True
i = 0
while still_searching and i < 2**64:
if isprime(i):
print(i, "is prime")
still_searching = False


Unrestricted jumps *into* a subroutine are hard to reason about. In general,
subroutines should have a single entry point. But the requirement for one
exit is too strict. From the caller's perspective, there is no way to tell
how many exit point a function has: subroutines are black boxes that the
caller cannot see into, and the difference between a single exit and multiple
exits is invisible.

But from the point of view of the subroutines, the rule "one exit" is like the
rule "no break" for loops: it makes the code more complex and less efficient.
If you're done, you're done, and you might as well return out of the function
rather than write boilerplate code to pad it out until you get to the very
end. With only a single condition to test, there's not much difference
between the two:

# with single exit  # with multiple exits
def foo():  def foo():
if condition:   if condition:
result = 1  return 1
else:   return 2
result = 2
return result


but as the number of decision points increase, the complexity required to keep
a single exit also increases.
  
Ironically, after telling us to stick to code with one entry and one exit,
Meyer then contradicts himself by recommending exceptions:

> |You can use exception handling as a technique of last resort
> |to handle unexpected events for which the normal control
> |structures let you down.

Even more ironically, exception handling is most similar to a COMEFROM, which
was invented as a joke.



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Static typing [was Re: Python and the need for speed]

[Paul Rubin]

Gregory Ewing  writes:
> A JIT compiler works by observing the actual values

To be pedantic, that's called a "tracing JIT".  Other runtime code
generation is also frequently called JIT compilation even when it's
fairly stupid combining of assembly code templates, or the like.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Looping [was Re: Python and the need for speed]

[Paul Rubin]

Chris Angelico  writes:
> while True:
> c = sys.stdin.read(1)
> if not c: break
> if c.isprintable(): text += c
> elif c == "\x08": text = text[:-1]
> # etc
> Can you write _that_ as a do-while?

I prefer to write that sort of thing with iterators:

 for c in iter(lambda: sys.stdin.read(1), ''):
 if c.isprintable(): text.append(c)
 elif c == '\x08': text.pop()
 ...
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python and the need for speed

[bartc]

On 19/04/2017 17:23, Marko Rauhamaa wrote:

bartc :

Enough works in that 'pcc' project, in terms of file i/o, that it can
still run plenty of useful programs, such as compilers.


This might have come up before, but do you have a language specification
somewhere?


(Nothing formal nor up-to-date.

(In the absence of other users, or enough people using programs written 
in it in used by others, then it is easy for the project to be volatile.


Previous, simpler versions I used in the 90s, which were part of a 
graphics app, /were/ used by others, both the language and the programs 
I and they wrote. So it had to be more stable. It even had a 350-page 
manual, created with the same graphics app of course.)


I'll try and throw something together, because it could do with it.)

--
bartc
--
https://mail.python.org/mailman/listinfo/python-list

Re: Python and the need for speed

[Marko Rauhamaa]

bartc :
> Enough works in that 'pcc' project, in terms of file i/o, that it can
> still run plenty of useful programs, such as compilers.

This might have come up before, but do you have a language specification
somewhere?


Marko
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python and the need for speed

[bartc]

On 19/04/2017 15:35, Chris Angelico wrote:

On Wed, Apr 19, 2017 at 11:46 PM, bartc  wrote:

You'd be surprised how easy it is to be non-OS-neutral.


I misread that as 'easy to be OS-neutral'. If I turn it around, you're 
saying it is easy to be OS-specific. But we know that! And that is the 
problem.



It's not so simple. By OS-neutral I mean code that doesn't depend on special
features of either OS (Ie. Windows and Linux). Not conditional code that
does either Windows stuff or Linux stuff.



Which means, therefore, that you either can't have any file system
facilities, or must transparently expose them to the application. You
also will have a lot of trouble making cross-platform subprocess
functionality. Even sockets, which are notionally the same on all
platforms, have a lot of little differences.

To be truly OS-neutral, you basically have to be either nerfed to
nothing, or so thin you're doing nothing.


Enough works in that 'pcc' project, in terms of file i/o, that it can 
still run plenty of useful programs, such as compilers. (See from line 
3357 of pcc64.c for example; a list of functions that are patched in to 
the host rather linked to an external msvcrt or libc.so shared library.)


In this project, the only OS-specifics it really needs are:

 Windows: use LoadLibrary/GetProcAddr, and msvcrt.dll
 Linux:   use dlopen/dlsym, and libc.so.6

which wouldn't be hard to arrange.

Any other specifics are in the domain of the user programs (different 
libraries for screen i/o for example).


However, getting it to choose between one of those two pairs of 
functions is trivial, compared with the dependencies of a a typical open 
source project.


(I've just downloaded CPython sources from your github link, and the 
first thing I see is an 18,000 line 'configure' script (somewhat bigger 
than my entire C compiler).


This doesn't run on Windows, not without importing most of Linux via 
cygwin or MSYS. But the sources do now appear to support a Windows build:


Step 1: Download Visual Studio Express 2015

Last time I looked, it seemed to be a 5GB to 11GB download (GB not MB!). 
Nevertheless I start the process to get a better idea. It says:


 "Visual Studio requires .NET Framework 4.6 or higher."

Here where I start to think it might be easier to download all the Linux 
stuff after all! As, with my track record, I could download tons of 
stuff, and it still won't work.


FWIW, my mcc64.c file compiles to a 0.2MB executable which is capable of 
compiling that pcc64.c interpreter (I think it does it in 50msec for 
good measure; Tiny C will also do it instantly). /That/ is how simple 
things should be, and can be.


Cue a million excuses for why things have to be the way they are...)

--
bartc
--
https://mail.python.org/mailman/listinfo/python-list

Re: Python and the need for speed

[Chris Angelico]

On Wed, Apr 19, 2017 at 11:46 PM, bartc  wrote:
>> You'd be surprised how easy it is to be non-OS-neutral.
>
> It's not so simple. By OS-neutral I mean code that doesn't depend on special
> features of either OS (Ie. Windows and Linux). Not conditional code that
> does either Windows stuff or Linux stuff.
>

Which means, therefore, that you either can't have any file system
facilities, or must transparently expose them to the application. You
also will have a lot of trouble making cross-platform subprocess
functionality. Even sockets, which are notionally the same on all
platforms, have a lot of little differences.

To be truly OS-neutral, you basically have to be either nerfed to
nothing, or so thin you're doing nothing.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python and the need for speed

[bartc]

On 19/04/2017 12:27, Chris Angelico wrote:

On Wed, Apr 19, 2017 at 8:33 PM, bartc  wrote:


[Warning: this is nothing to do with Python.]


My interpreter is on github as /one/ C source file (a link would be
inappropriate here). People can compile it with -O3 or -O2 if they wish.
It's a bit simpler than building CPython, and OS-neutral; that was
deliberate.


Then send me a link, and I'll try to compile it. You'd be surprised
how easy it is to be non-OS-neutral. Have you compiled it on the three
major platforms of today (Lin/Win/Mac)?


It should work on Windows and Linux, but I've only tested with gcc and 
Tiny C recently. I don't have a Mac.


The original link was for 32-bits, as that tended to be faster, but I've 
redone versions for both: click on either pcc32.c or pcc64.c here (and 
do what you have to do to download to your machine):


https://github.com/bartg/langs/tree/master/qlang

Build instructions are in the sources, but are basically just:

   gcc pcc64.c -opcc -lm

Test using:

  ./pcc hello # .q is assumed

I provide hello.q, or you can key it in; create a file hello.q with:

  proc start = println "Hello, World" end

To test on something bigger, download mc.qa (a composite of 27 .q 
files), and try that:


  ./pcc mc.qa

(mc.qa is an entire compiler for my static language; it should 
(byte-code) compile instantly then give usage instructions. To try it, 
copy hello.q to hello.m - the two languages conveniently have the same 
syntax - then try:


  ./pcc mc.qa hello   # defaults to .asm output I think
  ./pcc mc.qa /c64 hello  # hello.m to hello.c

I think it took 10ms to compile the 23Kloc of mc.qa to byte-code on my 
current machine.)


If you want another example of a trivial-to-build project, try:

   https://github.com/bartg/langs/blob/master/bccproj/mcc64.c

This is my C compiler in one source file (although not finished). Also 
OS-neutral (to run; however generated code runs on Win64).


This is how I wish other people would distributed their open source 
projects!



> You'd be surprised how easy it is to be non-OS-neutral.

It's not so simple. By OS-neutral I mean code that doesn't depend on 
special features of either OS (Ie. Windows and Linux). Not conditional 
code that does either Windows stuff or Linux stuff.


This means the 'pcc' program above is restricted (it needs either 
LoadLibrary/GetProcAddr or dlopen/dlsym to be able to access external 
libraries, although essential functions are patched in).



--
bartc
--
https://mail.python.org/mailman/listinfo/python-list

Re: Looping [was Re: Python and the need for speed]

[Antoon Pardon]

Op 16-04-17 om 19:07 schreef Terry Reedy:
> On 4/16/2017 11:35 AM, Michael Torrie wrote:
>> On 04/16/2017 07:57 AM, bartc wrote:
>>> But people just don't want it.
>>>
>>> /That/ is what surprises me, when people reject things that to me are
>>> no-brainers.
>
> Whereas to me, it is a no-brainer that we are better off *without*
> multiple while/loop constructs.
>
>> I simply don't care about these missing loop constructs.
>
> I do ;-)  I consider the current simplicity a feature.
>
> > Python works
>> great for what I use it for, and apparently works well for many people.
>
> The great majority* of 'repetition with variation' is sequentially
> processing items from a collection.  Python does that nicely with 'for
> item in collection: process(item)'.  While-loops take care of
> everthing else.

Not really, unless you count on the break statement.
But if you count on that, you don't even need a while,
you can start a for loop with a generator that never
stops and use breaks.

There was a time something like the following was
seriously considered for introduction in the language.

do
part1
while condition:
part2

which would be equivalent to the following:

while True:
part1
if not condition:
break
part2

But suddenly this was no longer considered. I still
wish they had followed through. I think such a construct
comes up often enough, to have such a loop construct.


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python and the need for speed

[Marko Rauhamaa]

Chris Angelico :

> On Wed, Apr 19, 2017 at 8:33 PM, bartc  wrote:
> You'd be surprised how easy it is to be non-OS-neutral. Have you
> compiled it on the three major platforms of today (Lin/Win/Mac)?

Generally, I don't try to be OS-neutral because

 1. I need it for Linux only

 2. I don't have Windows and Mac machines to test stuff on

 3. Windows and Linux are completely different system-programming-wise


Marko
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python and the need for speed

[Chris Angelico]

On Wed, Apr 19, 2017 at 8:33 PM, bartc  wrote:
> My interpreter is on github as /one/ C source file (a link would be
> inappropriate here). People can compile it with -O3 or -O2 if they wish.
> It's a bit simpler than building CPython, and OS-neutral; that was
> deliberate.

Then send me a link, and I'll try to compile it. You'd be surprised
how easy it is to be non-OS-neutral. Have you compiled it on the three
major platforms of today (Lin/Win/Mac)?

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python and the need for speed

[bartc]

On 19/04/2017 01:07, Erik wrote:

On 19/04/17 00:33, bartc wrote:


[Talking about an interpreter that is /not/ for Python]


With the sort of lower level programs I write (in another dynamic
language not Python), such an assembly layer improved performance 2-3
times over using 100% HLL compiled using C and gcc-O3.


Did you give the C compiler enough hints though?


If writing native C then I can spend a lot of time tweaking different 
things, and probably I could get a worthwhile improvement. But I doubt 
it would be double the speed.


But I anyway generate the C code from another language and I don't want 
it tied to a particular compiler or be full of specific pragmas and 
such, unless necessary. I don't need to because I can always bring out 
the asm version! [However that only works on Win64 platform. I use C for 
portability.]



Also, remember that -O3 might (and by that I mean probably will! ;))
make your code larger. If you have some specific core areas of your
interpreter that are now large enough to cause instruction cache misses
then a smaller -O2 (or even -Os) compiled version might perform better
on your hardware.


My interpreter is on github as /one/ C source file (a link would be 
inappropriate here). People can compile it with -O3 or -O2 if they wish. 
It's a bit simpler than building CPython, and OS-neutral; that was 
deliberate.


--
Bartc
--
https://mail.python.org/mailman/listinfo/python-list

Re: Python and the need for speed

[Erik]

On 19/04/17 00:33, bartc wrote:

So that's 'label-pointers' which I assume must correspond to computed
goto.


Yes - just different terminology. Being able to take the address of a 
label and "goto address" rather than "goto label".



(I don't know why they should be faster than a switch; they just
are.)


In C, the code generated for a switch() can do almost anything. It might 
cache the expression in a temporary and generate a linear "if .. else if 
.. else if ... else if" sequence (which is probably quite common for a 
sparsely populated set of values, after an initial range check is done) 
or it might generate a jump table (similar to the computed gotos) (which 
is probably quite common for a densely populated set of values, after an 
initial range check is done and an adjustment to make the index 
zero-based). It could also generate code that is some sort of hybrid of 
the two (for example, a switch with several densely-populated areas in 
an otherwise sparse set of values might do linear range-checks to find 
the right jump-table to use for each dense area).


What the computed goto stuff does is effectively reduce all of that to a 
single jump table with a known set of indices - there are 256 opcodes 
starting from 0 and each has an entry in an array of code pointers. To 
jump to the handler for an opcode, just 'goto handlers[op]'. No range 
checking, no index adjustment, no linear test sequences. This is what 
makes the dispatch fast.



With the sort of lower level programs I write (in another dynamic
language not Python), such an assembly layer improved performance 2-3
times over using 100% HLL compiled using C and gcc-O3.


Did you give the C compiler enough hints though? Much like the above 
computed-goto stuff (which is less of a hint and more of an instruction 
that the compiler should be using jump tables and nothing else for that 
bit of code) there are lots of other ways of spelling things differently 
in C that can give better performance than what you might get by default 
(and I'm not even talking about compiler-specific #pragmas or whatever).


Also, remember that -O3 might (and by that I mean probably will! ;)) 
make your code larger. If you have some specific core areas of your 
interpreter that are now large enough to cause instruction cache misses 
then a smaller -O2 (or even -Os) compiled version might perform better 
on your hardware.


E.
--
https://mail.python.org/mailman/listinfo/python-list

Re: Python and the need for speed

[bartc]

On 19/04/2017 00:19, Gregory Ewing wrote:

bartc wrote:



Another optimisation for something like ADD, was to to check for very
common types such as INT, and deal with those locally.


And then you're on the path to writing a JIT compiler. If you
can identify such cases, you might as well generate machine
code for them.


No, the fast code path to be executed already exists when writing the 
interpreter.


The decision to use it would be done on a per-byte-code basis.

Rather a crude way of speeding things up. If you're ADDing something 
that isn't two ints or is just a bit too complex (int+float for 
example), then that would slow down because of the extra check for two ints.


Where it gets more JITish is if it identifies that a sequence of 
byte-codes can be handled by dedicated code synthesised at runtime, 
which can also have hard-coded operands rather than picking them up from 
the byte-code stream.


But this can still be done during an initial pass. This is some way from 
the very ambitious tracing JITs that people are up to now.


(I'm not going down route because I specialise in doing things the easy 
way.)



--
bartc
--
https://mail.python.org/mailman/listinfo/python-list

Re: Static typing [was Re: Python and the need for speed]

[Gregory Ewing]

Steve D'Aprano wrote:


You seem to be describing a *tracing JIT* compiler.


Well, yes, but it seems to me that a JIT compiler that
*doesn't* use tracing is just an AOT compiler that you
happen to run immediately before executing the program.


Cython doesn't do any of that -- it's just a plain, boring,
standard ahead-of-time compiler that goes by the type info
you give it and nothing more.


That cannot be the full story, because Cython can optimize some regular,
unannotated Python code.


Maybe, but the point is that it does it using only
information present in the source. Also, it's usually
run once ahead of time to generate object code that's
stored.

I think we may be using different definitions of "JIT".
If you define it simply as running the compiler just
before you execute the code, then *some* ways of using
Cython might *barely* qualify -- you can set things up
so that a Python import will trigger compiling Cython
code if necessary.

But that doesn't seem like a useful way of defining JIT
to me. Such a compiler is not a different kind of beast
from an AOT compiler.

--
Greg
--
https://mail.python.org/mailman/listinfo/python-list

Re: Python and the need for speed

[bartc]

On 18/04/2017 23:33, Erik wrote:

On 18/04/17 11:30, bartc wrote:

On 18/04/2017 10:32, Erik wrote:


the
improvements over the original huge switch() to dispatch the bytecodes
to the correct handler appear to have made this type of optimization
less effective.


What did they do to it, and on which version?


It's the computed 'goto' stuff that I'm referring to. At the time, I was
looking at 2.7.9 (which just has the big switch) and 3.5.0 (which has
the computed gotos).


So that's 'label-pointers' which I assume must correspond to computed 
goto. (I don't know why they should be faster than a switch; they just are.)



Perhaps the Windows build process requires that things are stable and
doesn't support this type of development at all. You should take this
part of the discussion to python-dev if you want to be able to build and
experiment with it on Windows.


(OK, I've complained enough times in other groups about open source 
projects being too Linux-centric.)



If that had worked, then further optimisations are possible, such as
doing a pre-pass combining common operations, that would not be
worthwhile using 'official' byte-codes.)


That is effectively what my experiments were doing - sort of -, but
without introducing an assembler layer. I'm not convinced about that -
given the right hints, a half-decent C compiler these days will produce
pretty good code.


With the sort of lower level programs I write (in another dynamic 
language not Python), such an assembly layer improved performance 2-3 
times over using 100% HLL compiled using C and gcc-O3.


In fact the rest of the code didn't need such a hot compiler and didn't 
need to be in C.


With less numeric but still internally intensive tasks such as 
compilers, using the assembly layer was still over 50% faster than using 
C and gcc-O3.


However, with the very high level programs that many people (not me!) 
like to write in Python, any such advantages would narrow I think.


--
Bartc
--
https://mail.python.org/mailman/listinfo/python-list

Re: Python and the need for speed

[Erik]

On 19/04/17 00:08, Gregory Ewing wrote:

Erik wrote:

When considering special-casing this opcode sequence, remember that
in-place operations can be performed on anonymous objects (i.e., those
referenced by a collection and not bound directly to a namespace):


I think this means you would want multiple versions of the +=1 opcode:
ADD_ONE_TO_NAME, ADD_ONE_TO_ITEM, ADD_ONE_TO_ATTR, etc. Essentially
one for each variant of the STORE operation.


Yes, I get that ;) That's what I meant by a "set of opcodes" - some of 
which know that there's an implicit ROT3 to do. Perhaps in some other 
circumstances there's a ROT3 and another op before the STORE. Or perhaps 
three other ops ...


I'm just pointing out that concentrating on folding a very specific 
sequence of opcodes that you happen to see in your test code disassembly 
does not necessarily scale well.


Why just INPLACE_ADD? For each variant of each type of STORE operation 
(we know there are at least two), you also might want to have MULTIPLY, 
SUBTRACT, DIVIDE, BIT_OR, BIT_AND, SHIFT_LEFT, SHIFT_RIGHT, EXP and 
MODULO versions ... (have I forgotten any?)


E.


--
https://mail.python.org/mailman/listinfo/python-list

Re: Python and the need for speed

[bartc]

On 19/04/2017 00:08, Gregory Ewing wrote:

Erik wrote:

When considering special-casing this opcode sequence, remember that
in-place operations can be performed on anonymous objects (i.e., those
referenced by a collection and not bound directly to a namespace):


I think this means you would want multiple versions of the +=1 opcode:
ADD_ONE_TO_NAME, ADD_ONE_TO_ITEM, ADD_ONE_TO_ATTR, etc. Essentially
one for each variant of the STORE operation.


This where it starts to get out of hand, if using a fixed set of byte-codes.

I suggested earlier in the thread using special references, which can be 
created for each kind of access (GLOBAL, FAST, INDEX etc). But then you 
have one REF type and only need ADD_ONE_TO_REF.



Or you might choose to optimise just the local name case, on the
basis that the overhead of things like item and attribute lookup
would probably swamp any possible gains.


But you would reduce two lookups to one, so the gains might be greater 
with those more elaborate accesses.


--
bartc
--
https://mail.python.org/mailman/listinfo/python-list

Re: Python and the need for speed

[Gregory Ewing]

bartc wrote:

But another problem was, there /are/ no simple byte-codes in CPython!


Yes, that's the crux of the whole thing. It's the reason it's
so hard to compile Python to anything that runs significantly
faster than interpreted code.

Another optimisation for something 
like ADD, was to to check for very common types such as INT, and deal 
with those locally.


And then you're on the path to writing a JIT compiler. If you
can identify such cases, you might as well generate machine
code for them.

--
Greg
--
https://mail.python.org/mailman/listinfo/python-list

Re: Python and the need for speed

[Gregory Ewing]

Erik wrote:
When considering special-casing this opcode sequence, remember that 
in-place operations can be performed on anonymous objects (i.e., those 
referenced by a collection and not bound directly to a namespace):


I think this means you would want multiple versions of the +=1 opcode:
ADD_ONE_TO_NAME, ADD_ONE_TO_ITEM, ADD_ONE_TO_ATTR, etc. Essentially
one for each variant of the STORE operation.

Or you might choose to optimise just the local name case, on the
basis that the overhead of things like item and attribute lookup
would probably swamp any possible gains.

--
Greg
--
https://mail.python.org/mailman/listinfo/python-list

Re: Looping [was Re: Python and the need for speed]

[Gregory Ewing]

Ben Bacarisse wrote:


I fond the proportion on while True: loops surprising.  Is there
something about Python that encourages that kind of loop?


Maybe because for-loops take care of most of the ordinary
cases in Python, leaving while-loops to cover the weird
ones, many of which need one or more exits in the middle
somewhere.

--
Greg
--
https://mail.python.org/mailman/listinfo/python-list

Re: Python and the need for speed

[Erik]

On 18/04/17 11:30, bartc wrote:

On 18/04/2017 10:32, Erik wrote:


the
improvements over the original huge switch() to dispatch the bytecodes
to the correct handler appear to have made this type of optimization
less effective.


What did they do to it, and on which version?


It's the computed 'goto' stuff that I'm referring to. At the time, I was 
looking at 2.7.9 (which just has the big switch) and 3.5.0 (which has 
the computed gotos). I can't be any more specific than that without 
looking at commit histories - you can do that if you want ;).


I am running on Linux using GCC, so I am using the computed-goto version 
(I hadn't realised at the time that it was optional - so perhaps my 
changes _would_ still have shown an improvement on builds that still use 
the big switch ... I'll have to look again when I have time).


The opcodes have changed from bytecode to wordcode since then, so my 
changes would need looking at again anyway.



(I couldn't get anywhere with CPython because:


[snip]

It obviously builds on Windows (isn't there a MSVC project?), but FWIW, 
on Linux things do get a bit hairy when you want to introduce new 
opcodes, for example (IIRC, the build process itself uses a version of 
Python that it builds to generate some "frozen" .py files for the final 
build and that can cause things to get out of step where the 
build-process executable doesn't understand the new opcodes - I forget 
the details, but it's something like that. I think the answer was to 
keep "clean"ing the build area and building from scratch each time the 
opcode map changes).


Perhaps the Windows build process requires that things are stable and 
doesn't support this type of development at all. You should take this 
part of the discussion to python-dev if you want to be able to build and 
experiment with it on Windows.



If that had worked, then further optimisations are possible, such as
doing a pre-pass combining common operations, that would not be
worthwhile using 'official' byte-codes.)


That is effectively what my experiments were doing - sort of -, but 
without introducing an assembler layer. I'm not convinced about that - 
given the right hints, a half-decent C compiler these days will produce 
pretty good code.


E.
--
https://mail.python.org/mailman/listinfo/python-list

Re: Static typing [was Re: Python and the need for speed]

[Steve D'Aprano]

On Tue, 18 Apr 2017 03:43 pm, Gregory Ewing wrote:

> Steve D'Aprano wrote:
>> I'm not sure why the Cython devs maintain this is not a JIT compiler.
>> Perhaps I misunderstand something.
> 
> A JIT compiler works by observing the actual values taken on
> by variables at run time, and if it notices that a particular
> variable seems to always have a particular type, it compiles
> a special version of the code to handle that type, protected
> by appropriate guards.

You seem to be describing a *tracing JIT* compiler.

> Cython doesn't do any of that -- it's just a plain, boring,
> standard ahead-of-time compiler that goes by the type info
> you give it and nothing more.

That cannot be the full story, because Cython can optimize some regular,
unannotated Python code.



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python and the need for speed

[bartc]

On 18/04/2017 10:32, Erik wrote:


FWIW, I spent some time about a year ago looking at things like this
(small improvements to the peephole optimizer which allowed certain very
common sequences to be folded into a (new) opcode which in turn allowed
other optimizations to avoid branching). The changes worked, but didn't
actually improve performance significantly in my tests (which is why I
ended up not bothering to propose anything).

I remember back in the day (circa 1.5.2?) that
trips-around-the-interpreter-loop were significant and avoiding them
could give wins. However, in the current CPython interpreter, the
improvements over the original huge switch() to dispatch the bytecodes
to the correct handler appear to have made this type of optimization
less effective.


What did they do to it, and on which version? I've just downloaded 3.6.1 
and it was about 17% faster than 3.4.3 on running that test I posted a 
few days ago.


This was on Windows.

But I also looked at CPython a couple of years back with a view to doing 
something with it. I didn't manage anything, but learned that the 
gcc-compiled Linux version enabled label-pointers in place of the big 
switch, while the MSVC-compiled Windows version still used the switch.


The differences (on small integer benchmarks) were also 15% or so.

(I couldn't get anywhere with CPython because:

(1) It's so complicated that I could only build on Linux

(2) It depends on configure scripts and makefiles that I have no idea about

(3) None of my tools (editors etc) work on Linux so it was awkward 
(although cross-OS working using a real Windows and a virtual Linux 
would have been viable I suppose.)


(4) But the project was just too complicated to find my way around. It 
would have taken ages (the source is full of conditional blocks of code, 
and uses macros everywhere).


The thing I had in mind, that I've used elsewhere, was to add an extra 
layer to the byte-code dispatcher, written in assembly in its own module.


This has a dispatch loop (using 'threaded' code and with 
register-resident interpreter variables) which intercepts each 
byte-code, looks at it, and passes it on to the standard dispatcher if 
it can't deal with it.


So initially it would slow it down. But then you gradually add local 
handling of simple byte-codes, and eventually it gets faster than 
running the 100% HLL version.


But another problem was, there /are/ no simple byte-codes in CPython! Or 
very few (jump-absolute for example). Another optimisation for something 
like ADD, was to to check for very common types such as INT, and deal 
with those locally. But the handling required in the C code even for 
that looked horrendous.


You need to be able to able to handle such cases in a dozen lines of 
assembly, otherwise the code gets impractical.


If that had worked, then further optimisations are possible, such as 
doing a pre-pass combining common operations, that would not be 
worthwhile using 'official' byte-codes.)


--
bartc
--
https://mail.python.org/mailman/listinfo/python-list

Re: Python and the need for speed

[Erik]

On 13/04/17 18:50, MRAB wrote:

On 2017-04-13 09:08, Steven D'Aprano wrote:

On Wed, 12 Apr 2017 16:30:38 -0700, bart4858 wrote:
Is it possible to skip the STORE_NAME op-code? If you knew *for sure*
that the target (x) was a mutable object which implemented += using an
in-
place mutation, then you could, but the only built-in where that applies
is list so even if you could guarantee x was a list, it hardly seems
worth the bother.


If the reference to be stored by STORE_NAME is the same as the reference
returned by LOAD_NAME, then STORE_NAME could be omitted.

That would just mean remembering that address.


When considering special-casing this opcode sequence, remember that 
in-place operations can be performed on anonymous objects (i.e., those 
referenced by a collection and not bound directly to a namespace):


>>> import dis
>>> dis.dis(compile("x = [0, 1, 2]; x[1] += 1;", "", "single"))
  1   0 LOAD_CONST   0 (0)
  3 LOAD_CONST   1 (1)
  6 LOAD_CONST   2 (2)
  9 BUILD_LIST   3
 12 STORE_NAME   0 (x)
 15 LOAD_NAME0 (x)
 18 LOAD_CONST   1 (1)
 21 DUP_TOP_TWO
 22 BINARY_SUBSCR
 23 LOAD_CONST   1 (1)
 26 INPLACE_ADD
 27 ROT_THREE
 28 STORE_SUBSCR
 29 LOAD_CONST   3 (None)
 32 RETURN_VALUE

So in this case, the STORE_SUBSCR does the re-binding, but it is 
separated from the INPLACE_ADD by another opcode.


I'm not saying it's impossible to fold the re-binding into a (set of) 
special new opcode(s), but I am saying it's more complex than at first 
it appears.




FWIW, I spent some time about a year ago looking at things like this 
(small improvements to the peephole optimizer which allowed certain very 
common sequences to be folded into a (new) opcode which in turn allowed 
other optimizations to avoid branching). The changes worked, but didn't 
actually improve performance significantly in my tests (which is why I 
ended up not bothering to propose anything).


I remember back in the day (circa 1.5.2?) that 
trips-around-the-interpreter-loop were significant and avoiding them 
could give wins. However, in the current CPython interpreter, the 
improvements over the original huge switch() to dispatch the bytecodes 
to the correct handler appear to have made this type of optimization 
less effective. That was my conclusion at the time, anyway - I only had 
about a week to experiment with it.


E.
--
https://mail.python.org/mailman/listinfo/python-list

Re: Looping [was Re: Python and the need for speed]

[Jussi Piitulainen]

Christian Gollwitzer writes:

> Am 18.04.17 um 08:21 schrieb Chris Angelico:
>> On Tue, Apr 18, 2017 at 4:06 PM, Christian Gollwitzer  
>> wrote:
>>> Am 18.04.17 um 02:18 schrieb Ben Bacarisse:
>>>
 Thanks (and to Grant).  IO seems to be the canonical example.  Where
 some languages would force one to write

   c = sys.stdin.read(1)
   while c == ' ':
   c = sys.stdin.read(1)
>>>
>>> repeat
>>> c  = sys.stdin.read(1)
>>> until c != ' '
>>
>> Except that there's processing code after it.
>>
>
> Sorry, I misread it then - Ben's code did NOT have it, it looks like a
> "skip the whitespace" loop.

It also reads the first character that is not whitespace, so it's not
usable to *merely* skip the whitespace.

>> while True:
>> c = sys.stdin.read(1)
>> if not c: break
>> if c.isprintable(): text += c
>> elif c == "\x08": text = text[:-1]
>> # etc
>>
>> Can you write _that_ as a do-while?
>
> No. This case OTOH looks like an iteration to me and it would be most
> logical to write
>
> for c in sys.stdin:
>  if c.isprintable(): text += c
>  elif c == "\x08": text = text[:-1]
>  # etc
>
> except that this iterates over lines. Is there an analogous iterator
> for chars? For "lines" terminated by something else than "\n"?
> "for c in get_chars(sys.stdin)" and
> "for c in get_string(sys.stdin, terminate=':')" would be nicely
> readable IMHO. Or AWK-like processing:
>
> for fields in get_fields(open('/etc/passwd'), RS='\n', FS=':'):
>   if fields[2]=='0':
>   print 'Super-User found:', fields[0]

I don't know if those exist in some standard library, but they are easy
to write, and I do it all the time. I don't need the chars one, but I do
tab-separated fields and line-separated groups of tab-separated fields,
and variations.

for s, sentence in enumerate(sentences(sys.stdin)):
for k, token in enumerate(sentence):
...
token[LEMMA] or warn('empty LEMMA', s, k, sentence)
...

The wrapper function around sys.stdin or other text source is different
depending on the data format. Sometimes it's messy, sometimes not. Any
messy details are hidden the wrapper.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Looping [was Re: Python and the need for speed]

[Christian Gollwitzer]

Am 18.04.17 um 08:21 schrieb Chris Angelico:

On Tue, Apr 18, 2017 at 4:06 PM, Christian Gollwitzer  wrote:

Am 18.04.17 um 02:18 schrieb Ben Bacarisse:


Thanks (and to Grant).  IO seems to be the canonical example.  Where
some languages would force one to write

  c = sys.stdin.read(1)
  while c == ' ':
  c = sys.stdin.read(1)


repeat
c  = sys.stdin.read(1)
until c != ' '


Except that there's processing code after it.



Sorry, I misread it then - Ben's code did NOT have it, it looks like a 
"skip the whitespace" loop.



while True:
c = sys.stdin.read(1)
if not c: break
if c.isprintable(): text += c
elif c == "\x08": text = text[:-1]
# etc

Can you write _that_ as a do-while?


No. This case OTOH looks like an iteration to me and it would be most 
logical to write


for c in sys.stdin:
 if c.isprintable(): text += c
 elif c == "\x08": text = text[:-1]
 # etc

except that this iterates over lines. Is there an analogous iterator for 
chars? For "lines" terminated by something else than "\n"?

"for c in get_chars(sys.stdin)" and
"for c in get_string(sys.stdin, terminate=':')" would be nicely readable 
IMHO. Or AWK-like processing:


for fields in get_fields(open('/etc/passwd'), RS='\n', FS=':'):
if fields[2]=='0':
print 'Super-User found:', fields[0]


Christian
--
https://mail.python.org/mailman/listinfo/python-list

Re: Looping [was Re: Python and the need for speed]

[Marko Rauhamaa]

Christian Gollwitzer :

> Am 18.04.17 um 02:18 schrieb Ben Bacarisse:
>> Python opts for
>>
>>   while True:
>>  c = sys.stdin.read(1)
>>  if c != ' ': break
>
> This loop would be the archetypical do..while or repeat...until to me.
>
> do
>   c = sys.stdin.read(1)
> while c== ' '

No, the code continues. You want to do something with c, right?

> is the most clear to me - and in fact this "while True; do something;
> break" thingy is just an idiom to fake a do..while loop in Python. C
> does have it, for example, and it is way better like this than the
> abuse of assignment and comma operator in the condition.

I do use

   do ... while

in my C code whenever a loop would end in a conditional break. I happens
exceedingly rarely, though.


Marko
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Looping [was Re: Python and the need for speed]

[Chris Angelico]

On Tue, Apr 18, 2017 at 4:06 PM, Christian Gollwitzer  wrote:
> Am 18.04.17 um 02:18 schrieb Ben Bacarisse:
>
>> Thanks (and to Grant).  IO seems to be the canonical example.  Where
>> some languages would force one to write
>>
>>   c = sys.stdin.read(1)
>>   while c == ' ':
>>   c = sys.stdin.read(1)
>>
>> Python opts for
>>
>>   while True:
>>  c = sys.stdin.read(1)
>>  if c != ' ': break
>
>
> This loop would be the archetypical do..while or repeat...until to me.
>
> do
> c = sys.stdin.read(1)
> while c== ' '
>
>
> -or-
>
> repeat
> c  = sys.stdin.read(1)
> until c != ' '

Except that there's processing code after it.

while True:
c = sys.stdin.read(1)
if not c: break
if c.isprintable(): text += c
elif c == "\x08": text = text[:-1]
# etc

Can you write _that_ as a do-while?

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Looping [was Re: Python and the need for speed]

[Christian Gollwitzer]

Am 18.04.17 um 02:18 schrieb Ben Bacarisse:


Thanks (and to Grant).  IO seems to be the canonical example.  Where
some languages would force one to write

  c = sys.stdin.read(1)
  while c == ' ':
  c = sys.stdin.read(1)

Python opts for

  while True:
 c = sys.stdin.read(1)
 if c != ' ': break


This loop would be the archetypical do..while or repeat...until to me.

do
c = sys.stdin.read(1)
while c== ' '


-or-

repeat
c  = sys.stdin.read(1)
until c != ' '

is the most clear to me - and in fact this "while True; do something; 
break" thingy is just an idiom to fake a do..while loop in Python.
C does have it, for example, and it is way better like this than the 
abuse of assignment and comma operator in the condition.


Christian
--
https://mail.python.org/mailman/listinfo/python-list

Re: Static typing [was Re: Python and the need for speed]

[Gregory Ewing]

Steve D'Aprano wrote:

I'm not sure why the Cython devs maintain this is not a JIT compiler.
Perhaps I misunderstand something.


A JIT compiler works by observing the actual values taken on
by variables at run time, and if it notices that a particular
variable seems to always have a particular type, it compiles
a special version of the code to handle that type, protected
by appropriate guards.

Cython doesn't do any of that -- it's just a plain, boring,
standard ahead-of-time compiler that goes by the type info
you give it and nothing more.

--
Greg
--
https://mail.python.org/mailman/listinfo/python-list

Re: Looping [was Re: Python and the need for speed]

[breamoreboy]

On Tuesday, April 18, 2017 at 2:09:19 AM UTC+1, Paul Rubin wrote:
> Ben Bacarisse writes:
> > ?  I get "AttributeError: 'itertools.dropwhile' object has no attribute
> > 'next'" from your example.
> 
> Hmm, .next() worked ok for me in Python 2.7.5.  Not sure what happened.
> Maybe something went wrong with my paste.  Oh well.
> 

PEP 3114 -- Renaming iterator.next() to iterator.__next__() 
https://www.python.org/dev/peps/pep-3114/

Kindest regards.

Mark Lawrence.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Looping [was Re: Python and the need for speed]

[Marko Rauhamaa]

Ben Bacarisse :

> Python opts for
>
>   while True:
>  c = sys.stdin.read(1)
>  if c != ' ': break

I opt for that in C and bash as well.

In fact, when I start writing a loop, I first type:

while True:


Once it is done, I might notice that the loop begins:

while True:
if ...
break

and merge "while" and "if".


Marko
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Static typing [was Re: Python and the need for speed]

[Steve D'Aprano]

On Mon, 17 Apr 2017 08:52 pm, Steve D'Aprano wrote:

> but research does continue into using gradual typing for optimizations:
> 
> http://dl.acm.org/citation.cfm?id=2069175

Another interesting research project into speeding up Jython using type
hints:

http://scholar.colorado.edu/ecen_gradetds/57/

and more about gradual typing in Python:

http://homes.soic.indiana.edu/mvitouse/talks/stop12.pdf



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Looping [was Re: Python and the need for speed]

[Marko Rauhamaa]

Gregory Ewing :

> Marko Rauhamaa wrote:
>> What I notice in my numbers is that about one half of my while loops
>> are "while True", and about a third of my loops are while loops.
>
> Out of curiosity, what proportion of your 'while True' loops are
> infinite? (I.e. no break, return or raise in the loop.)

0%


Marko
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Looping [was Re: Python and the need for speed]

[eryk sun]

On Tue, Apr 18, 2017 at 1:37 AM, MRAB  wrote:
> In Python 3 it's:
>
> c = next(itertools.dropwhile(
>  lambda c: c==' ',
>  iter(lambda: sys.stdin.read(1),None)
>  ))

iter's sentinel should be an empty string.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Looping [was Re: Python and the need for speed]

[MRAB]

On 2017-04-18 02:09, Paul Rubin wrote:

Ben Bacarisse  writes:

?  I get "AttributeError: 'itertools.dropwhile' object has no attribute
'next'" from your example.


Hmm, .next() worked ok for me in Python 2.7.5.  Not sure what happened.
Maybe something went wrong with my paste.  Oh well.


Coming from the lazy language Haskell, I find your example natural...


Yep ;)


Your mistake was that you didn't say it was Python 2. :-)

In Python 3 it's:

c = next(itertools.dropwhile(
 lambda c: c==' ',
 iter(lambda: sys.stdin.read(1),None)
 ))
--
https://mail.python.org/mailman/listinfo/python-list

Re: Looping [was Re: Python and the need for speed]

[Paul Rubin]

Ben Bacarisse  writes:
> ?  I get "AttributeError: 'itertools.dropwhile' object has no attribute
> 'next'" from your example.

Hmm, .next() worked ok for me in Python 2.7.5.  Not sure what happened.
Maybe something went wrong with my paste.  Oh well.

> Coming from the lazy language Haskell, I find your example natural...

Yep ;)
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Looping [was Re: Python and the need for speed]

[Ben Bacarisse]

Paul Rubin  writes:

> Ben Bacarisse  writes:
>>   c = sys.stdin.read(1)
>>   while c == ' ':
>>   c = sys.stdin.read(1)

(for the record: I was not suggesting this was how you'd do it but how
you'd be forced to do it in some languages)

> c = itertools.dropwhile(
>  lambda c: c==' ',
>  iter(lambda: sys.stdin.read(1),None)
>  ).next()

Did you mean

  c = next(itertools.dropwhile(lambda c: c==' ',
   iter(lambda: sys.stdin.read(1), None)))

?  I get "AttributeError: 'itertools.dropwhile' object has no attribute
'next'" from your example.

Coming from the lazy language Haskell, I find your example natural
(though the syntax is a little busy for my taste) but would it be
considered Pythonic?

-- 
Ben.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Looping [was Re: Python and the need for speed]

[bartc]

On 18/04/2017 01:23, Paul Rubin wrote:

Ben Bacarisse  writes:

  c = sys.stdin.read(1)
  while c == ' ':
  c = sys.stdin.read(1)


c = itertools.dropwhile(
 lambda c: c==' ',
 iter(lambda: sys.stdin.read(1),None)
 ).next()



I tried this but it doesn't like the .next.

I wanted to see if it was any faster or slower than the more obvious loop.

(With the original loop, changing:

c=sys.stdin.read(1)
while c==' ':
c=sys.stdin.read(1)

to:

r=sys.stdin.read

c=r(1)
while c==' ':
c=r(1)

made it 75% faster (reading 10M spaces). Those 3 extra lookups per loop 
I guess (this was run inside a function otherwise the dynamics are 
different).


--
bartc
--
https://mail.python.org/mailman/listinfo/python-list

Re: Looping [was Re: Python and the need for speed]

[Ben Bacarisse]

Marko Rauhamaa  writes:

> Ben Bacarisse :
>
>> Marko Rauhamaa  writes:
>>> What I notice in my numbers is that about one half of my while loops
>>> are "while True", and about a third of my loops are while loops.
>>
>> I fo[u]nd the proportion on while True: loops surprising. Is there
>> something about Python that encourages that kind of loop?

(Thanks for th type correction.  There is also s/of/on/.)

> Here's a typical example of such a loop in Python (ver 2):
>
> while True:
> try:
> snippet = os.read(rdfd, 1000)
> except OSError as e:
> if e.errno == errno.EAGAIN:
> return
> raise
> if not snippet:
> break
> self.stdout_snippets.append(snippet)

Thanks (and to Grant).  IO seems to be the canonical example.  Where
some languages would force one to write

  c = sys.stdin.read(1)
  while c == ' ':
  c = sys.stdin.read(1)

and an Algol-68 style language would permit one to write

  while (c := read char; c) do skip od

Python opts for

  while True:
 c = sys.stdin.read(1)
 if c != ' ': break

(Forgive the atypical choice of input primitive -- it's for illustrating
the loop style only.)


-- 
Ben.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Looping [was Re: Python and the need for speed]

[Paul Rubin]

Ben Bacarisse  writes:
>   c = sys.stdin.read(1)
>   while c == ' ':
>   c = sys.stdin.read(1)

c = itertools.dropwhile(
 lambda c: c==' ',
 iter(lambda: sys.stdin.read(1),None)
 ).next()
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Looping [was Re: Python and the need for speed]

[Gregory Ewing]

Marko Rauhamaa wrote:

What I notice in my numbers is that about one half of my while loops are
"while True", and about a third of my loops are while loops.


Out of curiosity, what proportion of your 'while True' loops
are infinite? (I.e. no break, return or raise in the loop.)

--
Greg
--
https://mail.python.org/mailman/listinfo/python-list

Re: Static typing [was Re: Python and the need for speed]

[Paul Rubin]

Steve D'Aprano  writes:
> On the other hand, there's Cython. Cython claims *not* to be a JIT compiler,

One of the uses of "JIT compiler" these days is what's sometimes called
a tracing JIT, like PyPy or LuaJIT or the Javascript flavor-of-the-week.
That means it interprets the program while collecting execution traces
to figure out the actual types the program uses at runtime, then
generates machine code for those specific code paths, with some guards
in case an unexpectedly typed value shows up.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Looping [was Re: Python and the need for speed]

[Grant Edwards]

On 2017-04-17, Ben Bacarisse  wrote:
> Marko Rauhamaa  writes:
>
>> Terry Reedy :
>>
>>> On 4/17/2017 3:11 AM, Marko Rauhamaa wrote:
 Here's statistics from a medium-sized project of mine:

while True:34
while : 39
for ... in ...:   158
>>>
>>> As I posted previously, the ratio of for-loops in the stdlib is about 7
>>> to 1.
>>
>> What I notice in my numbers is that about one half of my while loops are
>> "while True", and about a third of my loops are while loops.
>
> I fond the proportion on while True: loops surprising.  Is there
> something about Python that encourages that kind of loop?

That's how you read data from a socket/pipe/file.

-- 
Grant Edwards   grant.b.edwardsYow! Your CHEEKS sit like
  at   twin NECTARINES above
  gmail.coma MOUTH that knows no
   BOUNDS --

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Looping [was Re: Python and the need for speed]

[Mikhail V]

On 17 April 2017 at 04:00, Steve D'Aprano  wrote:
> On Mon, 17 Apr 2017 05:49 am, Dennis Lee Bieber wrote:
>
>> On Mon, 17 Apr 2017 02:48:08 +1000, Steve D'Aprano
>>  declaimed the following:
>>
>>>On Sun, 16 Apr 2017 11:57 pm, bartc wrote:
>>>
 But people just don't want it.
>>>
>>>Damn straight. Now you get it. It's not about how easy it is to implement,
>>>it's about whether it is needed and wanted. It isn't needed, and it isn't
>>>wanted.
>>>
>>
>> Let's go all the way, and just get rid of "while ".
>
> Sure, why not? If it's your language, you can make it as minimalist or
> maximalist as you like. You get to choose.
>

I think it is right not to touch anything for .py file rules.
Seriously there are much more problems, and better code editors
should adress those, not current syntax.


As for loops, well, if we speak of some 'my' language,
it could be interesting whether there are common patterns.
But is it on-topic here to speak of, say PyBasic2020 fantasy
language?

For real-time application development the amount of
"while(condition)" vs "while True" can play bad jokes
with readability.
I would find such looping schema simpler:

loop:
if contitioin 1 : contitioin_break = 1
blabla bla = bla  + foo
make something
make something
if contitioin 2 : contitioin_break = 1
if contitioin 3 : contitioin_break = 0
if contitioin_break :  *break
/loop


Anyway it is just question where to show this last line,
or to make an emphasis on it.
so I'd just leave it there, for me it helps when I imagine
a cascade of functions and my eyes go down to the
bottom.  If I have kilometers of "if"s inside the loop,
then hiding one line can become more a problem than an aid.

Constructs like "repeat 100 times {}" or other sort of
"let's pronounce it in English" found in
some languages looks sort of confusing, it takes
me time to "unwrap" their sense into a cascade
especially when there are many different keywords.


Mikhail
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Looping [was Re: Python and the need for speed]

[Marko Rauhamaa]

Ben Bacarisse :

> Marko Rauhamaa  writes:
>> What I notice in my numbers is that about one half of my while loops
>> are "while True", and about a third of my loops are while loops.
>
> I fo[u]nd the proportion on while True: loops surprising. Is there
> something about Python that encourages that kind of loop?

Here's a typical example of such a loop in Python (ver 2):

while True:
try:
snippet = os.read(rdfd, 1000)
except OSError as e:
if e.errno == errno.EAGAIN:
return
raise
if not snippet:
break
self.stdout_snippets.append(snippet)

I find myself writing similar loops in C a lot, as well.


Marko
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Looping [was Re: Python and the need for speed]

[bartc]

On 17/04/2017 19:02, Ben Bacarisse wrote:

Marko Rauhamaa  writes:


Terry Reedy :


On 4/17/2017 3:11 AM, Marko Rauhamaa wrote:

Here's statistics from a medium-sized project of mine:

   while True:34
   while : 39
   for ... in ...:   158


As I posted previously, the ratio of for-loops in the stdlib is about 7
to 1.


What I notice in my numbers is that about one half of my while loops are
"while True", and about a third of my loops are while loops.


I fond the proportion on while True: loops surprising.  Is there
something about Python that encourages that kind of loop?



A few things:

(1) Python doesn't have the equivalent of C's comma operator, and no 
assignment in expressions. So if a condition relies on such set-up code, 
it can't do this:


   while (c=nextc())!=0:

But has to do this or use extra logic:

   while 1:
  c=nextc()
  if c==0: break

(2) There are exceptions ('raise') as an extra means of breaking out of 
such loops (as well as return, break, and exit()


(3) There's 'yield' too, as I've just seen this example:

while True:
   yield []

So there are more excuses to make use of it. Plus Python programmers may 
be more averse to using convoluted logic just to avoid a 'break' in the 
middle of a loop.



--
bartc

--
https://mail.python.org/mailman/listinfo/python-list

Re: Looping [was Re: Python and the need for speed]

[Ben Bacarisse]

Marko Rauhamaa  writes:

> Terry Reedy :
>
>> On 4/17/2017 3:11 AM, Marko Rauhamaa wrote:
>>> Here's statistics from a medium-sized project of mine:
>>>
>>>while True:34
>>>while : 39
>>>for ... in ...:   158
>>
>> As I posted previously, the ratio of for-loops in the stdlib is about 7
>> to 1.
>
> What I notice in my numbers is that about one half of my while loops are
> "while True", and about a third of my loops are while loops.

I fond the proportion on while True: loops surprising.  Is there
something about Python that encourages that kind of loop?

-- 
Ben.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Looping [was Re: Python and the need for speed]

[Marko Rauhamaa]

Terry Reedy :

> On 4/17/2017 3:11 AM, Marko Rauhamaa wrote:
>> Here's statistics from a medium-sized project of mine:
>>
>>while True:34
>>while : 39
>>for ... in ...:   158
>
> As I posted previously, the ratio of for-loops in the stdlib is about 7
> to 1.

What I notice in my numbers is that about one half of my while loops are
"while True", and about a third of my loops are while loops.


Marko
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Looping [was Re: Python and the need for speed]

[Terry Reedy]

On 4/17/2017 3:11 AM, Marko Rauhamaa wrote:

Gregory Ewing :


bartc wrote:

Most of my loops start off as endless loops, until I can determine
the actual terminating condition, and where it best goes.


Interesting. My experience is quite different. Most of the loops I
write start off with me thinking "Now I want to do this for each
element of this collection", which maps straight to a Python for-loop.

In the rare cases where I do use a while loop, usually I have a fairly
good idea what the terminating condition is going to be.


Here's statistics from a medium-sized project of mine:

   while True:34
   while : 39
   for ... in ...:   158


As I posted previously, the ratio of for-loops in the stdlib is about 7 
to 1.


--
Terry Jan Reedy

--
https://mail.python.org/mailman/listinfo/python-list

Re: Static typing [was Re: Python and the need for speed]

[Steve D'Aprano]

On Mon, 17 Apr 2017 04:18 pm, Paul Rubin wrote:

> Steve D'Aprano  writes:
>> Since it is *optional*, it is only a hint, not a fact. You can tell the
>> compiler that you believe that n will be an int, but it's not guaranteed.
> 
> The runtime could check at the entry to the function that n is an int,
> and then it wouldn't have to keep re-checking on the inside of a loop.
> That's what a JIT compiler does in the absence of annotations, more or
> less; 

Funnily enough, in the very post you quote, I did write:

"There are clever Just In Time compiler tricks that can optimize the code,
which is what Psycho did and PyPy still does, but they don't need type
hints."

The reason being that at runtime the compiler knows the type of the value
and the type hint is redundant.

On the other hand, there's Cython. Cython claims *not* to be a JIT compiler,
but I don't really understand why. As I understand it, Cython can operate
in two ways:

(1) as a Python interpreter, with some optimizations switched on or off at
runtime according to the types and/or values actually used by the program;

(2) as a pseudo-Python to C compiler, allowing you to write Python-ish code
that is compiled to C (plus a Python runtime for anything that's too hard
to translate into C).

I'm not sure why the Cython devs maintain this is not a JIT compiler.
Perhaps I misunderstand something.


> but the annotations make life easier for ahead-of-time compilers. 
> Again this type of thing has been standard in Lisp since the 1960's.

Indeed it has. The state of the art of optimizations for dynamically typed
languages has been badly neglected for many years. Fortunately in the last
decade or so there's been a lot more interest in this.

See, for example, Steve Yegge:

http://steve-yegge.blogspot.com.au/2008/05/dynamic-languages-strike-back.html



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Static typing [was Re: Python and the need for speed]

[Steve D'Aprano]

On Sun, 16 Apr 2017 02:27 pm, Steve D'Aprano wrote:

> Are you aware that optional static typing CANNOT be used for optimization?

I think I've over-stated that. Considerably. In other words, I was wrong.

As Steve Yegge points out, dynamic languages like Smalltalk and Lisp were,
back in the 1980s, heavily optimized and were competitive with
statically-typed languages like C:

http://steve-yegge.blogspot.com.au/2008/05/dynamic-languages-strike-back.html

While this is true:

> Since it is *optional*, it is only a hint, not a fact. You can tell the
> compiler that you believe that n will be an int, but it's not guaranteed.
> As the Dude says, the type hint is
> 
> that's, like, only your opinion man
> 
> and the compiler cannot trust it. It must still guard the operation with a
> runtime check that n actually is an int, 

The conclusion that I drew:

> and so you lose most or all of 
> the benefit of telling the compiler that it's an int. 

is not necessarily the case. If you're doing lots of little operations on
disparate variables, with lots and lots of guard checks and not much
optimized code, it will go badly for you, but on the other hand for big
loops with only a few up-front checks and predictable types, there may be
opportunities for optimization.

Honestly, I'm not quite sure what I was thinking: that sort of "optimized
code with a guard" is exactly what Psycho does, and what FATPython is
hoping to bring to CPython some day.

At the present time, the state of the art of gradual typing is still keeping
the emphasis on optional static typing for *correctness*, not optimization:

http://wphomes.soic.indiana.edu/jsiek/what-is-gradual-typing/

but research does continue into using gradual typing for optimizations:

http://dl.acm.org/citation.cfm?id=2069175


[...]
> Python's gradual typing is not about optimization, it is about testing
> program correctness and helping to find bugs at compile time*. 

That's the motivation for MyPy.




-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Moderating the list [was: Python and the need for speed]

[breamoreboy]

On Thursday, April 13, 2017 at 10:31:22 AM UTC+1, Tim Golden wrote:
> On 13/04/2017 03:39, Jason Friedman wrote:
> >>
> >> However, it's simply a technical fact: the thing which we moderate is the
> >>> mailing list. We can control which posts make it through from the 
> >>> newsgroup
> >>> by blocking them at the gateway. But the posts will continue to appear on
> >>> comp.lang.python which is, as the description says, unmoderated.
> >>>
> >>
> >> TJG
> >
> >
> > Thank you, Tim and Ethan and the other moderators, for performing that
> > function.
> > It makes the reading experience more pleasant for me.
> >
> 
> I looked, for the first time in a long while, at the GG mirror of the 
> newsgroup. Hadn't realised how much of that really nasty all-caps spam 
> we were blocking. (We set up some gateway rules when that started to 
> become a problem so we just don't see it on the mailing list).
> 
> Most of the time we're just binning run-of-the-mill spam. Doesn't take 
> too much effort if you keep on top of it, but it piles up quickly enough 
> if you leave it for a bit!
> 
> TJG

I routinely set all of the upper-case crap to "violent content" on GG just as a 
matter of principle.  There's even been a couple of English versions lately but 
the filters have stopped them, well done :-)

Kindest regards.

Mark Lawrence.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Looping [was Re: Python and the need for speed]

[Marko Rauhamaa]

Gregory Ewing :

> bartc wrote:
>> Most of my loops start off as endless loops, until I can determine
>> the actual terminating condition, and where it best goes.
>
> Interesting. My experience is quite different. Most of the loops I
> write start off with me thinking "Now I want to do this for each
> element of this collection", which maps straight to a Python for-loop.
>
> In the rare cases where I do use a while loop, usually I have a fairly
> good idea what the terminating condition is going to be.

Here's statistics from a medium-sized project of mine:

   while True:34
   while : 39
   for ... in ...:   158


Marko
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Static typing [was Re: Python and the need for speed]

[Paul Rubin]

Steve D'Aprano  writes:
> Since it is *optional*, it is only a hint, not a fact. You can tell the
> compiler that you believe that n will be an int, but it's not guaranteed.

The runtime could check at the entry to the function that n is an int,
and then it wouldn't have to keep re-checking on the inside of a loop.
That's what a JIT compiler does in the absence of annotations, more or
less; but the annotations make life easier for ahead-of-time compilers.
Again this type of thing has been standard in Lisp since the 1960's.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Static typing [was Re: Python and the need for speed]

[Steve D'Aprano]

On Mon, 17 Apr 2017 05:16 am, bartc wrote:

> But it was OK for Steve to 'win' the benchmark by substituting my test
> code with something only vaguely related, and much simpler?

Okay, you're now being obnoxious, and telling lies about me.

What I said was 

"FOR WHAT IT'S WORTH [emphasis added], on my machine (2GB RAM and 1.2GHz
CPU) I can add up 100 million ints in 14 seconds"

which is just to give an indication of the speed of my machine. I wasn't
trying to "win" your stupid, inefficient benchmark. If I was, would I then
have run *your code* and report a time nearly three times longer than you?

"time taken: 93.543194 seconds"

How is this me winning by running different code?

You also ignored the entire point of my post, which is to comment on
Justin's generator version:

"If I can run Bart's test on my slow old computer in a minute and a half, my
guess is that your generator must have been doing something terribly wrong
to still not be complete after six minutes."

and misrepresent me as doing something underhanded and dishonorable
to "win".

And that, Bart, is dirty pool. I don't think you're arguing in good faith.

I've spent months defending your right to criticise Python on this forum
even when others have wanted you banned (an excessive reaction) or have
kill-filed you so they don't see your posts. Even when I disagree with your
conclusions, even if I think they are terribly naive, the sorts of things a
lone cowboy developer can get away with but not suitable for a mature,
widely used language like Python, I've respected your point of view.

But at this point, I think *you* are being dishonest.




-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Looping [was Re: Python and the need for speed]

[Steve D'Aprano]

On Mon, 17 Apr 2017 05:49 am, Dennis Lee Bieber wrote:

> On Mon, 17 Apr 2017 02:48:08 +1000, Steve D'Aprano
>  declaimed the following:
> 
>>On Sun, 16 Apr 2017 11:57 pm, bartc wrote:
>>
>>> But people just don't want it.
>>
>>Damn straight. Now you get it. It's not about how easy it is to implement,
>>it's about whether it is needed and wanted. It isn't needed, and it isn't
>>wanted.
>>
> 
> Let's go all the way, and just get rid of "while ".

Sure, why not? If it's your language, you can make it as minimalist or
maximalist as you like. You get to choose.

Python's not your language, so it's not your choice for Python. But you can
always create your own language and implement as many or as few features as
you like.

Guido likes while loops with a condition, and Python is Guido's language.

And of course, since Python has had while loops with a condition for over 20
years now, backwards compatibility precludes removing it.




-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Looping [was Re: Python and the need for speed]

[Gregory Ewing]

bartc wrote:


 > - describing the various syntax forms;
 > - explaining how they differ;
 > - tutorials for beginners showing each form;

And you don't have to explain how an endless loop should be written as 
'while True', meanwhile advising against using 'while 1'?


You don't have to mention it in the reference docs, because it's
deducible from the rest of the language rules. Nor do you need
an explicit test for it, unless there is code in the compiler
specifically to optimise that case.

Tutorials might want to mention it, but it's not strictly
necessary, as a smart enough student could figure it out for
themselves.


So, how many kinds of sequences do you have in Python?

lists
tuples
namedtuples
arrays
bytearrays
string ?
memoryview


By far the most commonly used ones are list and tuple. The
rest are designed for specialised uses, and have characteristics
that can't easily be replicated using the fundamental types,
if at all.

Alternative looping constructs, in contrast, have *no*
functional difference, their only advantage being that they
provide a slightly more succint way of expressing certain
things.


  While a; b; c; d Do e; f; g End

with the test (the value of 'd') effectively in the middle.


That would be a rather obtuse way of expressing it, though.
It looks like the whole of 'a; b; c; d' is the condition,
whereas you're really only using a, b and c for their side
effects.

The whole point of a loop-and-a-half construct is to make
the condition stand out, despite being in the middle.

This is actually the only additional looping construct that
I wouldn't mind seeing. It gets brought up now and then;
last time, if I recall, the discussion fizzled out due to
a feeling that you don't really need it much in Python,
because those kinds of loops are usually better expressed
using an iterator.

Most of my loops start off as endless loops, until I can determine the 
actual terminating condition, and where it best goes.


Interesting. My experience is quite different. Most of
the loops I write start off with me thinking "Now I want
to do this for each element of this collection", which
maps straight to a Python for-loop.

In the rare cases where I do use a while loop, usually I
have a fairly good idea what the terminating condition is
going to be. Cases where a significant amount of head
scratching is needed to figure it out are pretty rare.
But maybe you're working on different kinds of problems
from me.


Sometimes they stay as endless loops.


By design or by accident? :-)

Every program I've every written that had a deliberate
endless loop in it had exactly *one* of them, at the top
level. And I knew it would be that way before I started
writing it. (And since the only way to stop it would be
ctrl-C or equivalent, I only write such things if they're
throwaway tools for my own use.)

--
Greg
--
https://mail.python.org/mailman/listinfo/python-list

Re: Looping [was Re: Python and the need for speed]

[bartc]

On 16/04/2017 19:42, Chris Angelico wrote:

On Mon, Apr 17, 2017 at 4:21 AM, bartc  wrote:

Here is a function from some old CPython source that appears to be something
to do with While statements:

static int
validate_while(node *tree)
{

...

Look, no comments! Are you going to castigate the developers of CPython for
that? (I guess not.)


Can you find the equivalent in the current sources? I think it might be here:

https://github.com/python/cpython/blob/master/Grammar/Grammar#L73




while_stmt: 'while' test ':' suite ['else' ':' suite]

A single, extremely readable line. Abstract, not concrete.


How do you know my code fragment wasn't also generated? Since it was 
parsing C, the refs for that language also define the while statement 
quite neatly:


  'while' '(' expression ')' statement

but actually it was done manually. It's part of a suite of functions 
that do things in a common pattern, and would be documented as a set 
rather than line by line. And elsewhere calls to such a function would 
also be part of a chain of tests.


That's if I needed to documented it (but I can't remember how many 
parsers I've done along the same lines since c.1979).


But I've glanced through the CPython code again, and it is quite lightly 
commented. Although sometimes it gets quite chatty.



Want to show me your source for this source?


The source files I looked at were dated around 2010. I've no idea where 
they came from. I think I'd wanted to compile it, but unfortunately, 
they're not written in C, but a combination of C, make-file, 
compiler-options, and bash-script with a bunch of Linux dependencies.




Next complaint, please.


No complaint, just saying other projects are sparse on comments too. And 
my code isn't intended as open source.


--
bartc

--
https://mail.python.org/mailman/listinfo/python-list

Re: Static typing [was Re: Python and the need for speed]

[Chris Angelico]

On Mon, Apr 17, 2017 at 5:16 AM, bartc  wrote:
> On 16/04/2017 20:00, Chris Angelico wrote:
>>
>> On Mon, Apr 17, 2017 at 4:43 AM, bartc  wrote:
>
>
>> Sure! If all you care about is winning benchmarks,
>
>
> The benchmarks should be about comparing things. But they have to be like
> for like.

You're the one who suggested using a formula to calculate the sum of a
sequence of integers.

> But it was OK for Steve to 'win' the benchmark by substituting my test
> code with something only vaguely related, and much simpler?

Steven was, far as I can tell, reimplementing the same algorithm in
more Pythonic code. But I'm not sure. I didn't look at the details.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Static typing [was Re: Python and the need for speed]

[bartc]

On 16/04/2017 20:00, Chris Angelico wrote:

On Mon, Apr 17, 2017 at 4:43 AM, bartc  wrote:



Sure! If all you care about is winning benchmarks,


The benchmarks should be about comparing things. But they have to be 
like for like.


Since this was about the effects of type annotations, it would be 
interesting to see how they would affect timings in the Python example.


(I showed the affect on another language, where I got a factor of about 
100:1 between slowest and fastest by using type declarations. A bit of a 
cheat though as the typed language was really a separate one.)


 you could easily

create a sum() that recognizes when it's given a range object. Let's
start with a baseline. I took the code from the link Steven posted
(and renamed Timer to Stopwatch), and:

49995000
time taken: 1.143872 seconds



Ehh, too fast.


It is fast. My Python 3.4 on Intel 3.2GHz took 6 seconds. Even pypy took 
over 4 seconds.



time taken: 0.16 seconds

Hey look! We win at benchmarks.


But it was OK for Steve to 'win' the benchmark by substituting my test 
code with something only vaguely related, and much simpler?


--
bartc
--
https://mail.python.org/mailman/listinfo/python-list

Re: Static typing [was Re: Python and the need for speed]

[Chris Angelico]

On Mon, Apr 17, 2017 at 4:43 AM, bartc  wrote:
>>
>> For what it's worth, on my machine (2GB RAM and 1.2GHz CPU) I can add up
>> 100
>> million ints in 14 seconds:
>>
>> py> with Stopwatch():
>> ... sum(range(1))
>> ...
>> 49995000
>> time taken: 14.032116 seconds
>
>
> I'm surprised it took that long. If a 'range' is no longer expanded to a
> list, and 'sum' is built-in, isn't there a formula that can be used to sum a
> sequence? At least when the step is +1.

Sure! If all you care about is winning benchmarks, you could easily
create a sum() that recognizes when it's given a range object. Let's
start with a baseline. I took the code from the link Steven posted
(and renamed Timer to Stopwatch), and:

49995000
time taken: 1.143872 seconds

Ehh, too fast. I'll add another zero onto it.

45
time taken: 11.413530 seconds

Better. Now then.

_orig_sum = sum
def sum(iterable, start=0):
if isinstance(iterable, range) and iterable.step == 1:
# I don't feel like dealing with non-unity steps
n = iterable.stop * (iterable.stop - 1) // 2
if iterable.start:
n -= iterable.start * (iterable.start + 1) // 2
return start + n
return _orig_sum(iterable, start)

45
time taken: 0.16 seconds

Hey look! We win at benchmarks. Of course, to do this, I had to make
every other sum() call slightly slower (maybe not a lot, but slower by
the price of one instance check at best), but what a benefit!

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Static typing [was Re: Python and the need for speed]

[bartc]

On 16/04/2017 18:13, Steve D'Aprano wrote:

On Mon, 17 Apr 2017 02:20 am, justin walters wrote:


On Sun, Apr 16, 2017 at 8:46 AM, bartc  wrote:


What were the results with Python on your machine?




Well, at first I tried to implement it with a generator. I gave up on
waiting for the program to complete
after about 6 minutes.


That seems... excessive.

For what it's worth, on my machine (2GB RAM and 1.2GHz CPU) I can add up 100
million ints in 14 seconds:

py> with Stopwatch():
... sum(range(1))
...
49995000
time taken: 14.032116 seconds


I'm surprised it took that long. If a 'range' is no longer expanded to a 
list, and 'sum' is built-in, isn't there a formula that can be used to 
sum a sequence? At least when the step is +1.



factor of ten between friends?

Of course the built-in sum() is a bit faster than the version Bart uses.
It's almost like he is deliberately writing the least efficient Python code
he can... (or maybe he just doesn't know Python very well).


Yes, it was low-level code, which is a large part of what optimising a 
language is about. You can't always write just high-level code that 
calls fast built-in routines; sometimes the language has to do some 
actual work of its own!


And the same algorithm (if you can call it that) was used in all the 
examples. That it wasn't doing anything meaningful is not important; it 
is testing the same mechanisms that would be involved in a real program.


--
bartc
--
https://mail.python.org/mailman/listinfo/python-list

Re: Looping [was Re: Python and the need for speed]

[Chris Angelico]

On Mon, Apr 17, 2017 at 4:21 AM, bartc  wrote:
> Here is a function from some old CPython source that appears to be something
> to do with While statements:
>
> static int
> validate_while(node *tree)
> {
> int nch = NCH(tree);
> int res = (validate_ntype(tree, while_stmt)
>&& ((nch == 4) || (nch == 7))
>&& validate_name(CHILD(tree, 0), "while")
>&& validate_test(CHILD(tree, 1))
>&& validate_colon(CHILD(tree, 2))
>&& validate_suite(CHILD(tree, 3)));
>
> if (res && (nch == 7))
> res = (validate_name(CHILD(tree, 4), "else")
>&& validate_colon(CHILD(tree, 5))
>&& validate_suite(CHILD(tree, 6)));
>
> return (res);
> }
>
> Look, no comments! Are you going to castigate the developers of CPython for
> that? (I guess not.)

Can you find the equivalent in the current sources? I think it might be here:

https://github.com/python/cpython/blob/master/Grammar/Grammar#L73

while_stmt: 'while' test ':' suite ['else' ':' suite]

A single, extremely readable line. Abstract, not concrete. And my
suspicion is that the code you're looking at above just might have
been generated from the same kind of grammar file - which means that
it is NOT the "CPython source", but rather an intermediate file. It's
the equivalent of looking at a .pyc file and complaining that it has
no comments.

Want to show me your source for this source? The best I can find (by
searching the git history (which was imported from the hg history) for
'validate_while') is the code prior to this commit:

https://github.com/python/cpython/commit/53595c

And I rather suspect, from some of the other comments in the area,
that this file may have been autogenerated. Even if it wasn't, it was
still not the "primary source", but exists merely to implement
Grammar/Grammar, as linked above.

Next complaint, please.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Looping [was Re: Python and the need for speed]

[bartc]

On 16/04/2017 17:30, Steve D'Aprano wrote:

On Sun, 16 Apr 2017 10:06 pm, bartc wrote:



(The 30 Loc figure is with support for loops /in
general/ already in place, and is for /adding/ a new loop statement, in
this case 'while')


What part of *testing* and *documenting* do you not understand?



Do you have any unit tests for your compiler?  How about regression tests --
when you fix a bug, do you write a regression test to ensure it never
creeps back in?

Do you have any documentation for your compiler? Does it include doctests?
Are there any tutorials that beginners can read?

I'm guessing you don't have any of those things. The code snippet you posted
doesn't even have any comments.


Here is a function from some old CPython source that appears to be 
something to do with While statements:


static int
validate_while(node *tree)
{
int nch = NCH(tree);
int res = (validate_ntype(tree, while_stmt)
   && ((nch == 4) || (nch == 7))
   && validate_name(CHILD(tree, 0), "while")
   && validate_test(CHILD(tree, 1))
   && validate_colon(CHILD(tree, 2))
   && validate_suite(CHILD(tree, 3)));

if (res && (nch == 7))
res = (validate_name(CHILD(tree, 4), "else")
   && validate_colon(CHILD(tree, 5))
   && validate_suite(CHILD(tree, 6)));

return (res);
}

Look, no comments! Are you going to castigate the developers of CPython 
for that? (I guess not.)


Anyway I took out the one or two comments from my posted extract.


If Python added something like:

loop N times:
...

we would need *at least* the following:

- a test that `loop` was interpreted as a keyword;
- a test that `times` was interpreted as a keyword;
- a test that `loop 0 times` didn't execute the body at all;
- a test that `loop 1 times` executed the body exactly once;
- a test that `loop N times` executed the body exactly N times, for
  a few different (and probably randomly selected) values of N;
- a test that the statement handled negative integers and floats correctly;
- a test that the statement handled non-numeric values correctly
  (probably by raising an exception);
- a test that `continue` worked as expected inside this statement;
- a test that `break` worked as expected;
- a test that `return` worked as expected;
- a test that `raise` worked as expected;
- a test that signals will be caught correctly inside the statement;

and possibly more.


Well, I was talking more about just 'loop:', to repeat endlessly. Then 
half of those tests disappear. See, a dedicated statement can make 
things simpler.


The other half would be the same for loops generally as, after you get 
to the compiler output (if I can't call it byte-code), then the chances 
are that the original identity of the loop is lost.


Loop N times could be on top of While or For. Or it could have its own 
implementation where it /knows/ the loop counter is an integer, and a 
64-bit one. See, another optimisation opportunity.



BTW supporting a dedicated endless loop was only about 20 lines in
another project. Believe me, it is /nothing/. I wish other aspects were
just as trivial. It didn't even need a dedicated keyword.


Of course it's "nothing" for you, since by the evidence given you don't
bother with the hard parts. No comments, no communication with other
developers (except to gloat over how awesome you are and how stupid their
choices are), no documentation, no tests.


There are a couple of aspects I like of dynamic languages, which is the 
informality and spontaneity.


The informality with Python seems to have disappeared (with all this 
paperwork, and committees and applications to get more features added; 
about as interesting as making a proposal for a new office block).


While the spontaneity seems to have suffered too if a feature takes 
years to be accepted, even if it would only take an hour or two to add 
(on a smaller project like mine).


At least, I get the benefits immediately. If I had to wait five years 
(when I might be dead anyway) then I might as well wait forever.



The loops I use are categorised as:

* Endless


That's just a special case of "until some condition is true".


No, because there is no condition to test or think about. There should 
not be one to specify. Python requires that a dummy condition is 
provided to satisfy the language syntax (but internally it compiles to 
an endless loop, as there there is no bureaucracy telling it what to do).



* N times


That's just a special case of "over an integer sequence".


No, because you don't care about what that sequence is, nor about having 
to present it as named object to the body of the loop. Another 
optimisation opportunity.


Neither does the user really want to be bothered with writing the name 
of a variable. And providing a range() is overkill, as you just want a 
count not a sequence.





* Until some condition is true


You missed one: do you check the condition 

Re: Looping [was Re: Python and the need for speed]

[Steve D'Aprano]

On Mon, 17 Apr 2017 03:00 am, Rustom Mody wrote:


> BTW I regard Steven's long list of things that youve missed such as
> regression tests, docs etc to be somewhat off the mark
> To see that try this experiment:
> Just add a feature to python that matters to you along with all these
> requirements: docs, tests etc etc
> Your chances of acceptance may go up marginally but most likely it will
> still not be accepted.

That's very true.

It's not enough to have working code, tests and documentation to get a
feature accepted. I didn't think I implied otherwise, but if it wasn't
clear, I'm sorry.

The feature first needs to have a reason (use-cases). It must be justified.
Unless you are Guido, "because I like it" is not enough.

When you create your own language, like Guido did with Python and Bart has
with his mystery language, your own personal taste is all it takes to get
features included. Everyone else has to justify their request, and have it
accepted by the core developers.

But even once a feature is accepted *in principle* it won't be actually
added to the language until it has tests and documentation. That's the
point I'm trying to get across to Bart. It doesn't matter how trivial the
implementation, or now much of a no-brainer it is, it still needs tests and
documentation.

In the specific case of "loop forever", I'm pretty sure that it would really
struggle to gain community favour. But even if it did, I'm pretty sure the
core devs and Guido in particular would reject it.

As I've said before, I actually like the syntax "repeat forever" -- in the
right context. In Hyperscript, it suits the rest of the language and reads
well. In Python, well, it is like grafting the head of a lion onto the
shoulder of a horse. Lions are handsome beasts, and horses are also very
attractive, individually, but making a freakish chimera of the two is not.

There are lots of ways to design programming languages, and personally I'm
particularly fond of Hyperscript's verbose English-like syntax, Forth and
other RPN stack-based languages, and Python. That's three very different
styles, and I like them all. But not all together at once.

Bart, if your aesthetic taste says different, that's your right, and good on
you. Nobody wants to tell you what syntactic features you should or
shouldn't add to your language.

I just wish you would return the favour to the Python community and stop
insisting that *your* taste is the only good taste.



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Static typing [was Re: Python and the need for speed]

[Steve D'Aprano]

On Mon, 17 Apr 2017 02:20 am, justin walters wrote:

> On Sun, Apr 16, 2017 at 8:46 AM, bartc  wrote:
> 
>> What were the results with Python on your machine?
> 
> 
> 
> Well, at first I tried to implement it with a generator. I gave up on
> waiting for the program to complete
> after about 6 minutes.

That seems... excessive.

For what it's worth, on my machine (2GB RAM and 1.2GHz CPU) I can add up 100
million ints in 14 seconds:

py> with Stopwatch():
... sum(range(1))
...
49995000
time taken: 14.032116 seconds

which isn't fast but it's within an order of magnitude of fast. What's a
factor of ten between friends?

Of course the built-in sum() is a bit faster than the version Bart uses.
It's almost like he is deliberately writing the least efficient Python code
he can... (or maybe he just doesn't know Python very well). Here's his
version again (slightly adapted):


def add(a, b):
return a + b

def testfn():
total = a = b = 0
for i in range(1):
total += add(a, b)
a += 1
b += 2
return (a, b, total)


And the results:

py> with Stopwatch():
... testfn()
...
(1, 2, 149985000)
time taken: 93.543194 seconds


If I can run Bart's test on my slow old computer in a minute and a half, my
guess is that your generator must have been doing something terribly wrong
to still not be complete after six minutes.


The timer I used (Stopwatch) is basically this:

http://code.activestate.com/recipes/577896-benchmark-code-with-the-with-statement/



> After using your code almost exactly I get:
 Sum:  149985000
 ***Execution Time:  17.33912420272827 ***
> 
> That's with Python 3.5 on an Intel I5-6600k 3.9Ghz cpu and 16GB DDR4
> memory.



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Looping [was Re: Python and the need for speed]

[Terry Reedy]

On 4/16/2017 11:35 AM, Michael Torrie wrote:

On 04/16/2017 07:57 AM, bartc wrote:

But people just don't want it.

/That/ is what surprises me, when people reject things that to me are
no-brainers.


Whereas to me, it is a no-brainer that we are better off *without* 
multiple while/loop constructs.



I simply don't care about these missing loop constructs.


I do ;-)  I consider the current simplicity a feature.

> Python works

great for what I use it for, and apparently works well for many people.


The great majority* of 'repetition with variation' is sequentially 
processing items from a collection.  Python does that nicely with 'for 
item in collection: process(item)'.  While-loops take care of everthing 
else.


*I grepped 3.6.1 .../lib/*.py with the REs '^ *while ' and '^ *for ', 
recursing into subdirectories, including tests and a few packages in 
site-packages.  These got 1389 and 9842 hits respectively. I am opposed 
to adding syntax to subdivide the 12% of looping using 'while'.  (As a 
note, '^ *while True' had 363 hits, about 1/4 of while loops.)


> I have yet to find a loop that I couldn't construct with Python's
> apparently-limited constructs.

For-loops, 'limited' to iterating through collections (iterables), cover 
at least 80% of cases.  While-loops + continue/break easily cover the 
remaining cases of linear repetition.  Branching repetition, as in naive 
Fibonacci calculation and tree processing, is more easily done with 
recursion.


--
Terry Jan Reedy

--
https://mail.python.org/mailman/listinfo/python-list

Re: Looping [was Re: Python and the need for speed]

[Rustom Mody]

On Sunday, April 16, 2017 at 7:27:49 PM UTC+5:30, bartc wrote:

> Technically, adding this one feature to Python /is/ trivial, 
^
You are not paying attention bart and I am not likely to pursue this beyond this
post. I tried to say as are others that the substantive reasons to reject a 
feature  are mostly non-technical

BTW I regard Steven's long list of things that youve missed such as regression
tests, docs etc to be somewhat off the mark
To see that try this experiment:
Just add a feature to python that matters to you along with all these 
requirements: docs, tests etc etc
Your chances of acceptance may go up marginally but most likely it will still
not be accepted.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Looping [was Re: Python and the need for speed]

[Steve D'Aprano]

On Sun, 16 Apr 2017 11:57 pm, bartc wrote:

> Yet countless other, far more elaborate features /are/ added all the time.

Indeed. Because they are needed. Because they add functionality that Python
doesn't already have, or seriously improves the interface to that
functionality.


> Technically, adding this one feature to Python /is/ trivial, for
> example, allowing while: as a synonym for while True:, but preferably
> using a new keyword such as loop. 

"Preferably using a new keyword"?

Dear gods, you really have *no clue* about the responsibilities of
maintaining a programming language used by tens of thousands of people.

Adding your new keyword will break thousands of peoples programs. That's not
to be done lightly, on a whim, just because you don't like the spelling
`while True`.


py> loop = True
  File "", line 1
loop = True
 ^
SyntaxError: invalid syntax


> Nothing else needs to be touched. 

Right. We don't need to test this new feature, because we have such awesome
mad skillz that of course it will work perfectly first time.

Nor do we need to document this new feature. We don't need to include it in
the "What's New" documentation, or include it in the tutorial, or mention
it in the list of keywords, or in the description of the grammar.

Documentation is for people who don't already know the source code because
they wrote it.

People will work it out for themselves. Eventually. After many hours, or
days, of pulling their hair out at why their working code has suddenly
stopped working after upgrading to a new Python version.

The only thing you could do that would be more irresponsible would be to
insist on introducing this new, undocumented, untested, unnecessary feature
in a bug-fix point release.

Ah, but I'm guessing that you don't understand the need for point releases
either? With only one user, yourself, who needs releases or version
numbers?


> And  
> it could have been done right at the start when the politics was simpler.
> 
> But people just don't want it.

Damn straight. Now you get it. It's not about how easy it is to implement,
it's about whether it is needed and wanted. It isn't needed, and it isn't
wanted.


> /That/ is what surprises me, when people reject things that to me are
> no-brainers.

Well, the thing is, there are two kinds of no-brainers. There are the kind
of things that are so obviously a good idea that it requires no brain to
want it.

And then there are the kinds of things that are so obviously a bad idea that
it requires no brain to want it.

Which is this, do you think?



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Looping [was Re: Python and the need for speed]

[Steve D'Aprano]

On Sun, 16 Apr 2017 10:06 pm, bartc wrote:

> On 16/04/2017 03:51, Steve D'Aprano wrote:
>> On Sat, 15 Apr 2017 10:17 pm, bartc wrote:
> 
>>> Yes, I'm constantly surprised at this, as such syntax has a very low
>>> cost (in my last compiler, supporting 'while' for example only added 30
>>> lines to the project).
>>
>> That's the advantage of writing your own private language and having no
>> users except for yourself. You get to cut corners. Python has tens of
>> thousands of users, and doesn't have that luxury.
> 
> Here are the lines of code in my C compiler which are necessary to
> support 'while': https://pastebin.com/BYFV7EWr
> 
> (45 lines shown, but there are exactly 30 Loc if blanks are excluded.)
> 
> I'd be interested in knowing why implementing While in Python would need
> significantly more. 

Have you looked at the source code?

https://github.com/python/cpython


> (The 30 Loc figure is with support for loops /in 
> general/ already in place, and is for /adding/ a new loop statement, in
> this case 'while')

What part of *testing* and *documenting* do you not understand?

Do you have any unit tests for your compiler?  How about regression tests --
when you fix a bug, do you write a regression test to ensure it never
creeps back in?

Do you have any documentation for your compiler? Does it include doctests?
Are there any tutorials that beginners can read?

I'm guessing you don't have any of those things. The code snippet you posted
doesn't even have any comments.

Here is the declaration and comment for a regression test in Python's test
suite, checking against the return of a bug in the while statement:

def test_break_continue_loop(self):
"""This test warrants an explanation. It is a test specifically 
for SF bugs #463359 and #462937. The bug is that a 'break' 
statement executed or exception raised inside a try/except inside
a loop, *after* a continue statement has been executed in that loop,
will cause the wrong number of arguments to be popped off the stack
and the instruction pointer reset to a very small number (usually
0.) Because of this, the following test *must* written as a
function, and the tracking vars *must* be function arguments with
default values. Otherwise, the test will loop and loop.
"""

(I've reformatted the comment as a docstring to make it easier to wrap.)

The test itself is 16 lines of code, plus 8 more lines of explanation for
why the test exists. That's for just one bug. Here's a snippet of another
test, testing that when "while 0" is optimized away, the else clause is
not:

# Issue1920: "while 0" is optimized away,
# ensure that the "else" clause is still present.
x = 0
while 0:
x = 1
else:
x = 2
self.assertEqual(x, 2)


Do I need to go on?

As a lone-wolf developer with a user base of exactly one person, perhaps you
don't care about tests, documentations, tutorials, or even comments in your
code. But Python has multiple dozens of developers who have to understand
each others code. It is open source and receives patches and contributions
from hundreds more. It has tens or hundreds of thousands of users with high
expectations about quality.

Python's requirements are a bit more strict than "well, it seems to be
working okay, so that's good enough for me".

If Python added something like:

loop N times:
...

we would need *at least* the following:

- a test that `loop` was interpreted as a keyword;
- a test that `times` was interpreted as a keyword;
- a test that `loop 0 times` didn't execute the body at all;
- a test that `loop 1 times` executed the body exactly once;
- a test that `loop N times` executed the body exactly N times, for
  a few different (and probably randomly selected) values of N;
- a test that the statement handled negative integers and floats correctly;
- a test that the statement handled non-numeric values correctly 
  (probably by raising an exception);
- a test that `continue` worked as expected inside this statement;
- a test that `break` worked as expected;
- a test that `return` worked as expected;
- a test that `raise` worked as expected;
- a test that signals will be caught correctly inside the statement;

and possibly more.



[...]
> No, we're talking about a loop. It must be just about the simplest thing
> to implement in a language (compared with a type system, or code
> generation).

It still needs tests and documentation. In the absence of formal correctness
proofs, any code not covered by tests should be considered buggy.


> BTW supporting a dedicated endless loop was only about 20 lines in
> another project. Believe me, it is /nothing/. I wish other aspects were
> just as trivial. It didn't even need a dedicated keyword.

Of course it's "nothing" for you, since by the evidence given you don't
bother with the hard parts. No comments, no communication with other

Re: Static typing [was Re: Python and the need for speed]

[justin walters]

On Sun, Apr 16, 2017 at 8:46 AM, bartc  wrote:

> What were the results with Python on your machine?



Well, at first I tried to implement it with a generator. I gave up on
waiting for the program to complete
after about 6 minutes.

After using your code almost exactly I get:
>>> Sum:  149985000
>>> ***Execution Time:  17.33912420272827 ***

That's with Python 3.5 on an Intel I5-6600k 3.9Ghz cpu and 16GB DDR4 memory.

> (Nice-looking language, I hadn't heard of it.)

Yeah, it's pretty neat. Some parts of Nim aren't really production ready
yet, but I think it has a bright future.

proc test() =
> var
> sum = 0
> a = 0
> b = 0
> for i in 0..<1:
>
>
Loop iterations should be 100 million, if this is one less. Missing one out
> won't affect the timing, but will give a different result if comparing
> implementations to see if they do the same thing.
>

It's not one less. It will execute exactly 100 million times. Meaning it
will execute when i is
99,999,999 but not when i is 100,000,000. This gives exactly 100,000,000
iterations including
the iteration for i = 0. It could alternatively be written as:


for i in 1..1:


Didn't realize I had messed up the increment on var b with my Nim
implementation!

Fixed:

===
import
times

proc add(a, b: int): int =
result = a + b

proc test() =
var
sum = 0
a = 0
b = 0
for i in 0..<1:
sum += add(a, b)
a += 1
b += 2
echo "a: " & $a & " b: " & $b & "\n"
echo "Sum: " & $sum

when isMainModule:
var t0 = cpuTime()
test()
var t1 = cpuTime()
echo "***Execution Time: " & $(t1 - t0) & "***\n"
===

The above Nim code is a tad bit slower with the larger var b:
>>> a: 1 b: 2
>>> Sum: 149985000
>>> ***Execution Time: 2.888533***

With Speed Optimization:
>>> a: 1 b: 2
>>> Sum: 149985000
>>> ***Execution Time: 2.843629***

With size optimization:
>>> a: 1 b: 2
>>> Sum: 149985000
>>> ***Execution Time: 2.844876***

Compiled with release flag:
>>> a: 1 b: 2
>>> Sum: 149985000
>>> ***Execution Time: 2.842312***

> Hmm, the optimiser is similar to mine then!

I'm pretty new when it comes to compilers. Been trying to learn more and
build my own but
progress is fairly slow. I'm not sure what `nimc` is doing under the hood,
but I do know that it
transpiles to C before running either `gcc` or `clang`.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Looping [was Re: Python and the need for speed]

[bartc]

On 16/04/2017 15:22, Chris Angelico wrote:

On Sun, Apr 16, 2017 at 11:57 PM, bartc  wrote:

Technically, adding this one feature to Python /is/ trivial, for example,
allowing while: as a synonym for while True:, but preferably using a new
keyword such as loop. Nothing else needs to be touched. And it could have
been done right at the start when the politics was simpler.

But people just don't want it.

/That/ is what surprises me, when people reject things that to me are
no-brainers.


Maybe it's not a no-brainer to them. Maybe you need to make the case that:

1) It's worth stopping people from using the word "loop" as a variable


Actually I think 'loop' could still be used as a variable, since 'name:' 
wouldn't occur in any other contexts. But a dual purpose 'loop' wouldn't 
be elegant.



2) It's worth documenting another type of loop
3) It's worth having everyone need to know how to read another type of loop


Compared to all the other stuff in Python, seriously? Lambdas, 
listcomps, generators, iterators, decorators, subclassing, class and 
instance attributes, ... no problemo.


But faced with:

  loop: print ("Hi!")

will lead to head-scratching?!


What's the benefit, aside from avoiding discussions like this?


Imagine if the discussion was instead about introducing a negate 
operator so that you could write:


  a = -b

instead of having to write:

  a = 0 - b

It wouldn't be a big deal having to do the latter, but it's nicer not to.

I just find writing 'while 1' (or while (1) or for(;;) elsewhere) an 
annoyance.


--
bartc
--
https://mail.python.org/mailman/listinfo/python-list

Re: Static typing [was Re: Python and the need for speed]

[bartc]

On 16/04/2017 16:00, justin walters wrote:

On Sun, Apr 16, 2017 at 3:55 AM, bartc  wrote:


Example C of the same silly program in Python:

def add(a,b):
return a+b

def testfn():
sum=a=b=0
for i in range(1):
sum += add(a,b)
a += 1
b += 2

print (a,b,sum)

testfn()


Timings:

A (Pure HLL**)  13   seconds  (dynamic/interpreted)
A (With ASM module)  3



B (my compiler)  0.5  (static/compiled)



C (Python 2/xrange) 30
C (Python 3)38
C (Pypy) 5




Just for fun I wanted to write this up in Nim to compare execution time.
Nim has Python-esqe syntax but is statically


(Nice-looking language, I hadn't heard of it.)


typed and compiled. I think this is relevant to the discussion.

Code looks like this:

```
import times

proc add(a, b: int): int =
result = a + b

proc test() =
var
sum = 0
a = 0
b = 0
for i in 0..<1:


Loop iterations should be 100 million, if this is one less. Missing one 
out won't affect the timing, but will give a different result if 
comparing implementations to see if they do the same thing.


With 100M, the results are (of a,b and sum):

1 2 149985000


sum += add(a, b)
a += 1
b += 1


b += 2, both to the ensure the same output, and in case there's a sneaky 
optimisation it can do for b+=1...



echo "a: " & $a & " b: " & $b & "\n"
echo "Sum: " & $sum

when isMainModule:
var t0 = cpuTime()
test()
var t1 = cpuTime()
echo "***Execution Time: " & $(t1 - t0) & "***\n"
```


No optimization: ***Execution Time: 2.876923***
Optimized for speed: ***Execution Time: 2.844163***
Optimized for size: ***Execution Time: 2.844901***


Hmm, the optimiser is similar to mine then!


Release option: ***Execution Time: 2.844021***

So, generally around 2.8 seconds.

Not too bad for a GC'd language. There are probably some more optimizations
I could make to improve execution time.


What were the results with Python on your machine?

--
bartc
--
https://mail.python.org/mailman/listinfo/python-list

Re: Looping [was Re: Python and the need for speed]

[Michael Torrie]

On 04/16/2017 07:57 AM, bartc wrote:
> But people just don't want it.
> 
> /That/ is what surprises me, when people reject things that to me are 
> no-brainers.

I simply don't care about these missing loop constructs.  Python works
great for what I use it for, and apparently works well for many people.
I have yet to find a loop that I couldn't construct with Python's
apparently-limited constructs.  I guess I don't see what having those
extra looping constructs will give me in terms of productivity or even
ease of use.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Static typing [was Re: Python and the need for speed]

[justin walters]

On Sun, Apr 16, 2017 at 3:55 AM, bartc  wrote:

> Example C of the same silly program in Python:
>
> def add(a,b):
> return a+b
>
> def testfn():
> sum=a=b=0
> for i in range(1):
> sum += add(a,b)
> a += 1
> b += 2
>
> print (a,b,sum)
>
> testfn()
>
>
> Timings:
>
> A (Pure HLL**)  13   seconds  (dynamic/interpreted)
> A (With ASM module)  3
> A (With ASM module)  2(older version; hmmm...)
>
> B (my compiler)  0.5  (static/compiled)
> B (via C/gcc-O3) 0.14
>
> C (Python 2)   163
> C (Python 2/xrange) 30
> C (Python 3)38
> C (Pypy) 5
>


Just for fun I wanted to write this up in Nim to compare execution time.
Nim has Python-esqe syntax but is statically
typed and compiled. I think this is relevant to the discussion.

Code looks like this:

```
import times

proc add(a, b: int): int =
result = a + b

proc test() =
var
sum = 0
a = 0
b = 0
for i in 0..<1:
sum += add(a, b)
a += 1
b += 1
echo "a: " & $a & " b: " & $b & "\n"
echo "Sum: " & $sum

when isMainModule:
var t0 = cpuTime()
test()
var t1 = cpuTime()
echo "***Execution Time: " & $(t1 - t0) & "***\n"
```


No optimization: ***Execution Time: 2.876923***
Optimized for speed: ***Execution Time: 2.844163***
Optimized for size: ***Execution Time: 2.844901***
Release option: ***Execution Time: 2.844021***

So, generally around 2.8 seconds.

Not too bad for a GC'd language. There are probably some more optimizations
I could make to improve execution time.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Looping [was Re: Python and the need for speed]

[Chris Angelico]

On Sun, Apr 16, 2017 at 11:57 PM, bartc  wrote:
> Technically, adding this one feature to Python /is/ trivial, for example,
> allowing while: as a synonym for while True:, but preferably using a new
> keyword such as loop. Nothing else needs to be touched. And it could have
> been done right at the start when the politics was simpler.
>
> But people just don't want it.
>
> /That/ is what surprises me, when people reject things that to me are
> no-brainers.

Maybe it's not a no-brainer to them. Maybe you need to make the case that:

1) It's worth stopping people from using the word "loop" as a variable
2) It's worth documenting another type of loop
3) It's worth having everyone need to know how to read another type of loop

What's the benefit, aside from avoiding discussions like this?

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Looping [was Re: Python and the need for speed]

[bartc]

On 16/04/2017 13:22, Rustom Mody wrote:

On Sunday, April 16, 2017 at 5:36:28 PM UTC+5:30, bartc wrote:

On 16/04/2017 03:51, Steve D'Aprano wrote:

On Sat, 15 Apr 2017 10:17 pm, bartc wrote:



Yes, I'm constantly surprised at this, as such syntax has a very low
cost (in my last compiler, supporting 'while' for example only added 30
lines to the project).


That's the advantage of writing your own private language and having no
users except for yourself. You get to cut corners. Python has tens of
thousands of users, and doesn't have that luxury.


Here are the lines of code in my C compiler which are necessary to
support 'while': https://pastebin.com/BYFV7EWr

(45 lines shown, but there are exactly 30 Loc if blanks are excluded.)

I'd be interested in knowing why implementing While in Python would need
significantly more. (The 30 Loc figure is with support for loops /in
general/ already in place, and is for /adding/ a new loop statement, in
this case 'while')


You walk down a mountain path (trail in US-ese?) stopping here and there to
smell a flower or spy a butterfly.
Change to driving down a mountain road — quiet for the most part — is it ok
to drive on the wrong side?
Change to an expressway is it ok to change lanes randomly, not maintain speeds
within upper (and lower!) limits?

As others have tried to point out maintaining your own 1-man-language is one 
thing
Maintaining a language
- used by industry and academia
- Used by FLOSS teams and commercial
- by kids and old-hands
- 1-man projects and large multiperson projects

is quite another

Dont get me wrong: I dont agree with everything about python.  And the 
current(est)
mess around async (and f-strings?) is (IMHO) going to cost more dearly than 
people expect...
And we will have to wait another 10 years for that decision...

My point is that you dont seem to have any estimate of the difference in
momentum between your 'its-just-another-30-lines' language and a million
user base language


Yet countless other, far more elaborate features /are/ added all the time.

Technically, adding this one feature to Python /is/ trivial, for 
example, allowing while: as a synonym for while True:, but preferably 
using a new keyword such as loop. Nothing else needs to be touched. And 
it could have been done right at the start when the politics was simpler.


But people just don't want it.

/That/ is what surprises me, when people reject things that to me are 
no-brainers.


--
bartc
--
https://mail.python.org/mailman/listinfo/python-list

Re: Static typing [was Re: Python and the need for speed]

[Chris Angelico]

On Sun, Apr 16, 2017 at 8:55 PM, bartc  wrote:
> On 16/04/2017 05:27, Steve D'Aprano wrote:
>>
>> On Sat, 15 Apr 2017 11:55 am, Rick Johnson wrote:
>>
>>> apparently, the py-devs believe we
>>> only deserve type declarations that do nothing to speed up
>>> code execution (aka: type-hints), instead of type
>>> declarations that could actually speed up the code. Go
>>> figure!
>>>
>>> I'm not a fan of forced static typing, but i am a fan of
>>> optional static typing.
>>
>>
>> Are you aware that optional static typing CANNOT be used for optimization?
>>
>> Since it is *optional*, it is only a hint, not a fact. You can tell the
>> compiler that you believe that n will be an int, but it's not guaranteed.
>
>
> No, Rick distinguished between hints, and actual declarations. It doesn't
> matter if the latter are optional.
>
> I played around with this at one time. All variables were of type 'variant',
> unless they had an explicit type declaration.

All of this is ignoring one very important point: subclassing. If I
declare a variable "int", can it or can it not accept a subclass of
int? Hint: neither answer is correct for the general case.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Looping [was Re: Python and the need for speed]

[Rustom Mody]

On Sunday, April 16, 2017 at 5:36:28 PM UTC+5:30, bartc wrote:
> On 16/04/2017 03:51, Steve D'Aprano wrote:
> > On Sat, 15 Apr 2017 10:17 pm, bartc wrote:
> 
> >> Yes, I'm constantly surprised at this, as such syntax has a very low
> >> cost (in my last compiler, supporting 'while' for example only added 30
> >> lines to the project).
> >
> > That's the advantage of writing your own private language and having no
> > users except for yourself. You get to cut corners. Python has tens of
> > thousands of users, and doesn't have that luxury.
> 
> Here are the lines of code in my C compiler which are necessary to 
> support 'while': https://pastebin.com/BYFV7EWr
> 
> (45 lines shown, but there are exactly 30 Loc if blanks are excluded.)
> 
> I'd be interested in knowing why implementing While in Python would need 
> significantly more. (The 30 Loc figure is with support for loops /in 
> general/ already in place, and is for /adding/ a new loop statement, in 
> this case 'while')

You walk down a mountain path (trail in US-ese?) stopping here and there to
smell a flower or spy a butterfly.
Change to driving down a mountain road — quiet for the most part — is it ok
to drive on the wrong side?
Change to an expressway is it ok to change lanes randomly, not maintain speeds
within upper (and lower!) limits?

As others have tried to point out maintaining your own 1-man-language is one 
thing
Maintaining a language
- used by industry and academia
- Used by FLOSS teams and commercial
- by kids and old-hands
- 1-man projects and large multiperson projects

is quite another

Dont get me wrong: I dont agree with everything about python.  And the 
current(est)
mess around async (and f-strings?) is (IMHO) going to cost more dearly than 
people expect...
And we will have to wait another 10 years for that decision...

My point is that you dont seem to have any estimate of the difference in 
momentum between your 'its-just-another-30-lines' language and a million
user base language
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Looping [was Re: Python and the need for speed]

[bartc]

On 16/04/2017 03:51, Steve D'Aprano wrote:

On Sat, 15 Apr 2017 10:17 pm, bartc wrote:



Yes, I'm constantly surprised at this, as such syntax has a very low
cost (in my last compiler, supporting 'while' for example only added 30
lines to the project).


That's the advantage of writing your own private language and having no
users except for yourself. You get to cut corners. Python has tens of
thousands of users, and doesn't have that luxury.


Here are the lines of code in my C compiler which are necessary to 
support 'while': https://pastebin.com/BYFV7EWr


(45 lines shown, but there are exactly 30 Loc if blanks are excluded.)

I'd be interested in knowing why implementing While in Python would need 
significantly more. (The 30 Loc figure is with support for loops /in 
general/ already in place, and is for /adding/ a new loop statement, in 
this case 'while')



For language developers with responsibilities to users, the job doesn't stop
at just implementing the feature in the language. It also needs to be
tested and documented.

As a general rule of thumb, every line of production code should expect to
add at least ten lines of test code (unit tests, regression tests,
integration tests, functional tests, doc tests, etc). So for your "30
lines" feature, that adds 300 lines of tests, and probably another page or
two of documentation:


No, we're talking about a loop. It must be just about the simplest thing 
to implement in a language (compared with a type system, or code 
generation).


BTW supporting a dedicated endless loop was only about 20 lines in 
another project. Believe me, it is /nothing/. I wish other aspects were 
just as trivial. It didn't even need a dedicated keyword.


> - describing the various syntax forms;
> - explaining how they differ;
> - tutorials for beginners showing each form;

And you don't have to explain how an endless loop should be written as 
'while True', meanwhile advising against using 'while 1'?



The more choices you offer, the harder that decision becomes:

- numeric Pascal or C-style loop
- foreach style loop
- repeat while condition (test at start)
- repeat until condition (test at start)
- do ... while condition (test at end)
- do ... until condition (test at end)
- repeat forever


So, how many kinds of sequences do you have in Python?

lists
tuples
namedtuples
arrays
bytearrays
string ?
memoryview

plus all those ordered types. My head is already spinning!

The loops I use are categorised as:

* Endless
* N times
* Until some condition is true
* Iterate over an integer sequence
* Iterate over a set of values of some object

Other languages like to have even more elaborate schemes. That includes 
advanced uses of Python's for loop, were, for example, there are 
multiple loop variables.


I wonder how much testing that took to get it right?


I don't remember the language, but I remember seeing one generalisation of
the repeat/do loop that puts the test in the middle, rather than at the
start or end of the loop. If I remember it was something like:

DO
setup code  # executed once only
REPEAT
loop body  # before the test
WHILE condition  # test
loop body  # after the test

thus combining both the

repeat while condition:
...

and

do:
...
until condition


styles of looping in one handy syntax.


Perhaps you misremembered as that looks too unwieldy to be practical. If 
implemented orthogonally, then a loop like this:


  While A Do B End

where both A and B can be any sequence of statements or expressions, 
would allow you to write:


  While a; b; c; d Do e; f; g End

with the test (the value of 'd') effectively in the middle.


But very common requirements are endless loops, and repeat N times
without needing an explicit counter.


If by "very common" you mean "occasionally", I agree.


Most of my loops start off as endless loops, until I can determine the 
actual terminating condition, and where it best goes. Sometimes they 
stay as endless loops.


(Sometimes, I turn a normal statement into an endless loop with a 'do' 
prefix. This is an unusual feature but I use it quite a bit:


   doswitch nextchar()   # looping version of 'switch'
   when 'A'..'Z' then 
   else exit # ie. break out of loop
   end
)


Python's byte-code does at least optimise out the check that '1' is
true, but that's not what the reader sees, which is 'loop while 1 is
true'. And one day it will be:

 while l:
 body

that can be mistaken for that common idiom.


The problem there is not the loop, but the foolish use of lowercase l as a
variable name.


Maybe. But if 'while 1' wasn't used, then that's one less thing to 
double-check. (I just looked at a random bunch of Python code; there 
were 25 instances of 'while True:', but 40 of 'while 1:'.)


--
bartc
--
https://mail.python.org/mailman/listinfo/python-list

Re: Looping [was Re: Python and the need for speed]

[Ben Bacarisse]

Steve D'Aprano  writes:


> I don't remember the language, but I remember seeing one generalisation of
> the repeat/do loop that puts the test in the middle, rather than at the
> start or end of the loop. If I remember it was something like:
>
> DO
> setup code  # executed once only
> REPEAT
> loop body  # before the test
> WHILE condition  # test
> loop body  # after the test
>
> thus combining both the 
>
> repeat while condition:
> ...
>
> and
>
> do:
> ...
> until condition
>
> styles of looping in one handy syntax.

Was the setup code in the loop because that seems unnecessary (I know
this is from memory)?  I have a recollection of using something like

  do
  code before test
while cond
  code after test
  od

which has much the same effect.  Algol 68 can also do this because the
while condition can be a block (any expression can be a block in A68).
The result was not to everyone's taste.

> On Sat, 15 Apr 2017 10:17 pm, bartc wrote:

>> Of course, it's possible to overdo it; if you look at Lisp, you'll lose
>> yourself in the myriad looping options.
>
> "Myriad"? As I understand it, Lisp offers just *one* more loop construct
> than the number you agree is the "minimum" needed: five.
>
> loop
> loop for
> do
> dotimes
> dolist

I suppose it depends on how you count, but BartC did talk about
options.  Here are some of the loop options in Common Lisp:

  (loop x with ...)
  (loop for i in (...) by ...)
  (loop for i downfrom 10 above x by -2 ...)
  (loop for x = ... then (f x) ...)
  (loop across ...)
  (loop for s being symbols of ...)
  (loop for s being hash-keys in ... using ...)

That's just the start.  Then you can add "doing", "if" "when", "unless",
"return", "collect", "sum[ming]", "maximizing" and others.  Or, if you
need them, you can have "initially", "finally", "repeat", "while",
"until", "always", "never" or "thereis" parts, all the time losing (some
of) those parts you don't need.


-- 
Ben.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Static typing [was Re: Python and the need for speed]

[bartc]

On 16/04/2017 05:27, Steve D'Aprano wrote:

On Sat, 15 Apr 2017 11:55 am, Rick Johnson wrote:


apparently, the py-devs believe we
only deserve type declarations that do nothing to speed up
code execution (aka: type-hints), instead of type
declarations that could actually speed up the code. Go
figure!

I'm not a fan of forced static typing, but i am a fan of
optional static typing.


Are you aware that optional static typing CANNOT be used for optimization?

Since it is *optional*, it is only a hint, not a fact. You can tell the
compiler that you believe that n will be an int, but it's not guaranteed.


No, Rick distinguished between hints, and actual declarations. It 
doesn't matter if the latter are optional.


I played around with this at one time. All variables were of type 
'variant', unless they had an explicit type declaration.


But it got complicated. I decided to keep the dynamic language pure, as 
after all I had a statically compiled language with exactly the same 
syntax if I needed it faster!


Examples A and B here: https://pastebin.com/D89zP3LF

Example C of the same silly program in Python:

def add(a,b):
return a+b

def testfn():
sum=a=b=0
for i in range(1):
sum += add(a,b)
a += 1
b += 2

print (a,b,sum)

testfn()


Timings:

A (Pure HLL**)  13   seconds  (dynamic/interpreted)
A (With ASM module)  3
A (With ASM module)  2(older version; hmmm...)

B (my compiler)  0.5  (static/compiled)
B (via C/gcc-O3) 0.14

C (Python 2)   163
C (Python 2/xrange) 30
C (Python 3)38
C (Pypy) 5

(Python 2 with 'range' locked up my machine for nearly 3 minutes because 
it presumably exhausted the 4GB memory just executing a simple loop. 
Have I mentioned that building a massive list just to execute 'for' was 
a stupid idea?)


The pypy timing is not bad, however it is misleading: you can't for 
example deduce the execution time of one iteration by dividing by 100 
million, as you can with the others (gcc-O3 excepted as I've no idea 
what it does). You only get that timing if you actually wanted 100 
million iterations.



(That doesn't necessarily mean that you have to use type declarations, like
in C, Pascal and Java. A good modern compiler can infer types, like in ML
and Haskell.)


Yes a good, but more sophisticated, compiler. Unlike mine!

(** The pure HLL timing might be reduced to ~ 9 seconds using 
'label-pointers' for dispatching, but this limited portability when 
converted to C code, as not all C compilers support them. But it doesn't 
matter as I can use the ASM version for speed.


I believe CPython also makes use of label-pointers when compiled with 
gcc - as happens on Linux - but not when compiled with MSVC, so that 
CPython on Windows is a little slower for that reason. I don't know if 
this is still the case.


Those Python timings were on Windows...)

--
bartc
--
https://mail.python.org/mailman/listinfo/python-list

Re: Static typing [was Re: Python and the need for speed]

[alister]

On Sun, 16 Apr 2017 09:48:15 +, alister wrote:

> On Sun, 16 Apr 2017 14:27:28 +1000, Steve D'Aprano wrote:
> 
>> On Sat, 15 Apr 2017 11:55 am, Rick Johnson wrote:
>> 
>>> apparently, the py-devs believe we only deserve type declarations that
>>> do nothing to speed up code execution (aka: type-hints), instead of
>>> type declarations that could actually speed up the code. Go figure!
>>> 
>>> I'm not a fan of forced static typing, but i am a fan of optional
>>> static typing.
>> 
>> Are you aware that optional static typing CANNOT be used for
>> optimization?
>> 
>> Since it is *optional*, it is only a hint, not a fact. You can tell the
>> compiler that you believe that n will be an int, but it's not
>> guaranteed. As the Dude says, the type hint is
>> 
>> that's, like, only your opinion man
>> 
>> and the compiler cannot trust it. It must still guard the operation
>> with a runtime check that n actually is an int, and so you lose most or
>> all of the benefit of telling the compiler that it's an int.
>> 
>> There are clever Just In Time compiler tricks that can optimize the
>> code, which is what Psycho did and PyPy still does, but they don't need
>> type hints. Type hints are superfluous to a JIT compiler, since it
>> already knows the runtime type information. But JIT compilers have
>> their limitations, like the necessity of warm up time, increased
>> complexity, and much heavier memory requirements.
>> 
>> If you want to use type hints for the purposes of compile-time
>> optimizations, then typing must be compulsory and enforced by the
>> compiler.
>> No longer optional hints, but mandatory statements of indisputable
>> fact.
>> 
>> (That doesn't necessarily mean that you have to use type declarations,
>> like in C, Pascal and Java. A good modern compiler can infer types,
>> like in ML and Haskell.)
>> 
>> Python's gradual typing is not about optimization, it is about testing
>> program correctness and helping to find bugs at compile time*. As such,
>> the fact that type hints are optional and "only your opinion" doesn't
>> matter. The promise made by the type checker is:
>> 
>> IF what you say is true (the type hints are correct)
>> THEN this code will be sound
>> 
>> (or alternatively, it's unsound because you have a type mismatch, and
>> get a compile-time error). As a fallback, you still have the runtime
>> dynamic type checks.
>> 
>> Unit and functional tests are great, and necessary. Just because your
>> code compiles in C, Java or Pascal doesn't mean that it is correct. You
>> still have to run the code and test that it does what you expect.
>> 
>> "It compiles! Quick, ship it!" -- attributed to Microsoft developers
>> 
>> Nevertheless, the problem with testing is that it can only prove the
>> presence of bugs, not their absence. A test that fails proves that your
>> code has at least one bug. A test that passes doesn't prove that there
>> are no bugs.
>> 
>> Static type checking helps to shrink the gap between "all of my tests
>> pass,
>> so there are no known bugs" and "there are no unknown bugs". And that's
>> an important and powerful tool, especially for large and complex
>> programs.
>> 
>> Python's type-hints are for writing correct code, not fast code. I find
>> it amusing when people argue that Python's type hints are pointless or
>> useless. Do they realise they're effectively arguing that correct,
>> bug-free code is pointless?
>> 
>> "Your code is buggy. It calculates the wrong result."
>> 
>> "Maybe so, but look how fast it does it!"
> 
> I think you have (deliberately?) miss understood RR by optional Static
> typing I think he means you don't have to use it as is in suing static
> types is optional)  but when you define a variable as static then its
> type is fixed, not simply a hint.
> 
> whether or not this could increase code deficiency or complicate the
> compiler further I have no idea

Actually isn't this a feature of cython?



-- 
Armadillo:
To provide weapons to a Spanish pickle.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Static typing [was Re: Python and the need for speed]

[alister]

On Sun, 16 Apr 2017 14:27:28 +1000, Steve D'Aprano wrote:

> On Sat, 15 Apr 2017 11:55 am, Rick Johnson wrote:
> 
>> apparently, the py-devs believe we only deserve type declarations that
>> do nothing to speed up code execution (aka: type-hints), instead of
>> type declarations that could actually speed up the code. Go figure!
>> 
>> I'm not a fan of forced static typing, but i am a fan of optional
>> static typing.
> 
> Are you aware that optional static typing CANNOT be used for
> optimization?
> 
> Since it is *optional*, it is only a hint, not a fact. You can tell the
> compiler that you believe that n will be an int, but it's not
> guaranteed. As the Dude says, the type hint is
> 
> that's, like, only your opinion man
> 
> and the compiler cannot trust it. It must still guard the operation with
> a runtime check that n actually is an int, and so you lose most or all
> of the benefit of telling the compiler that it's an int.
> 
> There are clever Just In Time compiler tricks that can optimize the
> code, which is what Psycho did and PyPy still does, but they don't need
> type hints. Type hints are superfluous to a JIT compiler, since it
> already knows the runtime type information. But JIT compilers have their
> limitations, like the necessity of warm up time, increased complexity,
> and much heavier memory requirements.
> 
> If you want to use type hints for the purposes of compile-time
> optimizations, then typing must be compulsory and enforced by the
> compiler.
> No longer optional hints, but mandatory statements of indisputable fact.
> 
> (That doesn't necessarily mean that you have to use type declarations,
> like in C, Pascal and Java. A good modern compiler can infer types, like
> in ML and Haskell.)
> 
> Python's gradual typing is not about optimization, it is about testing
> program correctness and helping to find bugs at compile time*. As such,
> the fact that type hints are optional and "only your opinion" doesn't
> matter. The promise made by the type checker is:
> 
> IF what you say is true (the type hints are correct)
> THEN this code will be sound
> 
> (or alternatively, it's unsound because you have a type mismatch, and
> get a compile-time error). As a fallback, you still have the runtime
> dynamic type checks.
> 
> Unit and functional tests are great, and necessary. Just because your
> code compiles in C, Java or Pascal doesn't mean that it is correct. You
> still have to run the code and test that it does what you expect.
> 
> "It compiles! Quick, ship it!" -- attributed to Microsoft developers
> 
> Nevertheless, the problem with testing is that it can only prove the
> presence of bugs, not their absence. A test that fails proves that your
> code has at least one bug. A test that passes doesn't prove that there
> are no bugs.
> 
> Static type checking helps to shrink the gap between "all of my tests
> pass,
> so there are no known bugs" and "there are no unknown bugs". And that's
> an important and powerful tool, especially for large and complex
> programs.
> 
> Python's type-hints are for writing correct code, not fast code. I find
> it amusing when people argue that Python's type hints are pointless or
> useless. Do they realise they're effectively arguing that correct,
> bug-free code is pointless?
> 
> "Your code is buggy. It calculates the wrong result."
> 
> "Maybe so, but look how fast it does it!"

I think you have (deliberately?) miss understood RR
by optional Static typing I think he means you don't have to use it as is 
in suing static types is optional)  but when you define a variable as 
static then its type is fixed, not simply a hint.

whether or not this could increase code deficiency or complicate the 
compiler further I have no idea




-- 
Fay: The British police force used to be run by men of integrity.
Truscott: That is a mistake which has been rectified.
-- Joe Orton, "Loot"
-- 
https://mail.python.org/mailman/listinfo/python-list

Static typing [was Re: Python and the need for speed]

[Steve D'Aprano]

On Sat, 15 Apr 2017 11:55 am, Rick Johnson wrote:

> apparently, the py-devs believe we
> only deserve type declarations that do nothing to speed up
> code execution (aka: type-hints), instead of type
> declarations that could actually speed up the code. Go
> figure!
> 
> I'm not a fan of forced static typing, but i am a fan of
> optional static typing.

Are you aware that optional static typing CANNOT be used for optimization?

Since it is *optional*, it is only a hint, not a fact. You can tell the
compiler that you believe that n will be an int, but it's not guaranteed.
As the Dude says, the type hint is 

that's, like, only your opinion man

and the compiler cannot trust it. It must still guard the operation with a
runtime check that n actually is an int, and so you lose most or all of the
benefit of telling the compiler that it's an int.

There are clever Just In Time compiler tricks that can optimize the code,
which is what Psycho did and PyPy still does, but they don't need type
hints. Type hints are superfluous to a JIT compiler, since it already knows
the runtime type information. But JIT compilers have their limitations,
like the necessity of warm up time, increased complexity, and much heavier
memory requirements.

If you want to use type hints for the purposes of compile-time
optimizations, then typing must be compulsory and enforced by the compiler.
No longer optional hints, but mandatory statements of indisputable fact.

(That doesn't necessarily mean that you have to use type declarations, like
in C, Pascal and Java. A good modern compiler can infer types, like in ML
and Haskell.)

Python's gradual typing is not about optimization, it is about testing
program correctness and helping to find bugs at compile time*. As such, the
fact that type hints are optional and "only your opinion" doesn't matter.
The promise made by the type checker is:

IF what you say is true (the type hints are correct)
THEN this code will be sound

(or alternatively, it's unsound because you have a type mismatch, and get a
compile-time error). As a fallback, you still have the runtime dynamic type
checks.

Unit and functional tests are great, and necessary. Just because your code
compiles in C, Java or Pascal doesn't mean that it is correct. You still
have to run the code and test that it does what you expect.

"It compiles! Quick, ship it!" -- attributed to Microsoft developers

Nevertheless, the problem with testing is that it can only prove the
presence of bugs, not their absence. A test that fails proves that your
code has at least one bug. A test that passes doesn't prove that there are
no bugs.

Static type checking helps to shrink the gap between "all of my tests pass,
so there are no known bugs" and "there are no unknown bugs". And that's an
important and powerful tool, especially for large and complex programs.

Python's type-hints are for writing correct code, not fast code. I find it
amusing when people argue that Python's type hints are pointless or
useless. Do they realise they're effectively arguing that correct, bug-free
code is pointless?

"Your code is buggy. It calculates the wrong result."

"Maybe so, but look how fast it does it!"




-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Looping [was Re: Python and the need for speed]

[Steve D'Aprano]

On Sat, 15 Apr 2017 10:17 pm, bartc wrote:

> On 15/04/2017 03:35, Rick Johnson wrote:
>> On Wednesday, April 12, 2017 at 8:44:30 AM UTC-5, bart...@gmail.com
>> wrote:
> 
>> At a minimum, every language should offer
>> the following four loop-forms (using Python semantics):
>>
>> while CONDITION:
>> doSomething()
>>
>> for VALUE in COLLECTION:
>> doSomething(value)
>>
>> loop(N):
>> doSomething()
>>
>> loop(N) as i:
>>doSomething(i)


What an extravagant, unnecessary waste of syntax. The last two just
duplicate the functionality of the second. AND Rick has forgotten about the
do...while form that executes the block at least once.

I've used, and to be honest really enjoyed using, a language with multiple
loop syntax like this. I can see some advantage in offering specialist
English-like loops in a programming language for non-programmers and
amateurs. But having a plethora of forms for what is essentially the same
thing is not so helpful for more skilled programmers who are more
comfortable with composing code rather than memorising fixed statements.


repeat forever:
...


is no more expressive than

while True:
...


since they're both just the same thing. You can mechanically and trivially
translate one to the other with no intelligence needed. So it becomes a
mere matter of personal taste as to which you prefer.


> Yes, I'm constantly surprised at this, as such syntax has a very low
> cost (in my last compiler, supporting 'while' for example only added 30
> lines to the project).

That's the advantage of writing your own private language and having no
users except for yourself. You get to cut corners. Python has tens of
thousands of users, and doesn't have that luxury.

For language developers with responsibilities to users, the job doesn't stop
at just implementing the feature in the language. It also needs to be
tested and documented.

As a general rule of thumb, every line of production code should expect to
add at least ten lines of test code (unit tests, regression tests,
integration tests, functional tests, doc tests, etc). So for your "30
lines" feature, that adds 300 lines of tests, and probably another page or
two of documentation:

- describing the various syntax forms;
- explaining how they differ;
- tutorials for beginners showing each form;

plus the on-going burden for every single user of the language, for ever, in
having to decide which form they need to use in any specific circumstance.

The more choices you offer, the harder that decision becomes:

- numeric Pascal or C-style loop
- foreach style loop
- repeat while condition (test at start)
- repeat until condition (test at start)
- do ... while condition (test at end)
- do ... until condition (test at end)
- repeat forever


I don't remember the language, but I remember seeing one generalisation of
the repeat/do loop that puts the test in the middle, rather than at the
start or end of the loop. If I remember it was something like:

DO
setup code  # executed once only
REPEAT
loop body  # before the test
WHILE condition  # test
loop body  # after the test

thus combining both the 

repeat while condition:
...

and

do:
...
until condition


styles of looping in one handy syntax.


> Of course, it's possible to overdo it; if you look at Lisp, you'll lose
> yourself in the myriad looping options.

"Myriad"? As I understand it, Lisp offers just *one* more loop construct
than the number you agree is the "minimum" needed: five.

loop
loop for
do
dotimes
dolist


https://www.tutorialspoint.com/lisp/lisp_loops.htm



> But very common requirements are endless loops, and repeat N times
> without needing an explicit counter. 

If by "very common" you mean "occasionally", I agree.


> The former /can/ be easily written 
> as:
> 
>  while 1:
>  body
> 
> but it's more psychological; I don't want to use an idiom to denote an
> endless loop, I want to be able to express it directly!

"Express it directly" is an idiom, and "while True" is just as direct
as "repeat forever".


> Python's byte-code does at least optimise out the check that '1' is
> true, but that's not what the reader sees, which is 'loop while 1 is
> true'. And one day it will be:
> 
>  while l:
>  body
> 
> that can be mistaken for that common idiom.

The problem there is not the loop, but the foolish use of lowercase l as a
variable name. It is simply poor programming practice to use easily
confused names, and the problem here is not the use of `while`.

No more than it is the use of `while` that makes this code bad:

while O000IlI1III111IllOO:
something()
O000IlI1III11I1II1IllOO = False
print(O00OIlI1III111IllOO)


Given:

mylist.append(l)
x += l
if l: ...

do you also conclude that the problem lies with append, += and `if` ?

 

-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure

Re: Python and the need for speed

[Steve D'Aprano]

On Sun, 16 Apr 2017 12:37 am, bartc wrote:

> What proportion of Python implementations depend on executing byte-code?

My guess is 100%.

Pretty much all modern interpreters of any language execute some form of
byte-code or another. The bad old days where interpreters repeatedly parsed
and executed each line of source are long gone.

PyPy generates some sort of byte-code, with a JIT compiler to generate
machine code on the fly for sufficiently "hot" code.

IronPython uses byte-code for the .Net virtual machine.

Jython uses byte-code for the Java virtual machine.

CPython uses byte-code for its own virtual machine.

There was a project for a Python interpreter for the Parrot VM, but I think
that's no longer in development.



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python and the need for speed

[Michael Torrie]

On 04/15/2017 08:37 AM, bartc wrote:
> What proportion of Python implementations depend on executing byte-code?

Presumably Nuitka does not depend on any byte code at all.  Jython uses
JVM byte codes.  Iron Python uses .net VM bytecodes.

While CPython's byte codes do take their form in part because of Python
and its syntax. But that doesn't mean that Python itself as a language
is necessarily dependent on the specifics of the CPython byte code
interpreter and architecture, though I concede that for some aspects of
Python, the behavior of the language is defined by the CPython
implementation.  The two aspects are definitely linked, but not as
solidly as you seem to think.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python and the need for speed

[Ben Bacarisse]

bartc  writes:


> 'do', in the original Algol 68 syntax, was part of its 'for'
> statement, where you could leave out the parts you don't need. The
> full syntax is something like:
>
>   for var := a to b by c while d do body od

  FOR name FROM e1 BY e2 TO e3 WHILE cond DO body OD

The significant part is the the name is bound to a value and can't be
assigned in the body.  (You don't say it can be but "var := a" makes var
look like a plain Algol68 variable.)


-- 
Ben.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python and the need for speed

[bartc]

On 15/04/2017 14:27, Marko Rauhamaa wrote:

bartc :



while 1:
body


Why not say it like it is:

   while True:
   body


but it's more psychological; I don't want to use an idiom to denote an
endless loop, I want to be able to express it directly!


C's:

for (;;)
statement

I would call an idiom.


I don't like that either!


Python's

while True:
body

is about as straightforward as it gets.


In the syntax I normally use (based on Algol 68), the endless loop is:

   do
 body
   end

(well, actually I use do body od, but that may have raised some eyebrows).

'do', in the original Algol 68 syntax, was part of its 'for' statement, 
where you could leave out the parts you don't need. The full syntax is 
something like:


  for var := a to b by c while d do body od

If you leave out the first part you get a while loop. Leave out all 
except 'to b do', you get a repeat-n-times loop. And so on.


Elegant. (I don't do it that way, I have discrete loop types, plus my 
own variations. Devising syntax is so easy, the hard bit is knowing when 
to stop...)





Python's byte-code does at least optimise


I don't understand why there's so much talk about CPython's bytecode on
this list. That should only be of interest to active CPython developers.


What proportion of Python implementations depend on executing byte-code?

--
Bartc
--
https://mail.python.org/mailman/listinfo/python-list

Re: Python and the need for speed

[Rick Johnson]

On Saturday, April 15, 2017 at 7:17:55 AM UTC-5, bartc wrote:
> On 15/04/2017 03:35, Rick Johnson wrote:
> > On Wednesday, April 12, 2017 at 8:44:30 AM UTC-5, bart...@gmail.com wrote:
>
> > At a minimum, every language should offer
> > the following four loop-forms (using Python semantics):
> >
> > while CONDITION:
> > doSomething()
> >
> > for VALUE in COLLECTION:
> > doSomething(value)
> >
> > loop(N):
> > doSomething()
> >
> > loop(N) as i:
> >doSomething(i)
> >
>
> Yes, I'm constantly surprised at this, as such syntax has a
> very low cost (in my last compiler, supporting 'while' for
> example only added 30 lines to the project).  Of course,
> it's possible to overdo it; if you look at Lisp, you'll
> lose yourself in the myriad looping options.  But very
> common requirements are endless loops, and repeat N times
> without needing an explicit counter. The former /can/ be
> easily written as:
>
>  while 1:
>  body
>
> but it's more psychological; I don't want to use an idiom
> to denote an endless loop, I want to be able to express it
> directly!


A good point. I don't like the idiom of using "always True
conditions" to denote an endless loop either. So it's just a
matter of recognizing that loop() without an "explicit
confining parameter", would be endless.

loop:
# This loop never ends.

Of course, recognizing that at some point that the loop will
need to end, or that something surprizing will cause it to
end, how then would you end it in a clean manner? Which
would cause the semantics to be:

loop:
# This loop never ends. Well "never" unless an
# exception occurs; or the hardware malfunctions; or
# melts; or unless nuclear war breaks out; or unless the
# sun expands and envelopes the earth -- except for
# those cases (and possibly a long list of others...),
# it never ends. ;-)

> Python's byte-code does at least optimise out the check
> that '1' is true, but that's not what the reader sees,
> which is 'loop while 1 is true'. And one day it will be:
>
>  while l:
>  body
>
> that can be mistaken for that common idiom.

Another good point. However that's more a consequence of
using 1 as a truth value than anything else. Python has
offered booleans for a lng time, so i don't understand
why some of us feel the need to use 0 and 1 in place of the
more intuitive (and less bug prone) False and True. The
obvious solution is:

while True:
# This loop never ends.
# (Note: see exceptions above)

Integers should only be utilized for their numeric value,
and nothing else. Python should never allow an integer to be
used as a truth value, only booleans or expressions that
evaluate to a truth value. Of course, these can perverted
also.

if 1 == 1: # superfluous! 1 will always equal 1

if None: # superfluous! None will always be False

if True == False: # superfluous! True and False will never be equal.

etc...

(of course, i _hope_ those examples are rock solid. Sometimes,
i forget how utterly and stubbornly dynamic this damned
language can be!)
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python and the need for speed

[Marko Rauhamaa]

bartc :

> Of course, it's possible to overdo it; if you look at Lisp, you'll lose
> yourself in the myriad looping options.

Funny, in Scheme, the only looping construct I use is named-let.

> The former /can/ be easily written as:
>
> while 1:
> body

Why not say it like it is:

   while True:
   body

> but it's more psychological; I don't want to use an idiom to denote an
> endless loop, I want to be able to express it directly!

C's:

for (;;)
statement

I would call an idiom.

Python's

while True:
body

is about as straightforward as it gets.

> Python's byte-code does at least optimise

I don't understand why there's so much talk about CPython's bytecode on
this list. That should only be of interest to active CPython developers.


Marko
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python and the need for speed

[Steve D'Aprano]

On Sat, 15 Apr 2017 10:23 am, bartc wrote:

> On 15/04/2017 00:40, Rick Johnson wrote:

>>   * `range(10)` will always produce a list of the _same_ 10
>>   integers.
> 
> You don't know if 'range' is still a range. If this has been executed
> first, then the answer will be 20:
> 
> oldrange=range
> def double(x): return oldrange(x*2)
> range=double

Correct.

It is true that *at the moment* CPython doesn't know if range has been
shadowed or redefined, but Victor Stinner is working on a project,
FATPython, which will. Or at least, FATPython can tell:

- there's no chance that range() has been redefined, so it is safe to
optimize the call to it;

- or there is a chance, in which case it is not safe to optimize.



>>   * `for i in range(10)` will aways iterate over the _same_
>>   10 integers
> 
> And range could be set to something else each time the loop is executed.

It could, but that won't effect the loop.

py> old_range = range
py> for i in range(5):
... range = i*100
... print(i)
...
0
1
2
3
4
py> range
400



>>   * `answer += 1` will always increment the "current integer
>>   value" by 1
> 
> I don't know if it's possible to override __iadd__ for integers so that
> +=1 does something different here.

No, Python doesn't allow the built-in types to be monkey-patched. However,
that's not the problem. The problem is that CPython doesn't know whether
answer is a built-in int or not until runtime.




-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python and the need for speed

[bartc]

On 15/04/2017 03:35, Rick Johnson wrote:

On Wednesday, April 12, 2017 at 8:44:30 AM UTC-5, bart...@gmail.com wrote:



At a minimum, every language should offer
the following four loop-forms (using Python semantics):

while CONDITION:
doSomething()

for VALUE in COLLECTION:
doSomething(value)

loop(N):
doSomething()

loop(N) as i:
   doSomething(i)



Yes, I'm constantly surprised at this, as such syntax has a very low 
cost (in my last compiler, supporting 'while' for example only added 30 
lines to the project).


Of course, it's possible to overdo it; if you look at Lisp, you'll lose 
yourself in the myriad looping options.


But very common requirements are endless loops, and repeat N times 
without needing an explicit counter. The former /can/ be easily written as:


while 1:
body

but it's more psychological; I don't want to use an idiom to denote an 
endless loop, I want to be able to express it directly!


Python's byte-code does at least optimise out the check that '1' is 
true, but that's not what the reader sees, which is 'loop while 1 is 
true'. And one day it will be:


while l:
body

that can be mistaken for that common idiom.

--
bartc
--
https://mail.python.org/mailman/listinfo/python-list

Re: Python and the need for speed

[Rick Johnson]

On Thursday, April 13, 2017 at 1:32:28 AM UTC-5, Steven D'Aprano wrote:
> On Wed, 12 Apr 2017 14:38:52 +0100, Ben Bacarisse wrote:
> > Steve D'Aprano writes:
> >> On Wed, 12 Apr 2017 03:39 am, Paul Rubin wrote:
>
> [...] Indeed, and this is a very common phenomenon:
> features which "ordinary" programmers imagine are "trendy"
> and new end up having their roots in some functional
> language dating back to the 1980s, or 70s, or in extreme
> cases the 1950s and Lisp.

Be advised that functional programmers are a fanatical
bunch, and saying something like that could be akin to
blowing a "functional purist dogwhistle". If this thread
becomes overrun with functional trolls, i'm blaming you. ;-)
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python and the need for speed

[Rick Johnson]

On Thursday, April 13, 2017 at 1:32:29 AM UTC-5, Gregory Ewing wrote:
> Jussi Piitulainen wrote:
> > Traceback (most recent call last):
> >   File "/dev/fd/63", line 37, in 
> > SanityClauseException: code is blatantly sub-optimal
> > 
> > As far as I know, no language does that. Because reasons?
> 
> Because the problem of making a compiler do that is
> probably AI-complete!

What? GvR's time machine only goes backwards?

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python and the need for speed

[Rick Johnson]

On Wednesday, April 12, 2017 at 8:44:30 AM UTC-5, bart...@gmail.com wrote:
> On Wednesday, 12 April 2017 12:56:32 UTC+1, Jussi Piitulainen  wrote:
> > bartc writes:
> > >
> > > These are straightforward language enhancements.
> >
> > FYI, the question is not how to optimize the code but how
> > to prevent the programmer from writing stupid code in the
> > first place. Someone suggested that a language should do
> > that.
> The 'stupid code' thing is a red herring. I assume the
> code people write is there for a reason.

Yeah, but you have to admit, sometimes there is no obvious
good reason -- as Steven's example code has proven. ;-)
Although, i'm convinved that any one of us could find
examples like that in our own repos. And if you cannot find
code that makes you feel embarassed, then you're not
evolving anymore.

> But the language can also play a part in not allowing
> certain things to be expressed naturally.

Exactly!

> So the for-loop in the example has to have a control-
> variable even if it's not referenced.

Yes, one of the design flaws of Python's "for loop" is that
the control variable is injected whether you need it or not
(Nightmarish images of petulant children being force-fed
castor oil comes to mind...). In fact, in Python, there are
only two loop forms avialable -- the "for" and the "while".
-- however, many other languages have relized that these two
forms are insufficent to cover the various types of loops
that are needed. At a minimum, every language should offer
the following four loop-forms (using Python semantics):

while CONDITION:
doSomething()

for VALUE in COLLECTION:
doSomething(value)

loop(N):
doSomething()

loop(N) as i:
   doSomething(i)

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python and the need for speed

[Rick Johnson]

On Wednesday, April 12, 2017 at 4:57:10 AM UTC-5, bart...@gmail.com wrote:
> On Wednesday, 12 April 2017 07:48:57 UTC+1, Steven D'Aprano  wrote:
> > On Tue, 11 Apr 2017 21:10:56 -0700, Rick Johnson wrote:
> > >
> > > high level languages like Python should make it
> > > difficult, if not impossible, to write sub- optimal code
> > > (at least in the blatantly obvious cases).
> >
> > Here's another example:
> >
> > answer = 0
> > for i in range(10):
> > answer += 1
> >
> > instead of
> >
> > answer = 10
> >
> > So... how exactly does the compiler prohibit stupid code?
>
> Actually, an optimising C compiler (not one of mine!)
> probably could reduce that to answer=10. And eliminate even
> that if 'answer' was never used.
>
> But that's not really the issue here. Assume that such a
> loop /is/ doing something more useful. The problems with
> Python (last time I looked anyway) were these:
>
> (1) If answer was a global variable, then it needs a symbol
> table lookup to find out what it is. That would not happen
> in a static language, or a less dynamic one, as you already
> have the address.

Indeed. I have argued for syntax that will resolve variable
scope many times. but apparently, the py-devs believe we
only deserve type declarations that do _nothing_ to speed up
code execution (aka: type-hints), instead of type
declarations that could actually speed up the code. Go
figure!

I'm not a fan of forced static typing, but i am a fan of
optional static typing.

The global keyword is one idiosyncrasy of Python that causes
a lot of confusion , especially to noobs, but also from a
standpoint of "general intuitiviness" for old hats. This
keyword does not define "true globals" (aka: a symbol that
can be accessed or reassigned from anywhere), no, but
rather, a symbol that is global only with respect to the
"current module scope" (aka: module level variable). You can
only get "true globals" in Python by going rogue and
injecting symbols into sys.modules like some mad scientist
-- which is not officially supported. Go figure! ;-)

> And this [global] lookup happens for every loop iteration.

I sure hope not. I have not taken the time to inspect the
inner workings of Python, but if the lookup happens every
time, that would be an awfully inefficient design.

> (2) There is also 'range', which could have been redefined
> to mean something else, so /that/ needs a lookup. The byte-
> code compiler can't just assume this loop is to be executed
> 10 times.

Yep, and more evidence that Python has taken dynamics to
such a fundamentalist extreme, that even ISIS is jealous!

# START INTERACTIVE SESSION  (Py2.x) ##
>>> def range(*args):
... return "Death to static infidels!"

>>> for i in range(10):
... print i

D
e
a
t
h

t
o

s
t
a
t
i
c

i
n
f
i
d
e
l
s
!

## END INTERACTIVE SESSION ##

> (3) This was fixed long ago but at one time, even when
> 'range' had been established to be a range, it involved
> constructing a list of items (10 here, but it could be a
> million), and then iterating over the list.

First, it was "range", then "xrange", and now, "range"
again. Third time's a charm i suppose. The last iteration of
the range function design removed the underlying
inefficiency, however, it created a semantical nightmare
(more on this below).

> This might seem crazy, but it might have been exceptable
> for a script language at one time. Not for a general
> purpose one however.

Yeah, this is one of those screw-ups that even GvR's famous
time machine could not fix. Must have been in the shop for a
tune-up...

> (4) Python's integer types being immutable, the +=
> operation means evaluating the result, then creating a new
> integer object and binding 'a' to that new value. (Correct
> me if I'm wrong.)
>
> These are all things the language could point a finger at
> before blaming the user for writing inefficient code.
>
> The problem is also the language encouraging people to use
> high-level but inefficient methods, as the emphasis is on
> productivity and readability** rather than performance. In
> fact most users probably have no idea on what is efficient
> and what isn't.

And that's the point i was trying to make earlier. Most
Python programmers are not CS majors, they are just
"technical folk" who need to write a logic program to do
this or that task. We shouldn't expect them to know about
things like memory management, neither should we
expect/burden them to know that calling range(100), just
to do one million iterations, is inefficient. Instead, we
should provide them with the proper "efficient tools" to get
the job done. So instead of:

for x in range(100): # Legacy python was slow here!
doSomething(x)

It should have _always_ been...

100.times:
doSomething()

or, if you need a counter:

100.times as x:
doSomething(x)

Or