[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Sven R. Kunze]

Hi Irit,

makes sense. So, in case of a *mixed-type ExceptionGroup,* SystemExit 
wins and forces the program to exit.



Could you add your reasoning to the PEP?


This would help future readers and illustrates the chain of thoughts. It 
might be obvious to you but from the outside it is really a long 
journey. I actually liked your wording below and the explicitly writing 
down the consequence I mentioned makes it very clear why this 
complication exists.


Best,
Sven

On 03.03.21 21:13, Irit Katriel wrote:


Hi Sven,

This is all about the popularity of "except Exception".

Look at the hierarchy of builtin exceptions: 
https://docs.python.org/3/library/exceptions.html#exception-hierarchy 
<https://docs.python.org/3/library/exceptions.html#exception-hierarchy>


Most exceptions are subclasses of Exception. There are a few which are 
not, because they typically mean "this process should exit" so you 
should not usually handle them in your code. People use "except 
Exception" as a way to "catch almost everything, but not the critical 
stuff like SystemExit".


If we make ExceptionGroup be a BaseException, then "except Exception" 
doesn't catch it. So we make it a subclass of Exception. But then we 
can't make it wrap things like SystemExit, which people expect will 
not be caught by "except Exception".  So we add BaseExceptionGroup, 
which is a subclass of BaseException and therefore is not caught by 
"except Exception", so it can wrap SystemExit.


Why is the choice automated?  Because it can be. You look at what 
you're wrapping. If it's all subclasses of Exception, then it can be 
ExceptionGroup. If there are BaseExceptions, then it needs to be 
BaseExceptionGroup. There is no reason to ever do anything else.


I hope that makes sense.


On Wed, Mar 3, 2021 at 7:32 PM Sven R. Kunze <mailto:srku...@mail.de>> wrote:


Hey Irit,

find my 3 answers below:

On 03.03.21 13:17, Irit Katriel wrote:
> Hi Sven,
>
> I like your formatting suggestion, thanks. I will do something
like that.

You're welcome.

>
> I'm not sure I understand your question. ExceptionGroup is a
subclass
> of Exception (which is a subclass of BaseException). So
ExceptionGroup
> is caught by "except Exception" or "except BaseException".

1) So I understand "try-except BaseException" (cf.
concurrent.futures)
will work without fixes (i.e. produce the same results).


> BaseExceptionGroup is a subclass only of BaseException so it is
caught
> by "except BaseException" but not "except Exception". And
> ExceptionGroup is allowed to wrap only Exceptions while
BaseException
> can wrap Exceptions and and BaseExceptions. Makes sense?


2) Can you add motivating examples for "BaseExceptionGroup vs
ExceptionGroup" in the PEP? Right now, I only see what the
consequences
are but not why it was done this way.

3) Can you explain (and show the reasoning behind) this automatic
choice
in the PEP? Sounds a bit like hidden magic to me.


Referring to: "The difference between them is that ExceptionGroup can
only wrap Exception subclasses while BaseExceptionGroup can wrap any
BaseException subclass. A factory method that inspects the nested
exceptions and selects between ExceptionGroup and BaseExceptionGroup
makes the choice automatic."


Best
Sven


PS:

the reason why I was a bit puzzled by the
BaseExceptionGroup/ExceptionGroup issue is that:
- if it doesn't matter (so we do it automatically, because we do not
want to bother anybody), why do we need ExceptionGroup at all,
BaseExceptionGroup seems more flexible?
- if it does matter, why is the choice automatic and what if it
was the
wrong choice?


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/NYBCJCQWEH3T4VMURSAP7S6AZKSFN2C7/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Sven R. Kunze]

Hey Irit,

find my 3 answers below:

On 03.03.21 13:17, Irit Katriel wrote:

Hi Sven,

I like your formatting suggestion, thanks. I will do something like that.


You're welcome.



I'm not sure I understand your question. ExceptionGroup is a subclass 
of Exception (which is a subclass of BaseException). So ExceptionGroup 
is caught by "except Exception" or "except BaseException".


1) So I understand "try-except BaseException" (cf. concurrent.futures) 
will work without fixes (i.e. produce the same results).



BaseExceptionGroup is a subclass only of BaseException so it is caught 
by "except BaseException" but not "except Exception". And 
ExceptionGroup is allowed to wrap only Exceptions while BaseException 
can wrap Exceptions and and BaseExceptions. Makes sense?



2) Can you add motivating examples for "BaseExceptionGroup vs 
ExceptionGroup" in the PEP? Right now, I only see what the consequences 
are but not why it was done this way.


3) Can you explain (and show the reasoning behind) this automatic choice 
in the PEP? Sounds a bit like hidden magic to me.



Referring to: "The difference between them is that ExceptionGroup can 
only wrap Exception subclasses while BaseExceptionGroup can wrap any 
BaseException subclass. A factory method that inspects the nested 
exceptions and selects between ExceptionGroup and BaseExceptionGroup 
makes the choice automatic."



Best
Sven


PS:

the reason why I was a bit puzzled by the 
BaseExceptionGroup/ExceptionGroup issue is that:
- if it doesn't matter (so we do it automatically, because we do not 
want to bother anybody), why do we need ExceptionGroup at all, 
BaseExceptionGroup seems more flexible?
- if it does matter, why is the choice automatic and what if it was the 
wrong choice?


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/4TEOPEX7EELLZUUFAODZ6XU7HF7FUE54/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Sven R. Kunze]

Hey Irit,

cool proposal.

I just have two questions regarding "except Exception" or "except 
BaseException" as it is used e.g. by concurrent.futures.process (def 
_process_worker) from the stdlib.


Almost similarly, I maintain a library using this pattern to wrap/unwrap 
exceptions from remote Python processes to create nicely formatted 
tracebacks (also recursively of course if needed) at the calling python 
process.


Usually these exceptions are wrapped, transferred to the parent process, 
there is the current call stack added, then reraised as a different 
exception, and so on and so forth if a chain of parents exist. The 
outermost parent process takes care of printing the tb.



My two questions regarding PEP 654:


1) What is the right "except pattern" when ExceptionGroup is introduced 
for the use cases described above (remote or concurrent python processes)?



2) What is the recommended approach of printing the traceback 
potentially incorporating multiple tracebacks - I couldn't find it in 
the PEP and tracebacks are a really neat tool for error hunting.



Best Regards,
Sven



On 23.02.21 01:24, Irit Katriel via Python-Dev wrote:


Hi all,

We would like to request feedback on PEP 654 -- Exception Groups and 
except*.


https://www.python.org/dev/peps/pep-0654/ 



It proposes language extensions that allow programs to raise and 
handle multiple unrelated
exceptions simultaneously, motivated by the needs of asyncio and other 
concurrency libraries,

but with other use cases as well.

* A new standard exception type, ExceptionGroup, to represent multiple 
exceptions with

  shared traceback.
* Updates to the traceback printing code to display (possibly nested) 
ExceptionGroups.

* A new syntax except* for handling ExceptionGroups.

A reference implementation (unreviewed) can be found at:
https://github.com/iritkatriel/cpython/pull/10 



Thank you for your help

Kind regards
Irit, Yury & Guido



___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/L5Q27DVKOKZCDNCAWRIQVOZ5DZCZHLRM/
Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BY27BOGXFGEYVXKO3CDOHHZFYRAKY2TP/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Sven R. Kunze]

Just to be on the safe side here, I would want to asked a little bit 
further.



On 02.03.21 12:22, Irit Katriel wrote:


1) What is the right "except pattern" when ExceptionGroup is
introduced for the use cases described above (remote or concurrent
python processes)?

If you want to catch the whole ExceptionGroup and format its 
traceback, then you can just do "except ExceptionGroup as eg" and then 
traceback.print_exception(eg).


The except* syntax is for when you want to handle only certain types 
of exceptions from the group, selectively.


Just to make sure, I understand this completely.


In order to make it more tangible here is an example from the stdlib:

https://github.com/python/cpython/blob/bf2e7e55d7306b1e2fce7dce767e8df5ff42cf1c/Lib/concurrent/futures/process.py#L215

As you can see, BaseException is caught multiple times as the only 
except-clause. _sendback_result(...) then used to transfer the 
return_val/exception back to parent process.



How is the supposed way of handling a raised ExceptionGroup?


a) will the existing code simply work as it?
b) if not what are the changes to this lib specifically as a blueprint 
example for others



Reading from the other subthread for this PEP, I am not 100% sure now 
that "except BaseException" will suffice if ExceptionGroup inherits from 
Exception instead of BaseException because it seems that ExceptionGroup 
somehow is handled specially. So, why I try to approach this very issue 
with existing code. Once that is clear, it should be easy for everybody 
to apply the same pattern to their code. Here is the copy from github:



try:
    r = call_item.fn(*call_item.args, **call_item.kwargs)
except BaseException as e:
    exc = _ExceptionWithTraceback(e, e.__traceback__)
    _sendback_result(result_queue, call_item.work_id, exception=exc)
else:
    _sendback_result(result_queue, call_item.work_id, result=r)
    del r


Maybe even _sendback_result could benefit from using ExceptionGroup 
itself. You can see there another "except BaseException" in case of an 
error. But that's maybe a different topic.



2) What is the recommended approach of printing the traceback
potentially incorporating multiple tracebacks - I couldn't find it
in the PEP and tracebacks are a really neat tool for error hunting.


Part of the proposal of the PEP is that we teach the builtin traceback 
formatting code to display ExceptionGroups with all that information.



As long as there's the possibility to receive the usual (two-line) 
entry-based list, I guess that is enough for every one to work with it 
(e.g. hiding unnecessary tb entries).



The reference implementation has this, and the examples in the PEP 
were produced with it. Some of the examples (the early ones) show 
exceptions that were never raised so there is no traceback. But if you 
scroll down to the later examples, you see the output for exceptions 
with tracebacks, cause, context etc.



Ah I see them now. Thank you. :-)


We didn't put too much effort into making the traceback output pretty, 
so at the moment it's a draft. If there is an ascii artist out there 
who has suggestions on improving this, that would be great.



Well, yes. It's not really pretty. I would recommend a tree-based 
solution such as the following:



Traceback (most recent call last):
  File "", line 3, in 
ExceptionGroup
|
+---+   Traceback (most recent call last):
|   | File "", line 3, in 
|   |   ExceptionGroup: eg
|   |   |
|   |   +---+ ValueError: a
|   |
|   |   During handling of the above exception, another exception occurred:
|   |
|   |   Traceback (most recent call last):
|   | File "", line 5, in 
|   |   KeyError: 'x'
|
|
+---+   Traceback (most recent call last):
|   | File "", line 3, in 
|   |   ExceptionGroup: eg
|   |   |
|   |   +---+ TypeError: b


I used a similar pattern for the remote-execution lib and received good 
user feedback (even requesting color encoding for each layer of the tree 
(not the text), so people know what belongs together). Besides colors, I 
used https://en.wikipedia.org/wiki/Box-drawing_character but I guess 
pipes, dashes and pluses could suffice for CPython.



One other remark from my side here: in the example of the PEP there seem 
to be missing a space before 'File ""'. Comparing outer tbs with 
inner tbs, you can see that the "F" of "File" is not underneath the a of 
"Traceback" but underneath the "r" of it.


Regards,
Sven

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6I2SLGQEGOI4XMWPLBUC2SMQYVKTRQJ5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Comments on PEP 558

[Sven R. Kunze]

Hi Mark,

On 04.02.21 12:47, Mark Shannon wrote:

Hi Sven,

On 04/02/2021 9:06 am, Sven R. Kunze wrote:
As long as it is possible to **write** to existing keys to **add new 
keys** to frame.f_locals, I am actually quite happy.


Out of interest, why would you want to add new keys to the locals of a 
function frame?



I use it for remote execution in human-friendly manner.

I plan to opensource the lib for everybody to use, so I was a worried 
that this change could break it.




The function will never be able to use those values.



I realize quite now that the use-case usually is on module-level where 
locals=globals:


>>> import sys
>>> frame=sys._getframe(0)
>>> frame.f_locals['testvar']='testvalue'
>>> print(testvar)
testvalue
>>>

So, setting a var was never an issue; also probably because it's 
seldomly used in this context.


Funny enough, that the lib would even start to work properly when 
functions-locals would be writable.


Regards,
Sven
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/S3QQKTVFXWBMAETNDNXNLYZDJABGHT63/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Comments on PEP 558

[Sven R. Kunze]

On 03.02.21 23:37, Nick Coghlan wrote:


No, PEP 558 doesn't remove it, it enhances it to be a live view of the 
frame state instead of an inconsistently updated snapshot.



As long as it is possible to **write** to existing keys to **add new 
keys** to frame.f_locals, I am actually quite happy.



The potential incompatibility Mark is referring to is the fact that 
even optimised frames currently allow writing to arbitrary keys in 
frame.f_locals and making the bound values visible to the locals() 
builtin and other consumers of frame.f_locals.


For PEP 558, it's an open question as to whether that behaviour will 
become limited to the PyEval_GetLocals() backwards compatible C API, 
with the updated Python frame API instead throwing KeyError for 
attempts to write to unknown keys on optimised frames.


So, it seems the restricting behavior is for special cases only (not 
including with-statements which I would care about).



I just wanted to mention that accessing problematic parts of the 
interpreter's internals can be quite helpful when doing concept-proofing 
or working the way through until a "batteries-included" solution exist.


Thanks,
Sven

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/AFOA54RN22MVWNL2AYKOYL27MHQE45KZ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Comments on PEP 558

[Sven R. Kunze]

Hi Mark,

I've been working on a project heavily relying on frame.f_locals.

Are you planning to remove it?


On 30.01.21 13:18, Mark Shannon wrote:


Given that f_locals is broken, why is keeping compatibility for this 
obscure, and probably unused case worthwhile?


The break in compatibility with locals() seems much more intrusive, 
yet you are OK with that (as am I).



Best,
Sven
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/VXLTLMEBWP6MYPQHMPIYAK7E2QVA3OFF/
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-Dev] Examples for PEP 572

[Sven R. Kunze]

On 04.07.2018 21:18, Steven D'Aprano wrote:

Read the Appendix to the PEP:
https://github.com/python/peps/blob/master/pep-0572.rst


Yes, I did after I realized where that example came from. But my point 
was actually to understand the evaluation order because Uncle Timmy 
won't be around to explain when new code appears.



And no, total is not always not equal to total.


Err, yeah. Double negation rules. ;-)


I read it as:

 while total != updated total:
 do stuff

and find it easier to follow than having to juggle the extra
book-keeping "old" variable in the original code.


updated_total rocks. Reminds me of those pattern, we usually use in 
those cases.


What just confused me is the evaluation order. It seems to me that it's 
like left-to-right first and then assignment expression.



Using some math-style-inspired markers (execution-irrelevant) would be cool:

while total != (total' := total + term):
    do stuff

total and total' can be different at the same whereas total is total (at 
least in my mental model).

But it seems I need to adapt here.

Regards,
Sven

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Examples for PEP 572

[Sven R. Kunze]

Sorry for adding yet another mail. :-(

On 04.07.2018 10:54, Serhiy Storchaka wrote:

Sorry, this PEP was rewritten so many times that I missed your Appendix.


while total != (total := total + term):
    term *= mx2 / (i*(i+1))
    i += 2
return total




This very example here caught my eye.

Isn't total not always equal to total? What would "regular" Python have 
looked like?


Regards,
Sven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 572: Write vs Read, Understand and Control Flow

[Sven R. Kunze]

On 24.04.2018 11:21, Victor Stinner wrote:

I have been asked to express myself on the PEP 572.


+1

I knew about the relationship between read and write. But your stance on 
debugging just makes it a thing. Thanks a lot!

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 572: Assignment Expressions

[Sven R. Kunze]

On 25.04.2018 01:19, Steven D'Aprano wrote:

Sorry, gcd(diff, n) is not the "perfect name", and I will tell you that
sometimes g is better. [...]


We were talking about the real-world code snippet of Tim (as a 
justification of := ) and alternative rewritings of it without resorting 
to new syntax.
Not about "sometimes", or in some toy examples, or code without unknown 
life-time and intention (interactive, library, etc).



You might know that, but how does somebody reading the code know
which functions are pure and which are not? How does the compiler know?


There are invisible optimizations in CPython already.
And for the most obvious examples, humans don't need to know. It's like 
wondering if constant folding really works.
It's the same with all optimizations: if it works, it's fine. If it 
cannot be made working, the original code still works and you can 
optimize by hand anyway.
The two obvious ways: the compiler might have a predefined list of pure 
functions OR the code tells him with an annotation of any sort.

Both ways work, the latter somehow feels cleaner.


It's easy to say that you recognise gcd as a pure function. How about
len(document.chapter[this_chapter].pages()), is that a pure function?


What exactly do you want me to answer? An algorithm? A trained neural 
network?
Usually, one starts with the low-hanging fruits and extends if possible 
and needed.

So that would be way out of scope.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 572: Assignment Expressions

[Sven R. Kunze]

On 23.04.2018 23:41, Tim Peters wrote:

Why?  "Give the result of an expression a name" is already heavily
used in Python - it's just that the _contexts_ in which it can be done
are very limited now.


Which is a good thing IMO. It keeps stuff simple to reason about.
At least to me, the objective of an expression is to generate a
result, not to introduce new names. YMMV.


After decades, CPython still does nothing of the sort, short of having
eventually made, e.g., "None" and "True" and "False" reserved words so
at least it can optimize uses of those.  It knows nothing at all about
which library functions are pure - and there's no code in the
implementation currently capable of exploiting such information even
if it were known.  That remains a fantasy in CPython.


CPython optimizes on dicts pretty heavily and on a lot of other things 
as well.
Victor, e.g., is pretty prolific here when it comes to optimizing things 
for the 95% usecases.

Maybe, considering pure functions might be another step.


Guido gave better ones, where binding expressions would allow to
collapse arbitrarily deep levels of nesting to just one (if ... elif
... elif ... elif ...). My example only eliminated a single level of
artificial indentation.  But my example did have the advantage of
being taken verbatim from actual, working code ;-)


I like the example, but not the solution it proposes.

gcd(diff, n) is to me a perfect name, and please don't tell me g is 
better. ;)
To me it seems, that many people consider function_of_parameter a better 
name than function(parameter). IMO it isn't.
I've seen lots of code where people do things like foo_of_bar = 
bar['foo']. Because they don't want another dict access, or they think 
it better reflects the concept. But it does not as does not g. Writing 
gcd(diff, n) twice is not more repeating as is writing foo_of_bar twice. 
Because gcd is a pure function, gcd(diff, n) is just a synonym/name of 
the very same number. That it needs to be calced twice, is a detail, 
like a double dict access.


In the end, it's only optimizing what could a be done by a machine much 
more reliably.
Though, people now want to save that extra assignment line (and 
indentations) via syntax,
because they do that kind of optimizing and want to do it by hand for 
some reason.


Just my 2 cents to let you see where I'm coming from.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 572: Assignment Expressions

[Sven R. Kunze]

On 23.04.2018 23:19, Barry Warsaw wrote:

I also think it effectively solves the switch-statement problem:

if (get_response() as answer) == 'yes':
 do_it()
elif answer == 'no':
 skip_it()
elif answer == 'maybe'
 okay_then()

That’s Pythonic enough for jazz!


Is it just me or since when has the following Python code fallen out of 
favor?


answer = get_response()
if answer == 'yes':
do_it()
elif answer == 'no':
skip_it()
elif answer == 'maybe'
okay_then()

I really don't see anything I would want to optimize here. Not even a 
single bit. But as said that might just be me.


What I like about this code is:
1) symmetry: all ifs are equally structured
2) classic IPO model: first get input, then process (, then output)

Sven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 572: Assignment Expressions

[Sven R. Kunze]

On 23.04.2018 22:37, Chris Angelico wrote:

Ah, are you one of those programmers who writes code once and it's
instantly perfect? I apologize, I didn't realize I was in the presence
of a unicorn.


Wow, constructive. Nothing is perfect, but if you don't consider your 
surroundings when doing changes, well, what could possibly go wrong...



Duplication works against that by forcing you to make changes in two places.


... in the very same line, a line below, few characters after/before it.


I've seen code that relies on duplication and compiler optimizations.
Sure, it'll run just as fast as the equivalent with actual variable
names; but that's beside the point. It takes extra effort to maintain
such code, and that is what matters.


That's exactly my point. Readability is what counts, especially for 
maintaining. "gcd(diff, n)" is a great name, much better than "g", if 
you ask me.


We aren't talking about 1000 lines of code here. The new syntax will 
enable one-liner optimizations. And I think Tim's example is as good as 
we can get realistically. Because if the expressions become longer or 
more complex and/or the numbers of expressions increase, I doubt that a 
majority want to have that in a single line even though the syntax would 
allow this. And if so the editor might include some line wraps, so then 
we are where we were before. Somewhere, you need to get your hands dirty.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 572: Assignment Expressions

[Sven R. Kunze]

On 23.04.2018 19:31, Tim Peters wrote:

Surely you're joking.  This is math.gcd(), which is expensive for
multi-thousand bit integers, and the interpreter knows nothing about
it.  Adding a cache of _any_ kind (LRU or otherwise) would make it
even slower.
Alright, if that problem is just about performance, then there must be a 
better way to resolve it rather than inventing a new syntax. Btw. 
storing the result in a local var is also a cache IMHO. And if gcd is 
immutable, I think Python can do a great job here of optimizing.


Anyway, your example is the best one I've seen so far.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 572: Assignment Expressions

[Sven R. Kunze]

On 23.04.2018 19:24, Chris Angelico wrote:

On Tue, Apr 24, 2018 at 3:13 AM, Sven R. Kunze <srku...@mail.de> wrote:

diff = x - x_base
if diff and gcd(diff, n) > 1:
 return gcd(diff, n)

# or

if (x - x_base) and gcd(x - x_base, n) > 1:
 return gcd(x - x_base, n)


and have the interpreter handle the optimization, or apply an lru_cache? ;-)

And then you want to change something, and you have to make an edit in
two places. Or, worse, you make it in only one of those places, they
become desynchronized, and nobody can figure out why the program
occasionally and bizarrely fails.


If you change any of those lines (including ones of my fore-posters) 
without knowing what you're doing, you'd better don't touch them at all.


The SQL folks btw. are pretty okay'ish with this kind of duplication 
because they can resolve it. Surely, Python isn't SQL but sometimes I 
wish Python could handle such things as easily without me having to 
babysit it all the time and using Perl'ish syntax (which := looks like 
to me). We then have :=, = and ==. Sorry, but Python wouldn't fit my 
brain then.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 572: Assignment Expressions

[Sven R. Kunze]

On 23.04.2018 17:59, Steve Holden wrote:


While Tim's expression might look (superficially) like C, the 
five-line alternative isn't exactly an inspiring example of 
Pythonicity, is it?




What about

diff = x - x_base
if diff and gcd(diff, n) > 1:
return gcd(diff, n)

# or

if (x - x_base) and gcd(x - x_base, n) > 1:
return gcd(x - x_base, n)


and have the interpreter handle the optimization, or apply an lru_cache? ;-)

Cheers,
Sven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Is static typing still optional?

[Sven R. Kunze]

On 21.12.2017 11:22, Terry Reedy wrote:



@dataclass
class C:
 a: int # integer field with no default
 b: float = 0.0 # float field with a default

And the types will be recognized by type checkers such as mypy.

And I think the non-typed examples should go first in the docs.




I still don't understand why "I don't care" can be defined by "leaving out"

@dataclass
class C:
 b = 0.0 # float field with a default


For non-default fields, I like ellipsis too.

Cheer,
Sven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Sven R. Kunze]

On 23.11.2017 08:38, Ivan Levkivskyi wrote:

I think this code should be just equivalent to this code

    def g():
    temp = [(yield i) for i in range(10)]
    return (v for v in temp)

Semantics of the comprehension should be clear here (just an 
equivalent for-loop without name leaking)


Excuse me if I disagree here. If I were to understand this in real-world 
code, I cannot imagine what will happen here.


A "yield" within a comprehension is like a "return" in a comprehension. 
It makes no sense at all.

Also a "yield" and a "return with value" is also rarely seen.

Comprehensions build new objects, they are not for control flow, IMO.

Cheers,
Sven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Sven R. Kunze]

Isn't yield like a return?

A return in a list/dict/set comprehension makes no sense to me.

So, +1 on SyntaxError from me too.

Cheers.


On 22.11.2017 21:29, David Mertz wrote:
Inasmuch as I get to opine, I'm +1 on SyntaxError. There is no 
behavior for that spelling that I would find intuitive or easy to 
explain to students. And as far as I can tell, the ONLY time anything 
has ever been spelled that way is in comments saying "look at this 
weird edge case behavior in Python."


On Nov 22, 2017 10:57 AM, "Jelle Zijlstra" > wrote:




2017-11-22 9:58 GMT-08:00 Guido van Rossum >:

Wow, 44 messages in 4 hours. That must be some kind of record.

If/when there's an action item, can someone summarize for me?

The main disagreement seems to be about what this code should do:

    g = [(yield i) for i in range(3)]

Currently, this makes `g` into a generator, not a list. Everybody
seems to agree this is nonintuitive and should be changed.

One proposal is to make it so `g` gets assigned a list, and the
`yield` happens in the enclosing scope (so the enclosing function
would have to be a generator). This was the way things worked in
Python 2, I believe.

Another proposal is to make this code a syntax error, because it's
confusing either way. (For what it's worth, that would be my
preference.)

There is related discussion about the semantics of list
comprehensions versus calling list() on a generator expression,
and of async semantics, but I don't think there's any clear point
of action there.

-- 
--Guido van Rossum (python.org/~guido

)

___
Python-Dev mailing list
Python-Dev@python.org 
https://mail.python.org/mailman/listinfo/python-dev

Unsubscribe:

https://mail.python.org/mailman/options/python-dev/jelle.zijlstra%40gmail.com





___
Python-Dev mailing list
Python-Dev@python.org 
https://mail.python.org/mailman/listinfo/python-dev

Unsubscribe:
https://mail.python.org/mailman/options/python-dev/mertz%40gnosis.cx





___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/srkunze%40mail.de


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Sven R. Kunze]

+1 from me too.


On 04.11.2017 21:55, Jim Baker wrote:
+1, as Guido correctly recalls, this language guarantee will work well 
with Jython when we get to the point of implementing 3.7+.


On Sat, Nov 4, 2017 at 12:35 PM, Guido van Rossum > wrote:


This sounds reasonable -- I think when we introduced this in 3.6
we were worried that other implementations (e.g. Jython) would
have a problem with this, but AFAIK they've reported back that
they can do this just fine. So let's just document this as a
language guarantee.

On Sat, Nov 4, 2017 at 10:30 AM, Stefan Krah > wrote:


Hello,

would it be possible to guarantee that dict literals are
ordered in v3.7?


The issue is well-known and the workarounds are tedious, example:

https://mail.python.org/pipermail/python-ideas/2015-December/037423.html




If the feature is guaranteed now, people can rely on it around
v3.9.



Stefan Krah



___
Python-Dev mailing list
Python-Dev@python.org 
https://mail.python.org/mailman/listinfo/python-dev

Unsubscribe:
https://mail.python.org/mailman/options/python-dev/guido%40python.org





-- 
--Guido van Rossum (python.org/~guido )


___
Python-Dev mailing list
Python-Dev@python.org 
https://mail.python.org/mailman/listinfo/python-dev

Unsubscribe:
https://mail.python.org/mailman/options/python-dev/jbaker%40zyasoft.com





___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/srkunze%40mail.de


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Remove typing from the stdlib

[Sven R. Kunze]

On 03.11.2017 21:34, Paul Moore wrote:

Consider someone who's downloaded Python and PyCharm (or Visual
Studio). They want to get the benefit of the IDE code completion
facilities, so they declare their argument as List[int], following the
information they've found on how to declare lists of integers.


The PyCharm I know is capable of detecting such simple types on its own, 
without type hints.



I for one like the idea of a faster evolution of typing.py.

Cheers,
Sven


PS: pip is pretty standard these days, so I don't think it's much of an 
issue for guys who really needs it installed.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 557: Data Classes

[Sven R. Kunze]

Thanks for the PEP! :)

I like the naming. ;) Though, I would like to add to Michel's argument 
in favor of a base class.



On 11.09.2017 08:38, Michel Desmoulin wrote:

- I read Guido talking about some base class as alternative to the
generator version, but don't see it in the PEP. Is it still considered ?

I'm going to put some words in explaining why I don't want to use base
classes (I don't think it buys you anything). Do you have a reason for
preferring base classes?

Not preferring, but having it as an alternative. Mainly for 2 reasons:

1 - data classes allow one to type in classes very quickly, let's
harvest the benefit from that.

Typing a decorator in a shell is much less comfortable than using
inheritance. Same thing about IDE: all current ones have snippet with
auto-switch to the class parents on tab.

All in all, if you are doing exploratory programming, and thus
disposable code, which data classes are fantastic for, inheritance will
keep you in the flow.

2 - it will help sell the data classes

I train a lot of people to Python each year. I never have to explain
classes to people with any kind of programming background. I _always_
have to explain decorators.

People are not used to it, and even kind fear it for quite some time.

Inheritance however, is familiar, and will not only push people to use
data classes more, but also will let them do less mistakes: they know
the danger of parent ordering, but not the ones of decorators ordering.


3)  - the order of base classes can arranged appropriately

In our day-to-day work, we use mixins and cooperative multiple 
inheritance a lot.

So, having dataclasses as a base class or a mixin would be great! :)

Combined with 1) and 2), I am much in favor of having dataclasses as 
base class/mixin than as a decorator.

What are the benefits of the decorator? Maybe both is possible?

Cheers,
Sven


PS: @Michel good observation 1). Typing decorators in shell is annoying.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 549: Instance Properties (aka: module properties)

[Sven R. Kunze]

Why not adding both? Properties do have their uses as does __getattr__.

Cheers,
Sven

On 13.09.2017 11:43, Larry Hastings wrote:


On 09/12/2017 12:38 AM, Larry Hastings wrote:

On 09/11/2017 07:22 PM, Guido van Rossum wrote:


The prototype is linked to from the PEP; for your convenience
here's a link:

https://github.com/larryhastings/cpython/tree/module-properties


I found that link in the PEP, but it's just a branch of a fork of 
cpython. It would be easier to review the prototype as a PR to 
upstream cpython.


Okay, I'll make a PR tomorrow and post a reply here (and update the PEP).


PR is here:

https://github.com/python/cpython/pull/3534


Cheers,


//arry/


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/srkunze%40mail.de


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 550 v4

[Sven R. Kunze]

On 26.08.2017 19:23, Yury Selivanov wrote:

ChainMap is constrained to be a Mapping-like object, but I get your
point.  Let's see what others say about the "lookup()".  It is kind of
an experiment to try a name and see if it fits.


I like "get" more. ;-)


Best,
Sven

PS:
This might be a result of still leaning towards attribute access despite 
the discussion you referenced.
I still don't think complicating and reinventing terminology (which 
basically results in API names) buys us much.


And I am still with Ethan, a context stack is just a ChainMap. Renaming 
basic methods won't hide that fact. That's my only criticism of the PEP. 
The rest is fine and useful.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 550 v4

[Sven R. Kunze]

On 26.08.2017 04:19, Ethan Furman wrote:

On 08/25/2017 03:32 PM, Yury Selivanov wrote:


A *context variable* is an object representing a value in the
execution context.  A new context variable is created by calling
the ``new_context_var()`` function.  A context variable object has
two methods:

* ``lookup()``: returns the value of the variable in the current
   execution context;

* ``set()``: sets the value of the variable in the current
   execution context.


Why "lookup" and not "get" ?  Many APIs use "get" and it's 
functionality is well understood.




Why not the same interface as thread-local storage? This has been the 
question which bothered me from the beginning of PEP550. I don't 
understand what inventing a new way of access buys us here.



Python features regular attribute access for years. It's even simpler 
than method-based access.



Best,
Sven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] API design: where to add async variants of existing stdlib APIs?

[Sven R. Kunze]

On 07.03.2017 19:37, Jelle Zijlstra wrote:



2017-03-07 10:15 GMT-08:00 Ethan Furman >:


On 03/07/2017 09:41 AM, Brett Cannon wrote:

I don't think a common practice has bubbled up yet for when
there's both synchronous and asynchronous versions of an API
(closest I have seen is appending an "a" to the async version
but that just looks like a spelling mistake to me most of
the time). This is why the question of whether separate
modules are a better idea is coming up.


I'm undoubtedly going to show my ignorance with this question, but
is it feasible to have both sync and async support in the same object?

It's possible, but it quickly gets awkward and will require a lot of 
code duplication.


Correct me if I'm wrong, but we would get the code duplication anyway.

async intrinsically does the same thing (just a little bit different) as 
its sync counterpart. Otherwise, you wouldn't use it.


For example, we could make @contextmanager work for async functions by 
making the _GeneratorContextManager class implement both enter/exit 
and aenter/aexit, but then you'd get an obscure error if you used with 
on an async contextmanager or async with on a non-async contextmanager.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] GitHub migration scheduled for Friday

[Sven R. Kunze]

On 09.02.2017 00:03, Victor Stinner wrote:

2017-02-08 23:42 GMT+01:00 Brett Cannon :

Don't forget we are doing squash merges,

Ah, I didn't know. Why not using merges?



Same question here. I see no benefit just overhead, mistakes and longer 
processes.



Sven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] re performance

[Sven R. Kunze]

On 26.01.2017 22:33, Vlastimil Brom wrote:

Hi,
I can't speak about the details of mrab's implementation, but using
regex, I get the resulting match instantly: [...]


Nice! I focused on the stdlib re module as this is mainly used by other 
frameworks (like Django).



(I personally prefer to use regex for other advantages, than this
artificial case, but it certainly doesn't hurt to have better
performance here too:)


Me, too.

So, it seems as if regex already uses a better algorithm although I 
couldn't find any reference to any regex theoretical framework like dfa, 
nfa, thompson multiple-state simulation or something.



Cheers,
Sven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] re performance

[Sven R. Kunze]

Hi folks,

I recently refreshed regular expressions theoretical basics *indulging 
in reminiscences* So, I read https://swtch.com/~rsc/regexp/regexp1.html


However, reaching the chart in the lower third of the article, I saw 
Python 2.4 measured against a naive Thompson matching implementation. 
And I was surprised about how bad it performed compared to an 
unoptimized version of an older than dirt algorithm.


So, I decided to give it a try with Python2.7 and Python3.5. Eh, voilà, 
100% cpu and no results so far. Nothing has changed at all since 2007.


>>> import re
>>> re.match(r'a?'*30 + r'a'*30, 'a'*30)
 (still waiting)

Quoting from the article: " Some might argue that this test is unfair to 
the backtracking implementations, since it focuses on an uncommon corner 
case. This argument misses the point: given a choice between an 
implementation with a predictable, consistent, fast running time on all 
inputs or one that usually runs quickly but can take years of CPU time 
(or more) on some inputs, the decision should be easy."


Victor, as the head of Python performance department, and Matthew, as 
the maintainer of the new regex module, what is your stance on this 
particular issue?


From my perspective, I can say, that regular expressions might worth 
optimizing especially for web applications (url matching usually uses 
regexes) but also for other applications where I've seen many tight 
loops using regexes as well. So, I am probing interest on this topic here.


Best,
Sven

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Generator objects and list comprehensions?

[Sven R. Kunze]

On 25.01.2017 07:28, Joe Jevnik wrote:


That was a long way to explain what the problem was. I think that that 
solution is to stop using `yield` in comprehensions because it is 
confusing, or to make `yield` in a comprehension a syntax error.




Same here; mixing comprehensions and yield (from) can't be explained 
easily. A SyntaxError would be most appropriate.


Regards,
Sven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Imports with underscores

[Sven R. Kunze]

Interesting to see that others have the same problem.

We also had this kind of "over-protective" behavior. As far as I know, 
our devs stopped doing it as it feels cumbersome.



Another argument for this is: when using PyCharm, this IDE will suggest 
imports from those modules which aren't the original ones. So, you might 
import from a third-party module. Over time, however, people learn to 
pick the "right" module to import from.


Cheers,
Sven


On 09.01.2017 12:42, Steve Holden wrote:
One of my developers recently submitted a pull request incuding a 
number of lines like


import os as _os

When I asked him why he suggested a) this would improve encapsulation, 
and b) the practice was supported in the stdlib. Further investigation 
reveals that some modules (e.g. argparse, crypt, difflib, random) do 
use this technique, but it is far from universal.


So I thought it would be useful to get input from current devs about 
the value of this practice, since to me it seems somewhat 
anti-pythonic. What advantages does it confer?


regards
Steve Holden


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/srkunze%40mail.de


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

[Sven R. Kunze]

On 13.09.2016 20:21, Tres Seaver wrote:

*Lots* of library authors have to straddle Python versions: consumers of
those libraries only get to pick and choose when their code is at the
"leaf" of the dependency tree (the application).


Maybe, I didn't express myself well but this was not my intended 
question. Using this argument for not driving the evolution of the 
language spec, doesn't seem reasonable. Changes are necessary from time 
to time and this one in particular is not breaking compatibility with 
older versions. So, existing libs are okay. But why shouldn't the 
ordering of dicts not be an advertisable feature for application 
developers or developers of future libs? My reasoning so far is that in 
those circumstances people **won't switch** from CPython 3.6 to Cython 
to PyPy back to CPython 2.7 once a week (drawn from my experience at 
least). But maybe I'm wrong here.


Cheers,
Sven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

[Sven R. Kunze]

On 13.09.2016 19:59, MRAB wrote:
The recommended way of dealing with features across different versions 
of Python is to check for them and see if they raise NameError or 
whatever, but I wonder if there would be any benefit to recording such 
things somewhere, e.g. sys.features['ordered_args'] returns True if 
arguments are passed in an ordered dict.


Just to check: do people really that often change between Python 
implementations?


My personal experience with this kind of compatibility is that it is 
rarely needed for large and complex programs. That is due to deployment 
and testing issues (at least in our environment as we run multiple 
Python services on a multitude of servers).


Best,
Sven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

[Sven R. Kunze]

On 11.09.2016 01:41, Nathaniel Smith wrote:

I feel like I'm missing something here... by this reasoning, we should
*never* change the language spec when new features are added. E.g. if
people use async/await in 3.5 then their code won't be compatible with
3.4, but async/await are still part of the language spec. And in any
case, the distinction between "CPython feature" and "Python
language-spec-guaranteed feature" is *extremely* arcane and
inside-basebally -- it seems really unlikely that most users will even
understand what this distinction means, never mind let it stop them
from writing CPython-and-PyPy-specific code. Emphasizing that this is
a new feature that only exists in 3.6+ of course makes sense, I just
don't understand why that affects the language spec bit.

(OTOH it doesn't matter that much anyway... the language spec is
definitely a useful thing, but it's largely aspirational in practice
-- other implementations target CPython compatibility more than they
target language spec compatibility.)


The new dict has thousands and one advantages: no need to import 
OrderDict anymore, standard syntax for OrderDict, etc.


People will love it. But is it legal to use? I tend to agree with you 
here and say "CPython mostly is the living spec" but I'm not 100% sure 
(I even restrain from writing a blog post about it although it so 
wonderful).


Cheers,
Sven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] A Pseudo-Post-Mortem (not dead yet) on my Multi-Core Python Project.

[Sven R. Kunze]

Thanks for the post. :) There's some typo in the title and url. :/ :D

On 07.09.2016 01:56, Eric Snow wrote:

I'm not anticipating much discussion on this, but wanted to present a
summary of my notes from the project I proposed last year and have
since tabled.

http://ericsnowcurrently.blogspot.com/2016/09/solving-mutli-core-python.html

-eric
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/srkunze%40mail.de


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Do PEP 526 type declarations define the types of variables or not?

[Sven R. Kunze]

Didn't Koos say this works more like an expression annotation?

IMO, the type of the expression is what is specified but the type of the 
variable can change over time (as you demonstrated).


Sven


PS: thinking this way, the new syntax is actually confusing as it 
annotates the variable not the expression. :-/



On 05.09.2016 17:26, Mark Shannon wrote:

Hi,

PEP 526 states that "This PEP aims at adding syntax to Python for 
annotating the types of variables" and Guido seems quite insistent 
that the declarations are for the types of variables.


However, I get the impression that most (all) of the authors and 
proponents of PEP 526 are quite keen to emphasise that the PEP in no 
way limits type checkers from doing what they want.


This is rather contradictory. The behaviour of a typechecker is 
defined by the typesystem that it implements. Whether a type 
annotation determines the type of a variable or an expression alters 
changes what typesystems are feasible. So, stating that annotations 
define the type of variables *does* limit what a typechecker can or 
cannot do.


Unless of course, others may have a different idea of what the "type 
of a variable" means.

To me, it means it means that for all assignments `var = expr`
the type of `expr` must be a subtype of the variable,
and for all uses of var, the type of the use is the same as the type 
of the variable.


In this example:

def bar()->Optional[int]: ...

def foo()->int:
x:Optional[int] = bar()
if x is None:
return -1
return x

According to PEP 526 the annotation `x:Optional[int]`
means that the *variable* `x` has the type `Optional[int]`.
So what is the type of `x` in `return x`?
If it is `Optional[int]`, then a type checker is obliged to reject 
this code. If it is `int` then what does "type of a variable" actually 
mean,

and why aren't the other uses of `x` int as well?

Cheers,
Mark.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/srkunze%40mail.de



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Sven R. Kunze]

Hi Mark,

I agree with you about postponing.

Not so much because of the issues you mentioned. Those all seem 
resolvable to me and mostly concerns type checkers, linters and coding 
styles not Python itself. However, I also don't like the rushing through 
as if this beta were the only chance to get it into Python.


Python haven't had variable annotations for decades until now and Python 
will still be there in 10 decades or so. Thus, 2 years more to wait and 
to hammer out the details does not seem much compared to the entire 
lifetime of this language.


The PEP also remains silent about when to use annotations. Compared to 
recent additions like f-strings or async, it's completely clear when to 
use it. However for variable annotations it's not (all variables? most 
used variables? least used variables?). So, I also agree with you that 
improving type checkers is the better way than adding all static type 
annotations all over the place. Python is dynamic and also types should 
be as dynamic and redundant-free as possible. Thus, some guidance would 
be great here.


Cheers,
Sven


On 02.09.2016 15:47, Mark Shannon wrote:

Hi everyone,

I think we should reject, or at least postpone PEP 526.

PEP 526 represents a major change to the language, however there are, 
I believe, a number of technical flaws with the PEP.


It is probable that with significant revisions it can be a worthwhile 
addition to the language, but that cannot happen in time for 3.6 beta 
1 (in 11 days).


PEP 526 claims to be an extension of PEP 484, but I don't think that 
is entirely correct.
PEP 484 was primarily about defining the semantics of pre-existing 
syntax. PEP 526 is about adding new syntax.
Historically the bar for adding new syntax has been set very high. I 
don't think that PEP 526, in its current form, reaches that bar.


Below is a list of the issues I have with the PEP as it stands.

In many cases it makes it more effort than type comments


Type hints should be as easy to use as possible, and that means 
pushing as much work as possible onto the checker, and not burdening 
the programmer.


Attaching type hints to variables, rather than expressions, reduces 
the potential for inference. This makes it harder for programmer, but 
easier for the checker, which is the wrong way around.


For example,, given a function:
def spam(x: Optional[List[int]])->None: ...

With type comments, this is intuitively correct and should type check:
def eggs(cond:bool):
if cond:
x = None
else:
x = [] # type: List[int]
spam(x)  # Here we can infer the type of x

With PEP 526 we loose the ability to infer types.
def eggs(cond:bool):
if cond:
x = None # Not legal due to type declaration below
else:
   x: List[int] = []
spam(x)

So we need to use a more complex type
def eggs(cond:bool):
x: Optional[List[int]]
if cond:
x = None # Now legal
else:
x: = []
spam(x)

I don't think this improves readability.
Whether this is an acceptable change is debatable, but it does need 
some debate.


It limits the use of variables
==

In Python a name (variable) is just a binding that refers to an object.
A name only exists in a meaningful sense once an object has been 
assigned to it. Any attempt to use that name, without an object bound 
to it, will result in a NameError.


PEP 526 makes variables more than just bindings, as any rebinding must 
conform to the given type. This looses us some of the dynamism for 
which we all love Python.


Quoting from the PEP:
```
a: int
a: str # Static type checker will warn about this.
```
In other words, it is illegal for a checker to split up the variable, 
even though it is straightforward to do so.


However, without the type declarations,
```
a = 1
a = "Hi"
```
is just fine. Useless, but fine.

We should be free to add extra variables, whenever we choose, for 
clarity. For example,

total = foo() - bar()
should not be treated differently from:
revenue = foo()
tax = bar()
total = revenue - tax

If types are inferred, there is no problem.
However, if they must be declared, then the use of meaningfully named 
variables is discouraged.


[A note about type-inference:
Type inference is not a universal panacea, but it can make life a lot 
easier for programmers in statically type languages.
Languages like C# use local type inference extensively and it means 
that many variables often do not need their type declared. We should 
take care not to limit the ability of checkers to infer values and 
types and make programmers' lives easier.
Within a function, type inference is near perfect, failing only 
occasionally for some generic types.
One place where type inference definitely breaks down is across calls, 
which is why PEP 484 is necessary.

]

It is premature
===

There are still plenty of issues to iron out w.r.t. PEP 484 

Re: [Python-Dev] PEP 526 ready for review: Syntax for Variable and Attribute Annotations

[Sven R. Kunze]

Thanks Guido, also to the rest of the PEP team (4 people) :)


On 30.08.2016 23:20, Guido van Rossum wrote:

I'm happy to present PEP 526 for your collective review:
https://www.python.org/dev/peps/pep-0526/ (HTML)
https://github.com/python/peps/blob/master/pep-0526.txt (source)

There's also an implementation ready:
https://github.com/ilevkivskyi/cpython/tree/pep-526

I don't want to post the full text here but I encourage feedback on
the high-order ideas, including but not limited to

- Whether (given PEP 484's relative success) it's worth adding syntax
for variable/attribute annotations.


I'd say no, especially because of the negative feedback by not a few 
thread participants.



- Whether the keyword-free syntax idea proposed here is best:
   NAME: TYPE
   TARGET: TYPE = VALUE


If it will come, it's the best because of its similarity with parameter 
annotations and IIRC there are languages that already do it like this.



Note that there's an extensive list of rejected ideas in the PEP;
please be so kind to read it before posting here:
https://www.python.org/dev/peps/pep-0526/#rejected-proposals-and-things-left-out-for-now


I find everything else well covered in the PEP especially corner-cases 
like variables without initialization, scopes etc.



Sven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 525

[Sven R. Kunze]

On 24.08.2016 21:05, Yury Selivanov wrote:
Sorry for making you irritated.  Please feel free to remind me about 
any concrete changes to the PEP that I promised to add on 
python-ideas.  I'll go and re-read that thread right now anyways.


No problem as it seems I wasn't the only one. So, it doesn't matter 
anymore. :)


Best,
Sven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 525

[Sven R. Kunze]

On 24.08.2016 21:00, Yury Selivanov wrote:


For an async generator there are two cases: either it tries to yield 
another value (the first time this happens you can throw an error 
back into it) or it tries to await -- in that case you can also throw 
an error back into it, and if the error comes out unhandled you can 
print the error (in both cases actually).


It's probably to specify all this behavior using some kind of default 
finalizer (though you don't have to implement it that way).


Hopefully there will be other discussion as well, otherwise I'll have 
to accept the PEP once this issue is cleared up. :-)


Curious to hear your thoughts on two different approaches to 
finalization.  At this point, I'm inclined to change the PEP to use 
the second approach.  I think it gives much more power to event loops, 
and basically means that any kind of APIs to control AG (or to 
finalize the loop) is possible.


I think your alternative approach is the better one. It feels more 
integrated even though it's harder for event loop implementors (which 
are rarer than normal event loop users). Also AG finalization is 
something that's not really custom to each AG but makes more sense at 
the event loop level, I think.


Best,
Sven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 525

[Sven R. Kunze]

On 24.08.2016 18:35, Guido van Rossum wrote:
On Wed, Aug 24, 2016 at 8:17 AM, Yury Selivanov 
> wrote:


On 2016-08-23 10:38 PM, Rajiv Kumar wrote:

I was playing with your implementation to gain a better
understanding of the operation of asend() and friends. Since I
was explicitly trying to "manually" advance the generators, I
wasn't using asyncio or other event loop. This meant that the
first thing I ran into with my toy code was the RuntimeError
("cannot iterate async generator without finalizer set").

As you have argued elsewhere, in practice the finalizer is
likely to be set by the event loop. Since the authors of event
loops are likely to know that they should set the finalizer,
would it perhaps be acceptable to merely issue a warning
instead of an error if the finalizer is not set? That way
there isn't an extra hoop to jump through for simple examples.

In my case, I just called
sys.set_asyncgen_finalizer(lambda g: 1)
to get around the error and continue playing :) (I realize
that's a bad thing to do but it didn't matter for the toy cases)


Yeah, maybe warning would be sufficient.  I just find it's highly
unlikely that a lot of people would use async generators without a
loop/coroutine runner, as it's a very tedious process.


Heh, I had the same reaction as Rajiv. I think the tediousness is 
actually a good argument that there's no reason to forbid this. I 
don't even think a warning is needed. People who don't use a coroutine 
runner are probably just playing around (maybe even in the REPL) and 
they shouldn't get advice unasked.


I also was irritated as Yury said there were absolutely no changes after 
python-ideas. He said he might consider a clearer warning for those 
examples at the beginning of the PEP to make them work for the reader.




Would it be possible to print a warning only when an async generator 
is being finalized and doesn't run straight to the end without 
suspending or yielding? For regular generators we have a similar 
exception (although I don't recall whether we actually warn) -- if you 
call close() and it tries to yield another value it is just GC'ed 
without giving the frame more control. For an async generator there 
are two cases: either it tries to yield another value (the first time 
this happens you can throw an error back into it) or it tries to await 
-- in that case you can also throw an error back into it, and if the 
error comes out unhandled you can print the error (in both cases 
actually).


It's probably to specify all this behavior using some kind of default 
finalizer (though you don't have to implement it that way).


Does a default finalizer solve the "event loop does not know its AGs" 
problem?


Hopefully there will be other discussion as well, otherwise I'll have 
to accept the PEP once this issue is cleared up. :-)


--
--Guido van Rossum (python.org/~guido )


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/srkunze%40mail.de


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] The devguide is now hosted on GitHub

[Sven R. Kunze]

Thanks a lot. :)


On 22.07.2016 22:04, Brett Cannon wrote:

https://github.com/python/devguide

I have also moved all issues over as well and hooked up Read The Docs 
so that there's a mirror which is always up-to-date (vs. 
docs.python.org/devguide  which is on 
a cronjob).



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/srkunze%40mail.de


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Request for CPython 3.5.3 release

[Sven R. Kunze]

On 03.07.2016 16:39, Guido van Rossum wrote:

Another thought recently occurred to me. Do releases really have to be
such big productions? A recent ACM article by Tom Limoncelli[1]
reminded me that we're doing releases the old-fashioned way --
infrequently, and with lots of manual labor. Maybe we could
(eventually) try to strive for a lighter-weight, more automated
release process?


I can only recommend such an approach. We use it internally for years 
now and the workload for releasing, quality assurance and final 
deployment dropped significantly. We basically automated everything.
The devs are pretty happy with it now and sometimes "mis-use" it for 
some of its side-products; but that's okay as it's very convenient to use.


For some parts we use pip to install/upgrade the dependencies but 
CPython might need to use a different tooling for the stdlib and its 
C-dependencies.



If you need some assistance here, let me know.



It would be less work, and it would reduce stress for
authors of stdlib modules and packages -- there's always the next
release. I would think this wouldn't obviate the need for carefully
planned and timed "big deal" feature releases, but it could make the
bug fix releases *less* of a deal, for everyone.

[1] 
http://cacm.acm.org/magazines/2016/7/204027-the-small-batches-principle/abstract
(sadly requires login)



Best,
Sven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Our responsibilities (was Re: BDFL ruling request: should we block forever waiting for high-quality random bits?)

[Sven R. Kunze]

I also think it’s a great module for providing defaults that we can’t
provide in os.urandom, like the number of bytes that are considered
“secure” [1].

What I don’t think is that the secrets module means that all of a sudden
os.urandom is no longer an API that is primarily used in a security
sensitive context


Not all of a sudden. However, I guess things will change in the future.

If we want the secrets module to be the first and only place where 
crypto goes, we should work towards that goal. It needs proper 
communication, marketing etc.


Deprecation periods can be years long. This change (whatever form it 
will take) can be carried out over 3 or 4 releases when the ultimate 
goal is made clear to everybody reading the docs. OTOH I don't know 
whether long deprecation periods are necessary here at all. Other 
industries are very sensitive to fast changes.


Furthermore, next generations will be taught using the new way, so the 
Python community should not be afraid of some changes because most of 
them are for the better.



On 16.06.2016 15:02, Donald Stufft wrote:

I think that os.urandom is the most obvious thing that someone will reach for 
given:

* Pages upon pages of documentation both inside the Python community
   and outside saying “use urandom”.
* The sheer bulk of existing code that is already out there using
   os.urandom for it’s cryptographic properties.


That's maybe you. However, as stated before, I am not expert in this 
field. So, when I need to, I first would start researching the current 
state of the art in Python.


If the docs says: use the secrets module (e.g. near os.urandom), I would 
happily comply -- especially when there's reasonable explanation.



That's from a newbie's point of view.


Best,
Sven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Sven R. Kunze]

On 10.06.2016 21:17, Donald Stufft wrote:


On Jun 10, 2016, at 3:05 PM, David Mertz > wrote:


OK.  My understanding is that Guido ruled out introducing an 
os.getrandom() API in 3.5.2.  But would you be happy if that 
interface is added to 3.6?


It feels to me like the correct spelling in 3.6 should probably be 
secrets.getrandom() or something related to that.




I am not a security expert but your reply makes it clear to me. So, for 
me this makes:


os -> os-dependent and because of this varies from os to os (also 
quality-wise)

random -> pseudo-random, but it works for most non-critical use-cases
secret -> that's for crypto


If don't need crypto, secret would be a waste of resources, but if you 
need crypto, then os and random are unsafe. I think that's simple 
enough. At least, I would understand it.



Just my 2 cents: if I need crypto, I would pay the price of blocking 
rather then to get an exception (what are my alternatives? I need those 
bits! ) or get unsecure bits.



Sven


Well we have 
https://docs.python.org/dev/library/secrets.html#secrets.token_bytes so adding 
a getrandom() function to secrets would largely be the same as that 
function.


The problem of course is that the secrets library in 3.6 uses 
os.urandom under the covers, so it’s security rests on the security of 
os.urandom. To ensure that the secrets library is actually safe even 
in early boot it’ll need to stop using os.urandom on Linux and use the 
getrandom() function.


That same library exposes random.SystemRandom as secrets.SystemRandom 
[1], and of course SystemRandom uses os.urandom too. So if we want 
people to treat secrets.SystemRandom as “always secure” then it would 
need to stop using os.urandom and start using the get random() 
function on Linux as well.



[1] This is actually documented as "using the highest-quality sources 
provided by the operating system” in the secrets documentation, and 
I’d argue that it is not using the highest-quality source if it’s 
reading from /dev/urandom or getrandom(GRD_NONBLOCK) on Linux systems 
where getrandom() is available. Of course, it’s just an alias for 
random.SystemRandom, and that is documented as using os.urandom.


—
Donald Stufft





___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/srkunze%40mail.de


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Adding Type[C] to PEP 484

[Sven R. Kunze]

On 15.05.2016 19:30, Guido van Rossum wrote:
Right. I still have no good intuition for what Type[BasicUser, 
ProUser] would mean so I think you should be required to use the 
Union, which is clear. Type[] should only allow one parameter.


Type[A, B] reminds me of isinstance(obj, (A, B)).


Sven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] file system path protocol PEP

[Sven R. Kunze]

On 13.05.2016 18:43, Chris Angelico wrote:

https://docs.python.org/3/c-api/unicode.html#c.PyUnicode_Check

https://docs.python.org/3/c-api/unicode.html#c.PyUnicode_CheckExact


Thanks for pointing me at this.
I searched via github and found usages only: 
https://github.com/python/cpython/search?utf8=%E2%9C%93=PyUnicode_Check



"Check" accepts subclasses; "CheckExact" doesn't (it's like "type(x)
is str"). The question is, which one SHOULD be being done? What should
this do:

class TmpPath(str):
 def __fspath__(self):
 return "/tmp/"+self
x = TmpPath("foo/bar")
open(x, "w")

Does that respect __fspath__, or respect the fact that it's a string?


Fine example. Thinking naively, I would say, when somebody made an 
effort to write __fspath__, it should be respected. Maybe, that's just me.


Not sure if that can be a source of errors, when somebody adds str as 
baseclass later because the team needs str functionality. At least I 
would expect __fspath__ to still work .


Best,
Sven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] file system path protocol PEP

[Sven R. Kunze]

On 13.05.2016 17:29, Brett Cannon wrote:
Purposeful change. It was what I had in my head after I posted my 
"groups" breakdown email and what Guido suggested as well 
independently. This helps alleviate any perf worries as type checks in 
C are pointer checks that are cheap to make compared to attribute 
lookup. And even if we do get the community to move on to path objects 
the check is still cheap enough to not worry. And finally, this is 
simply how we have almost always coded this kind of special-casing at 
the C level throughout Python and the stdlib, so it now is more inline 
with standard coding practices than the original proposal in the PEP.


Maybe, we have a misunderstanding here. Does "PyUnicode_Check(path)" 
respect subclass hierarchies such as isinstance(path, str) does? Could 
not find the definition of it in the source.


You said it's just a pointer comparison. To me that implies it could not 
respect subclasses as isinstance does. If that is the case, the C 
implementation differs from the Python version in semantics. I'm sorry 
if I misunderstood the whole implementation here.



Best,
Sven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] file system path protocol PEP

[Sven R. Kunze]

On 13.05.2016 11:48, Koos Zevenhoven wrote:

This issue is coupled with the future optimization questions.


AFAIC coupling API design to optimization is called premature optimization.


However, the proposed semantics will change if the checks are swapped. So,
my actual question is:

Is that an intended API inconsistency or a known bug supposed to be resolved
later?


Taking into account the description (and the drafted type hint), which
the documentation will probably reflect, the semantic effects of that
are very minor or nonexistent.


From your perspective. As far as I remember, one goal of this proposal 
was to avoid wallet gardens. During the lengthy discussion on 
python-ideas people brought up that some third-party libs indeed 
subclass from str. They are currently locked out.



I do think the documentation of the protocol should say that str or
bytes subclasses should not implement __fspath__.


Indeed. Just one minor note here: str or bytes subclasses *can* 
implement __fspath__ and currently it will be *ignored*. Maybe that 
changes in the future. So, that's the reason it should not be implemented.



So no API inconsistency there.


API consistency is not defined by docs-matching-implementation but by 
implementation-matching-expectations.



Best,
Sven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] file system path protocol PEP

[Sven R. Kunze]

On 13.05.2016 10:36, Koos Zevenhoven wrote:

This has just been discussed very recently in this thread (and earlier
too).


Could you point me to that? It seems I missed that part. I only found 
posts related to performance degradation.


However, the proposed semantics will change if the checks are swapped. 
So, my actual question is:


Is that an intended API inconsistency or a known bug supposed to be 
resolved later?



It may make sense, but it's not among our current worries.


It might not be yours but mine. ;) That's why I was asking.


Besides, we already added the new fspath semantics to the PEP.

While I hope Brett is asleep in his time zone, I'm guessing he will
agree (just saying this because you write "@Brett").

-- Koos


On Fri, May 13, 2016 at 10:58 AM, Sven R. Kunze <srku...@mail.de> wrote:

On 12.05.2016 18:24, Guido van Rossum wrote:

def fspath(p: Union[str, bytes, PathLike]) -> Union[str, bytes]:
 if isinstance(p, (str, bytes)):
 return p
 try:
 return p.__fspath__
 except AttributeError:
 raise TypeError(...)


@Brett
Would you think it makes sense to swap the str/bytes check and the
__fspath__ check?


I just thought of a class subclassing str/bytes and defines __fspath__. Its
__fspath__ method would be ignored currently.


Best,
Sven

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] file system path protocol PEP

[Sven R. Kunze]

On 12.05.2016 18:24, Guido van Rossum wrote:

def fspath(p: Union[str, bytes, PathLike]) -> Union[str, bytes]:
if isinstance(p, (str, bytes)):
return p
try:
return p.__fspath__
except AttributeError:
raise TypeError(...)


@Brett
Would you think it makes sense to swap the str/bytes check and the 
__fspath__ check?



I just thought of a class subclassing str/bytes and defines __fspath__. 
Its __fspath__ method would be ignored currently.



Best,
Sven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] file system path protocol PEP

[Sven R. Kunze]

On 12.05.2016 19:27, Ethan Furman wrote:

Maybe, but a bad idea for two reasons:

1) Reducing a str to the exact same str is silly; and, more importantly


Finding something silly is no technical argument.

Best,
Sven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] file system path protocol PEP

[Sven R. Kunze]

On 12.05.2016 18:56, Ethan Furman wrote:

On 05/12/2016 09:26 AM, Sven R. Kunze wrote:

str and bytes will receive the __fspath__ attribute when this PEP is
accepted?


No, they won't.  The __fspath__ protocol will reduce the rich path 
object down to a str/bytes object.


Would this make the implementation of os.fspath simpler?

After all, str and bytes are to some extend path-like objects.


Best,
Sven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] file system path protocol PEP

[Sven R. Kunze]

On 12.05.2016 17:42, Ethan Furman wrote:

On 05/12/2016 01:31 AM, Sven R. Kunze wrote:


I think the "Rationale" section ignores the fact the Path also supports
the .path attribute now. Which indeed defines a common interface between
path objects.


The version of Python that has Path.path has not been released yet.  
And even so, .path is not a "common interface" as neither str nor 
bytes have it, and they also are used as path objects.


str and bytes will receive the __fspath__ attribute when this PEP is 
accepted?


And even given all that, for smoother interoperability with the rest 
of the stdlib, or at least the os.* portion, those functions would 
still need to be upgraded to check for .path on the incoming arguments 
-- at which point we may as well make a protocol to properly support 
file system paths instead of relying on the rather generic attribute 
name of 'path'.


Just so, if you accept changing os.* as a necessary solution.

If not, keeping .path would suffice and would be much simpler.


Best,
Sven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] file system path protocol PEP

[Sven R. Kunze]

On 11.05.2016 23:57, Brett Cannon wrote:
On Wed, 11 May 2016 at 14:29 Nikolaus Rath > wrote:


On May 11 2016, Brett Cannon > wrote:
> This PEP proposes a protocol for classes which represent a file
system
> path to be able to provide a ``str`` or ``bytes`` representation.
[...]

As I said before, to me this seems like a lot of effort for a very
specific use-case.



Exactly. Especially when considering what else can be done to improve 
the situation considerably.



So let me put forward two hypothetical scenarios to
better understand your position:

- A new module for URL handling is added to the standard library (or
  urllib is suitably extended). There is a proposal to add a new
  protocol that allows classes to provide a ``str`` or ``bytes``
  representation of URLs.

- A new (third-party) library for natural language processing arises
  that exposes a specific class for representing audio data. Existing
  language processing code just uses bytes objects. To ease transition
  and interoperability, it is proposed to add a new protocol for
classes
  that represend audio data to provide a bytes representation.



You can even add the timedelta-to-seconds protocol that somebody thought 
would be good idea:


https://mail.python.org/pipermail/python-dev/2016-April/144018.html
https://mail.python.org/pipermail/python-ideas/2016-May/040226.html

The generalization is straight-forward and a result of this discussion. 
If it works and is a good idea for pathlib, then there's absolutely no 
reason not to do this for the datetime lib and other rich-object libs. 
Same goes the other way round. Question still is: is it a good idea?


Maybe, it will become a successful pattern. Maybe not.


Do you think you would you be in favor of adding these protocols to
the stdlib/languange reference as well?


Maybe for URLs, not for audio data (at least not in the stdlib; 
community can do what they want).


If not, what's the crucial
difference to file system paths?


Nearly everyone uses file system paths on a regular basis, less so 
than URLs but still a good amount of people. Very few people work with 
audio data.


Amount of usage should be taken into account of course. However, 
question remains if that suffices as a justification for the effort.



Best,
Sven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] file system path protocol PEP

[Sven R. Kunze]

Thanks Brett for your hard work. My comments below:

On 11.05.2016 18:43, Brett Cannon wrote:

Rationale
=

Historically in Python, file system paths have been represented as
strings or bytes. This choice of representation has stemmed from C's
own decision to represent file system paths as
``const char *`` [#libc-open]_. While that is a totally serviceable
format to use for file system paths, it's not necessarily optimal. At
issue is the fact that while all file system paths can be represented
as strings or bytes, not all strings or bytes represent a file system
path.


I can remember this argument being made during the discussion. I am not 
sure if that 100% correct as soon as we talk about PurePaths.



This can lead to issues where any e.g. string duck-types to a
file system path whether it actually represents a path or not.

To help elevate the representation of file system paths from their
representation as strings and bytes to a more appropriate object
representation, the pathlib module [#pathlib]_ was provisionally
introduced in Python 3.4 through PEP 428. While considered by some as
an improvement over strings and bytes for file system paths, it has
suffered from a lack of adoption. Typically the key issue listed
for the low adoption rate has been the lack of support in the standard
library. This lack of support required users of pathlib to manually
convert path objects to strings by calling ``str(path)`` which many
found error-prone.

One issue in converting path objects to strings comes from
the fact that only generic way to get a string representation of the
path was to pass the object to ``str()``. This can pose a
problem when done blindly as nearly all Python objects have some
string representation whether they are a path or not, e.g.
``str(None)`` will give a result that
``builtins.open()`` [#builtins-open]_ will happily use to create a new
file.

Exacerbating this whole situation is the
``DirEntry`` object [#os-direntry]_. While path objects have a
representation that can be extracted using ``str()``, ``DirEntry``
objects expose a ``path`` attribute instead. Having no common
interface between path objects, ``DirEntry``, and any other
third-party path library had become an issue. A solution that allowed
any path-representing object to declare that is was a path and a way
to extract a low-level representation that all path objects could
support was desired.


I think the "Rationale" section ignores the fact the Path also supports 
the .path attribute now. Which indeed defines a common interface between 
path objects.




[...]

Proposal


This proposal is split into two parts. One part is the proposal of a
protocol for objects to declare and provide support for exposing a
file system path representation.


https://docs.python.org/3/whatsnew/changelog.html says:

"Add ‘path’ attribute to pathlib.Path objects, returning the same as 
str(), to make it more similar to DirEntry. Library code can now write 
getattr(p, ‘path’, p) to get the path as a string from a Path, a 
DirEntry, or a plain string. This is essentially a small one-off protocol."


So, in order to promote the "small one-off protocol" to a more broader 
protocol, this PEP proposes a simple rename of .path to .__fspath__, is 
that correct?



The only issue I see with it is that it requires another function 
(os.fspath) to extract the "low-level representation". .path seems far 
easier to me.



The other part is changes to Python's
standard library to support the new protocol.


I think this could be another PEP unrelated to the first part.

These changes will also have the pathlib module drop its provisional 
status.


Not sure if that should be part of the PEP, maybe yes.


[...]


The remainder of the PEP unfolds as a flawless implication of the 
rationale and the proposed idea.


Unfortunately, I don't have anything to contribute to the open issues. 
All solutions have their pros and cons and everything that could be said 
has been said. I think you need to decide.


Sven

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] file system path protocol PEP

[Sven R. Kunze]

On 12.05.2016 00:13, Brett Cannon wrote:
I see this whole discussion breaking down into a few groups which 
changes what gets done upfront and what might be done farther down the 
line:


 1. Maximum acceptance: do whatever we can to make all representation
of paths just work, which means making all places working with a
path in the stdlib accept path objects, str, and bytes.
 2. Safely use path objects: __fspath__() is there to signal an object
is a file system path and to get back a lower-level representation
so people stop calling str() on everything, providing some
interface signaling that someone doesn't misuse an object as a
path and only changing path consumptions APIs -- e.g. open() --
and not path manipulation APIs -- e.g. os.path -- in the stdlib.
 3. It ain't worth it: those that would rather just skip all of this
and drop pathlib from the stdlib.



Sorry for being picky here. I think the last group needs to be split up:

3. It ain't worth it: those that would rather just skip all of this
4. drop pathlib from the stdlib.

I put myself into camp 3, mostly because I don't consider the "wallet 
garden problem" a problem at all and I realized that our past issues 
with pathlib resulted from missing features in pathlib not in the rest 
of the stdlib.



Sven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Maybe, just maybe, pathlib doesn't belong.

[Sven R. Kunze]

On 12.04.2016 18:04, Chris Barker wrote:
On Tue, Apr 12, 2016 at 7:54 AM, Sven R. Kunze <srku...@mail.de 
<mailto:srku...@mail.de>> wrote:



My conclusion is that these changes are not optional and tweaking
os, io and shutil is just yet another workaround for a clean
solution. :)


Is the clean solution to re-implement EVERYTHING in the stdlib that 
involves a path in a new, fancy pathlib way?


If we were starting from scratch, I _might_ like that idea, but we're 
not starting from scratch. And that would cement in pathlib itself, 
leaving no room for other path implementations. kind of like how the 
pre-__Index__ python cemented in python integers as the only objects 
once could use to index a sequence.


I cannot remember us using another datetime library. So, I don't value 
this "advantage" as much as you do.



Best,
Sven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Maybe, just maybe, pathlib doesn't belong.

[Sven R. Kunze]

On 12.04.2016 16:59, Random832 wrote:


Strings are strings. Paths are paths. That's were the difference is. 

Yes but why aren't these both "things that you may want to use to open a
file"?


Because "things that you may want to use to open a file" is a bit vague 
and thus conceal the fact that we really need.


As an example: time.sleep takes a number of seconds (notice the 
primitive datatype just like a string) and does not take timedelta.


Why don't we add datetime.timedelta support to time.sleep? Very same thing.


The fact that there is one obvious thing to want to do
with open and a Path strongly suggests that that should be able to be
done by passing the Path to open.

Path(...).open() is your friend then. I don't see why you need os.open.

Because I'm passing it to modfoo.dosomethingwithafile() which takes a
filename and passes it to shutils, which passes it to builtin open,
which passes it to os.open.

Should Path grow a dosomethingwithmodfoo method?


Because we can argue here the other way round and say:

"oh, pathlib can do things, I cannot do with os.path."

Should os.path grow those things?


Put differently, you cannot do everything. But the most common issues 
should be resolved in the correct module. This is no argument for or 
against either solution.



I am sorry, if my contribution on the threads of python-ideas made it 
seem that I would always support this idea. I don't anymore. However, I 
will still be happy with the outcome even if not perfect, will help 
making the Python stdlib better. :)


Best,
Sven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] pathlib - current status of discussions

[Sven R. Kunze]

Sorry for disturbing this thread's harmony.


On 12.04.2016 08:00, Ethan Furman wrote:

On 04/11/2016 10:14 PM, Chris Barker - NOAA Federal wrote:


Consider os.path.join:


Why in the world do the  os.path functions need to work with Path
objects? ( and other conforming objects)


Because library XYZ that takes a path and wants to open it shouldn't 
have to care whether that path is a string or pathlib.Path -- but if 
os.open can't use pathlib.Path then the library has to care (or the 
user has to care).



This all started with the goal of using Path objects in the stdlib,
but that's for opening files, etc.


Etc. as in os.join?  os.stat? os.path.split?


Path is an alternative to os.path -- you don't need to use both.




I agree with that quote of Chris.

As a user you don't, no.  As a library that has no control over what 
kind of "path" is passed to you -- well, if os and os.path can accept 
Path objects then you can just use os and os.path; otherwise you have 
to use os and os.path if passed a str or bytes, and pathlib.Path if 
passed a pathlib.Path -- so you do have to use both.


I don't agree here. There's no need to increase the convenience for a 
library maintainer when it comes to implicit conversions.


When people want to use your library and it requires a string, the can 
simply use "my_path.path" and everything still works for them when they 
switch to pathlib.



Best,
Sven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Maybe, just maybe, pathlib doesn't belong.

[Sven R. Kunze]

On 12.04.2016 12:41, Paul Moore wrote:

As your thoughts appear to have been triggered by my comments, I feel
I should clarify.

1. I like pathlib even as it is right now, and I'm strongly -1 on removing it.
2. The "external dependency" aspect of 3rd party solutions makes them
far less useful to me.
3. The work on improving integration with the stdlib (which is nearly
sorted now, as far as I can see) is a big improvement, and I'm all in
favour. But even without it, I wouldn't want pathlib to be removed.
4. There are further improvements that could be made to pathlib,
certainly, but again they are optional, and pathlib is fine without
them.


My conclusion is that these changes are not optional and tweaking os, io 
and shutil is just yet another workaround for a clean solution. :)


Just my two cents.


5. I wish more 3rd party code integrated better with pathlib. The
improved integration work might help with this. But ultimately, Python
2 compatibility is likely to be the biggest block (either perceived or
real - we can make pathlib support as simple as possible, but some 3rd
party authors will remain unwilling to add support for Python 3 only
features in the short term). This isn't a pathlib problem.
6. There will probably always be a place for low-level os/os.path
code. Adding support in those modules for pathlib doesn't affect that
fact, but does make it easier to use pathlib "seamlessly", so why not
do so?

tl; dr; I'm 100% in favour of pathlib, and in the direction the
current discussion (excluding "let's give up on pathlib" digressions)
is going.


Best,
Sven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Maybe, just maybe, pathlib doesn't belong.

[Sven R. Kunze]

On 12.04.2016 00:56, Random832 wrote:

Fully general re-dispatch from argument types on any call to a function
that raises TypeError or NotImplemented? [e.g. call
Path.__missing_func__(os.open, path, mode)]

Have pathlib monkey-patch things at import?


Implicit conversion. No, thanks.


On Mon, Apr 11, 2016, at 17:43, Sven R. Kunze wrote:

So, I might add:

3. add more high-level features to pathlib to prevent a downgrade to os
or os.path

3. reimplement the entire ecosystem in every walled garden so no-one has
to leave their walled gardens.

What's the point of batteries being included if you can't wire them to
anything?


Huh? That makes not sense to me.


I don't get what you mean by this whole "different level of abstraction"
thing, anyway.


Strings are strings. Paths are paths. That's were the difference is.


The fact that there is one obvious thing to want to do
with open and a Path strongly suggests that that should be able to be
done by passing the Path to open.


Path(...).open() is your friend then. I don't see why you need os.open.

Refusing to upgrade it like saying, everything was better in the old 
days. So let's use os.open instead of Path(...).open().



Also, what level of abstraction is builtin open? Maybe we should _just_
leave os alone on the grounds of some holy sacred lowest-level-itude,
but allow io and shutils to accept Path?


os, io and shutils accept strings. Not Path objects. Why? Because the 
semantics of "being a path" are applied implicitly by those modules. You 
are free to use a random string as a path and later as the name of your 
pet. Semantics of a string comes from usage. Path objects however have 
built-in semantics.


Furthermore, if os, io and shutils are changed, we allow code like the 
following:



my_path.touch()
os.remove(my_path)


I don't know how to explain reasonably why my_path sometimes stays in 
front of the method call and sometimes behind it to newbies.


Best,
Sven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Maybe, just maybe, pathlib doesn't belong.

[Sven R. Kunze]

On 11.04.2016 23:15, Ethan Furman wrote:

We've pretty decided that we have two options:

1. remove pathlib
2. make the stdlib work with pathlib

So we're trying to make option 2 work before falling back to option 1.

If you have a way to make pathlib work with the stdlib that doesn't 
involve "fixing" os and os.path, now is the time to speak up.


As I said, I don't like messing with os or os.path. They are built with 
a different level of abstraction in mind.



What makes people want to go down from pathlib to os (speaking in terms 
of abstraction) is the fact that pathlib suggests/promise a convenience 
that it cannot hold. You might have seen my "feedback" post here on 
python-dev. If those points were corrected in a reasonable way, we 
wouldn't have had the need to go down to os or other stdlib modules. As 
it presents itself, it feels like a poor wrapper for os and os.path. I 
hope that makes sense.


So, I might add:

3. add more high-level features to pathlib to prevent a downgrade to os 
or os.path



Best,
Sven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Maybe, just maybe, pathlib doesn't belong.

[Sven R. Kunze]

On 11.04.2016 23:05, Random832 wrote:

On Mon, Apr 11, 2016, at 16:48, Sven R. Kunze wrote:

On 11.04.2016 22:33, Alexander Walters wrote:

If there is headway being made, I do not see it.

Funny that you brought it up. I was about posting something myself. I
cannot agree completely. But starting with a comment from Paul, I
realized that pathlib is something different than a string. After doing
the research and our issues with pathlib, I found:


- pathlib just needs to be improved (see my 5 points)
- os[.path] should not tinkered with

I'm not so sure. Is there any particular reason os.path.join should
require its arguments to be homogenous, rather than allowing
os.path.join('a', b'b', Path('c')) to return 'a/b/c'?


Besides the fact, that I don't like mixing types (this was something 
that worried me about the discussion from the beginning), you can 
achieve the same using pathlib alone.


There's no need of it let alone the maintenance and slowdown of these 
implicit conversions.



I know that all of those discussions of a new protocol (path->str,
__fspath__ etc. etc.) might be rendered worthless by these two
statements. But that's my conclusion.

"os" and "os.path" are just lower level. "pathlib" is a high-level,
convenience library. When using it, I don't want to use "os" or
"os.path" anymore. If I still do, "pathlib" needs improving. *Not "os"
nor "os.path"*.

The problem isn't you using os. It's you using other modules that use
os. or io, shutil, or builtins.open. Or pathlib, if what *you're* using
is some other path library. Are you content living in a walled garden
where there is only your code and pathlib, and you never might want to
pass a Path to some function someone else (who didn't use pathlib)
wrote?

os is being used as an example because fixing os probably gets you most
other things (that just pass it through to builtins.open which passes it
through to os.open) for free.


Hypothetical assumptions meeting implicit type conversions. You might 
prefer those, I don't because of good reason. I was one of those 
starting the discussion around pathlib improvements. I understand now, 
that this is one of its minor issues. And btw. using some "other 
pathlib" is no argument for or against improving "THE pathlib".


The .path attribute will do it from what I can see.


Best,
Sven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Maybe, just maybe, pathlib doesn't belong.

[Sven R. Kunze]

On 11.04.2016 23:08, Random832 wrote:

On Mon, Apr 11, 2016, at 17:04, Sven R. Kunze wrote:

PS: The only way out that I can imagine is to fix pathlib. I am not in
favor of fixing functions of "os" and "os.path" to except "path"
objects;

Why not?


It occurred to me after pondering over Paul's comments.

"os" and "os.path" is just a completely different level of abstraction. 
There is just no need to mess with them.


The initial failure of my colleague and me of using pathlib can be 
solely attributed to pathlib's lack of functionality. Not to the 
incompatibility of "os" nor "os.path" with "Path" objects.



Best,
Sven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Maybe, just maybe, pathlib doesn't belong.

[Sven R. Kunze]

On 11.04.2016 22:55, Alexander Walters wrote:
Every conceivable way to fix pathlib have already been argued. Are any 
of them worth doing?  Can we get consensus enough to implement one of 
them?  If not, we should consider either dropping the matter or 
dropping the module.


Right now, I don't see pathlib removed. Why? Because using strings alone 
has its caveats (we all know that). So, I cannot imagine an alternative 
concept to pathlib right now. We might call it differently, but the 
concept stays unchanged.


MAYBE, if there's an alternative concept, I could be convinced to 
support dropping the module.


Best,
Sven

PS: The only way out that I can imagine is to fix pathlib. I am not in 
favor of fixing functions of "os" and "os.path" to except "path" 
objects; which does the majority here discuss now with the new 
__fspath__ protocol. But shaping what we have is definitely worth it.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Maybe, just maybe, pathlib doesn't belong.

[Sven R. Kunze]

On 11.04.2016 22:33, Alexander Walters wrote:

If there is headway being made, I do not see it.


Funny that you brought it up. I was about posting something myself. I 
cannot agree completely. But starting with a comment from Paul, I 
realized that pathlib is something different than a string. After doing 
the research and our issues with pathlib, I found:



- pathlib just needs to be improved (see my 5 points)
- os[.path] should not tinkered with


I know that all of those discussions of a new protocol (path->str, 
__fspath__ etc. etc.) might be rendered worthless by these two 
statements. But that's my conclusion.


"os" and "os.path" are just lower level. "pathlib" is a high-level, 
convenience library. When using it, I don't want to use "os" or 
"os.path" anymore. If I still do, "pathlib" needs improving. *Not "os" 
nor "os.path"*.



Best,
Sven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] pathlib+os/shutil feedback

[Sven R. Kunze]

On 10.04.2016 16:51, Paul Moore wrote:

On 10 April 2016 at 15:07, Sven R. Kunze <srku...@mail.de> wrote:

If there's some agreement to change things with respect to those 5 points, I
am willing to put some time into it.

In broad terms I agree with these points. Thanks for doing the
research. It would certainly be good to try to improve pathlib based
on this sort of feedback while it is still provisional.


I'd appreciate some guidance on this. Just let me know what I can do 
since I don't know the processes of hacking CPython.



"""
Path.rglob(pattern)
Walk down a given path; a wrapper for "os.scandir"/"os.listdir".
"""

However, at least in 3.5, Path.rglob does *not* wrap scandir. There's
a difference in principle, in that scandir (DirEntry) objects cache
stat data, where pathlib does not. Whether that makes using scandir in
Path.rglob impossible, I don't know. Ideally I'd like to see pathlib
modified to use scandir (because otherwise there will always be people
saying "use os.walk rather than scandir, as it's faster) - or if it's
not possible to do so because of the difference in principle, then I'd
like to see a clear discussion of the issue in the docs, including the
recommended approach for people who want scandir performance *without*
having to abandon pathlib for lower level functions.


Good point. The proposed docstring was just to illustrate the 
functionality to the uninformed reader. People mostly trust the docs 
without digging deeper but they should be accurate of course.



Best,
Sven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] pathlib+os/shutil feedback

[Sven R. Kunze]

I talked to my colleague. He didn't remember the concrete use-case, 
though, he instantly mentioned three possible things (no order of 
preference):


1) pathlib + mtime
2) os.walk and pathlib
3) creation/removal of paths

He wasn't too sure but I checked with the docs and his memories seemed 
to be correct:



-

1) https://docs.python.org/3/library/pathlib.html#pathlib.Path.stat

High-level path objects should return high-level [insert type here] 
objects. Put differently, an API for retrieving time-stats as real 
date/time objects would be nice. I think that can be expanded to other 
pathlib methods as well, to make them less "os-wrapper"-like and provide 
added value.



-

2) I remember a discussion on python-ideas about using "glob" or 
"rglob". However, when searching the docs for "walk" like in "os.walk" 
or for "iter", I don't find "glob"/"rglob". I can imagine ourselves 
(pathlib newbies back then), we didn't discover them.


It would be great if the docs could be improved like the following:

"""
Path.rglob(pattern)
Walk down a given path; a wrapper for "os.scandir"/"os.listdir". This is 
like calling glob() with “**” added in front of the given pattern:

"""

I think it would make "glob" and "rglob" more discoverable to new users.

NOTE: """ Using the “**” pattern in large directory trees may consume an 
inordinate amount of time.""" sounds not really encouraging. This is 
especially true for  "rglob" as it is defined as "like calling glob() 
with “**”".


That leads to wondering whether "rglob" performs slow globbing instead 
of a "os.scandir"/"os.listdir".


https://docs.python.org/3/library/pathlib.html#basic-use even promotes 
"glob" with "**" in the beginning which seems rather discouraging to use 
"rglob" as a fast alternative to "os.walk/scandir/listdir". Renaming 
"rglob"/adding a "scan" method would definitely help here.



-

3) Again searching the docs for "create", "delete" (nothing found) and 
"remove", I found "Path.touch", "Path.rmdir" and "Path.unlink".


It would be great if we had an easy way to remove a complete subtree as 
with "shutil.rmtree". We mostly don't care if a directory is empty. We 
need the system to be in a state of "this path does not exist anymore".


Moreover, touching a file is good enough to "create" it if you don't 
care about changing its mtime. It you care about its mtime, it's a 
problem to "touch".


--


That's it for our issues with pathlib from the past. Additionally, I got 
two further observations:


A) pathlib tries to mimic/publish some low-level APIs to its users. 
"unlink" is not something people would expect to use when they want to 
"delete" or to "remove" a file or a directory. I know where the term 
stems from but it's the wrong layer of abstraction IMHO. Same for 
"touch" or "chmod".


B) "rename" vs "replace". The difference is not really clear from the 
docs. You need to read "Path.replace" in order to understand 
"Path.rename" completely. (one raises an exception, the other don't if I 
read it correctly).



If there's some agreement to change things with respect to those 5 
points, I am willing to put some time into it.



Best,
Sven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] When should pathlib stop being provisional?

[Sven R. Kunze]

Yeah, sure. But it was more like this on a single line:

   os.missing1(str(our_path.something1)) *** 
os.missing2(str(our_path.something1)) *** 
os.missing1(str(our_path.something1))


And then it started to get messy because you need to work on a single 
long line or you need to open more than one line.


It was a simple thing actually. Like repeating the same calls to pathlib 
just because we need to switch to os.path I will ask my colleague if 
he remembers or if we can recover the code tommorrow...



Best,
Sven


NOTE to myself: getting old, need to write down everything


On 06.04.2016 23:03, Ethan Furman wrote:

On 04/06/2016 01:47 PM, Sven R. Kunze wrote:


I still cannot remember what the concrete issue was why we dropped
pathlib the same day we gave it a try. It was something really stupid
and although I hoped to reduce the size of the code, it was less
readable. But it was not the path->str issue but something more mundane.
It was something that forced us to use os[.path] as Path didn't provide
something equivalent. Cannot remember.


I'm willing to guess that if you had been able to just call

  os.whatever(your_path_obj)

it would have been at most a minor annoyance.

--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/srkunze%40mail.de


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Defining a path protocol

[Sven R. Kunze]

On 06.04.2016 22:55, Brett Cannon wrote:
On Wed, 6 Apr 2016 at 13:54 Sven R. Kunze <srku...@mail.de 
<mailto:srku...@mail.de>> wrote:


Furthermore, we MIGHT later want some URI support, so I don't know
off the top of my head if there's a difference between __fspath__
and __urlpath__ but better separate it now. Later we can re-merge
then if necessary.


There's a difference as a URL represents something different than a 
file system path (URI doesn't necessarily). Plus the serialized format 
would be different, etc.


Sure. URLs and URIs are more than just paths. I would expect __urlpath__ 
to be different than __url__ itself but if that's is a different discussion.


So, __fspath__ for me. :)

Best,
Sven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Defining a path protocol

[Sven R. Kunze]

On 06.04.2016 22:28, Brett Cannon wrote:
On Wed, 6 Apr 2016 at 13:20 Sven R. Kunze <srku...@mail.de 
<mailto:srku...@mail.de>> wrote:



What about

__file_path__


Can be a directory as well (and you could argue semantics of file 
system inodes, beginners won't know the subtlety and/or wonder where 
__dir_path__ is).


Good point.

Well, then __fspath__ for me.


I knew instantly what it means especially considering btrfs, ntfs, xfs, 
zfs, etc.


Furthermore, we MIGHT later want some URI support, so I don't know off 
the top of my head if there's a difference between __fspath__ and 
__urlpath__ but better separate it now. Later we can re-merge then if 
necessary.



Best,
Sven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] When should pathlib stop being provisional?

[Sven R. Kunze]

On 06.04.2016 07:00, Guido van Rossum wrote:

On Tue, Apr 5, 2016 at 9:29 PM, Ethan Furman  wrote:

[...] we can't do:

 app_root = Path(...)
 config = app_root/'settings.cfg'
 with open(config) as blah:
 # whatever

It feels like instead of addressing this basic disconnect, the answer has
instead been:  add that to pathlib!  Which works great -- until a user or a
library gets this path object and tries to use something from os on it.

I agree that asking for config.open() isn't the right answer here
(even if it happens to work).


How come?


But in this example, once 3.5.2 is out,
the solution would be to use open(config.path), and that will also
work when passing it to a library. Is it still unacceptable then?


I think so. Although in this example I would prefer the shorter 
config.open alternative as I am lazy.



I still cannot remember what the concrete issue was why we dropped 
pathlib the same day we gave it a try. It was something really stupid 
and although I hoped to reduce the size of the code, it was less 
readable. But it was not the path->str issue but something more mundane. 
It was something that forced us to use os[.path] as Path didn't provide 
something equivalent. Cannot remember.



Best,
Sven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Defining a path protocol

[Sven R. Kunze]

On 06.04.2016 21:02, Alexander Belopolsky wrote:


On Wed, Apr 6, 2016 at 2:32 PM, Brett Cannon > wrote:


+1 for __path__, +0 for __fspath__Â (I don't know how widespread
the notion that "fs" means "file system" is).


Same here.  In the good old days, "fs" stood for a "Font Server." 
 And in even older (and better?) days, FS was a "Field Separator."


The future is not the past. ;)


What about

__file_path__

?


Best,
Sven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Opcode cache in ceval loop

[Sven R. Kunze]

On 05.02.2016 00:06, Matthias Bussonnier wrote:

On Feb 4, 2016, at 08:22, Sven R. Kunze <srku...@mail.de> wrote:

On 04.02.2016 16:57, Matthias Bussonnier wrote:

On Feb 3, 2016, at 13:22, Yury Selivanov <yselivanov...@gmail.com> wrote:


An ideal way would be to calculate a hit/miss ratio over time
for each cached opcode, but that would be an expensive
calculation.

Do you mean like a sliding windows ?
Otherwise if you just want a let's say 20% miss threshold, you increment by 1 
on hit,
and decrement by 4 on miss.

Division is expensive.

I'm not speaking about division here.
if you +M / -N the counter will decrease in average only if the hit/miss ratio
is below N/(M+N), but you do not need to do the division.

Then you deoptimize only if you get < 0.


I see but it looks still more complicated. :)






On Feb 3, 2016, at 13:37, Sven R. Kunze <srku...@mail.de> wrote:


On 03.02.2016 22:22, Yury Selivanov wrote:

One way of tackling this is to give each optimized opcode
a counter for hit/misses.  When we have a "hit" we increment
that counter, when it's a miss, we decrement it.

Within a given range, I suppose. Like:

c = min(c+1, 100)

Min might be overkill, maybe you can use a or mask, to limit the windows range
to 256 consecutive call ?

Sure, that is how I would have written it in Python. But I would suggest an AND 
mask. ;-)


Sure, implementation detail I would say. Should not write emails before 
breakfast...


;-)


The other problem, with the mask, is if your increment hit 256 you wrap around 
back to 0
where it deoptimize (which is not what you want), so you might need to not mask 
the
sign bit and deoptimize only on a certain negative threshold.


Does it make sens ?


Definitely. I am curious about the actual implementation of this idea.

Best,
Sven




___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Opcode cache in ceval loop

[Sven R. Kunze]

On 04.02.2016 16:57, Matthias Bussonnier wrote:

On Feb 3, 2016, at 13:22, Yury Selivanov <yselivanov...@gmail.com> wrote:


An ideal way would be to calculate a hit/miss ratio over time
for each cached opcode, but that would be an expensive
calculation.

Do you mean like a sliding windows ?
Otherwise if you just want a let's say 20% miss threshold, you increment by 1 
on hit,
and decrement by 4 on miss.


Division is expensive.



On Feb 3, 2016, at 13:37, Sven R. Kunze <srku...@mail.de> wrote:


On 03.02.2016 22:22, Yury Selivanov wrote:

One way of tackling this is to give each optimized opcode
a counter for hit/misses.  When we have a "hit" we increment
that counter, when it's a miss, we decrement it.

Within a given range, I suppose. Like:

c = min(c+1, 100)


Min might be overkill, maybe you can use a or mask, to limit the windows range
to 256 consecutive call ?


Sure, that is how I would have written it in Python. But I would suggest 
an AND mask. ;-)


Best,
Sven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] More optimisation ideas

[Sven R. Kunze]

On 04.02.2016 14:09, Nick Coghlan wrote:

On 2 February 2016 at 06:39, Andrew Barnert via Python-Dev
 wrote:

On Feb 1, 2016, at 09:59, mike.romb...@comcast.net wrote:

  If the stdlib were to use implicit namespace packages
( https://www.python.org/dev/peps/pep-0420/ ) and the various
loaders/importers as well, then python could do what I've done with an
embedded python application for years.  Freeze the stdlib (or put it
in a zipfile or whatever is fast).  Then arrange PYTHONPATH to first
look on the filesystem and then look in the frozen/ziped storage.

This is a great solution for experienced developers, but I think it would be 
pretty bad for novices or transplants from other languages (maybe even 
including Python 2).

There are already multiple duplicate questions every month on StackOverflow from people 
asking "how do I find the source to stdlib module X". The canonical answer 
starts off by explaining how to import the module and use its __file__, which everyone is 
able to handle. If we have to instead explain how to work out the .py name from the 
qualified module name, how to work out the stdlib path from sys.path, and then how to 
find the source from those two things, with the caveat that it may not be installed at 
all on some platforms, and how to make sure what they're asking about really is a stdlib 
module, and how to make sure they aren't shadowing it with a module elsewhere on 
sys.path, that's a lot more complicated. Especially when you consider that some people on 
Windows and Mac are writing Python scripts without ever learning how to use the terminal 
or find their Python packages via Explorer/Finder.

For folks that *do* know how to use the terminal:

$ python3 -m inspect --details inspect
Target: inspect
Origin: /usr/lib64/python3.4/inspect.py
Cached: /usr/lib64/python3.4/__pycache__/inspect.cpython-34.pyc
Loader: <_frozen_importlib.SourceFileLoader object at 0x7f0d8d23d9b0>

(And if they just want to *read* the source code, then leaving out
"--details" prints the full module source, and would work even if the
standard library were in a zip archive)


I want to see and debug also core Python in PyCharm and this is not 
acceptable.


If you want to make it opt-in, fine. But opt-out is a no-go. I have a 
side-by-side comparison as we use Java and Python in production. It's 
the *ease of access* that makes Python great compared to Java.


@Andrew
Even for experienced developers it just sucks and there are more 
important things to do.



Best,
Sven

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Opcode cache in ceval loop

[Sven R. Kunze]

On 03.02.2016 22:22, Yury Selivanov wrote:

One way of tackling this is to give each optimized opcode
a counter for hit/misses.  When we have a "hit" we increment
that counter, when it's a miss, we decrement it.


Within a given range, I suppose. Like:

c = min(c+1, 100)



I kind of have something like that right now:
https://github.com/1st1/cpython/blob/opcache5/Python/ceval.c#L3035

But I only decrement that counter -- the idea is that LOAD_ATTR
is allowed to "miss" only 20 times before getting deoptimized.

I'll experiment with inc/dec on hit/miss and see how that affects
the performance.

An ideal way would be to calculate a hit/miss ratio over time
for each cached opcode, but that would be an expensive
calculation.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Opcode cache in ceval loop

[Sven R. Kunze]

On 02.02.2016 20:41, Yury Selivanov wrote:

Hi Victor,

On 2016-02-02 4:33 AM, Victor Stinner wrote:

Hi,

Maybe it's worth to write a PEP to summarize all your changes to
optimize CPython? It would avoid to have to follow different threads
on the mailing lists, different issues on the bug tracker, with
external links to GitHub gists, etc. Your code changes critical parts
of Python: code object structure and Python/ceval.c.


Not sure about that... PEPs take a LOT of time :(


True.


Besides, all my changes are CPython specific and
can be considered as an implementation detail.



At least, it would help to document Python internals ;-)


I can write a ceval.txt file explaining what's going on
in ceval loop, with details on the opcode cache and other
things.  I think it's even better than a PEP, to be honest.


I would love to see that. :)


Best,
Sven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Speeding up CPython 5-10%

[Sven R. Kunze]

On 02.02.2016 00:27, Greg Ewing wrote:

Sven R. Kunze wrote:
Are there some resources on why register machines are considered 
faster than stack machines?


If a register VM is faster, it's probably because each register
instruction does the work of about 2-3 stack instructions,
meaning less trips around the eval loop, so less unpredictable
branches and less pipeline flushes.


That's was I found so far as well.


This assumes that bytecode dispatching is a substantial fraction
of the time taken to execute each instruction. For something
like cpython, where the operations carried out by the bytecodes
involve a substantial amount of work, this may not be true.


Interesting point indeed. It makes sense that register machines only 
saves us the bytecode dispatching.


How much that is compared to the work each instruction requires, I 
cannot say. Maybe, Yury has a better understanding here.



It also assumes the VM is executing the bytecodes directly. If
there is a JIT involved, it all gets translated into something
else anyway, and then it's more a matter of whether you find
it easier to design the JIT to deal with stack or register code.


It seems like Yury thinks so. He didn't tell use so far.


Best,
Sven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Opcode cache in ceval loop

[Sven R. Kunze]

On 01.02.2016 20:51, Yury Selivanov wrote:
If LOAD_ATTR gets too many cache misses (20 in my current patch) it 
gets deoptimized, and the default implementation is used.  So if the 
code is very dynamic - there's no improvement, but no performance 
penalty either.


Will you re-try optimizing it?

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Opcode cache in ceval loop

[Sven R. Kunze]

On 01.02.2016 21:35, Yury Selivanov wrote:
It's important to understand that if we have a lot of cache misses 
after the code object was executed 1000 times, it doesn't make sense 
to keep trying to update that cache.  It just means that the code, in 
that particular point, works with different kinds of objects.


So, the assumption is that the code makes the difference here not time. 
That could be true for production code.


FWIW, I experimented with different ideas (one is to never 
de-optimize), and the current strategy works best on the vast number 
of benchmarks.


Nice.

Regarding the magic constants (1000, 20) what is the process of updating 
them?



Best,
Sven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Speeding up CPython 5-10%

[Sven R. Kunze]

On 01.02.2016 19:28, Brett Cannon wrote:
A search for [stack vs register based virtual machine] will get you 
some information.


Alright. :) Will go for that.

You aren't really supposed to yet. :) In Pyjion's case we are still 
working on compatibility, let alone trying to show a speed improvement 
so we have not said much beyond this mailing list (we have a talk 
proposal in for PyCon US that we hope gets accepted). We just happened 
to get picked up on Reddit and HN recently and so interest has spiked 
in the project.


Exciting. :)



So, it could be that we will see a jitted CPython when Pyjion
appears to be successful?


The ability to plug in a JIT, but yes, that's the hope.


Okay. Not sure what you mean by plugin. One thing I like about Python is 
that it just works. So, plugin sounds like unnecessary work.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Speeding up CPython 5-10%

[Sven R. Kunze]

On 01.02.2016 18:18, Brett Cannon wrote:



On Mon, 1 Feb 2016 at 09:08 Yury Selivanov > wrote:




On 2016-01-29 11:28 PM, Steven D'Aprano wrote:
> On Wed, Jan 27, 2016 at 01:25:27PM -0500, Yury Selivanov wrote:
>> Hi,
>>
>>
>> tl;dr The summary is that I have a patch that improves CPython
>> performance up to 5-10% on macro benchmarks. Benchmarks results on
>> Macbook Pro/Mac OS X, desktop CPU/Linux, server CPU/Linux are
available
>> at [1].  There are no slowdowns that I could reproduce
consistently.
> Have you looked at Cesare Di Mauro's wpython? As far as I know,
it's now
> unmaintained, and the project repo on Google Code appears to be
dead (I
> get a 404), but I understand that it was significantly faster than
> CPython back in the 2.6 days.
>
>

https://wpython.googlecode.com/files/Beyond%20Bytecode%20-%20A%20Wordcode-based%20Python.pdf
>
>

Thanks for bringing this up!

IIRC wpython was about using "fat" bytecodes, i.e. using 64bits per
bytecode instead of 8.  That allows to minimize the number of
bytecodes,
thus having some performance increase.  TBH, I don't think it was
"significantly faster".

If I were to do some big refactoring of the ceval loop, I'd probably
consider implementing a register VM.  While register VMs are a bit
faster than stack VMs (up to 20-30%), they would also allow us to
apply
more optimizations, and even bolt on a simple JIT compiler.


If you did tackle the register VM approach that would also settle a 
long-standing question of whether a certain optimization works for Python.


Are there some resources on why register machines are considered faster 
than stack machines?


As for bolting on a JIT, the whole point of Pyjion is to see if that's 
worth it for CPython, so that's already being taken care of (and is 
actually easier with a stack-based VM since the JIT engine we're using 
is stack-based itself).


Interesting. Haven't noticed these projects, yet.

So, it could be that we will see a jitted CPython when Pyjion appears to 
be successful?


Best,
Sven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] More optimisation ideas

[Sven R. Kunze]

Thanks, Brett. Wasn't aware of lazy imports as well. I think that one is 
even better reducing startup time as freezing stdlib.


On 31.01.2016 18:57, Brett Cannon wrote:
I have opened http://bugs.python.org/issue26252 to track writing the 
example (and before ppl go playing with the lazy loader, be aware of 
http://bugs.python.org/issue26186).


On Sun, 31 Jan 2016 at 09:26 Brett Cannon > wrote:


There are no example docs for it yet, but enough people have asked
this week about how to set up a custom importer that I will write
up a generic example case which will make sense for a lazy loader
(need to file the issue before I forget).


On Sun, 31 Jan 2016, 09:11 Donald Stufft > wrote:



On Jan 31, 2016, at 12:02 PM, Brett Cannon > wrote:

A lazy importer was added in Python 3.5


Is there any docs on how to actually use the LazyLoader in
3.5? I can’t seem to find any but I don’t really know the
import system that well.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F
6E3C BCE9 3372 DCFA



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/srkunze%40mail.de


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Opcode cache in ceval loop

[Sven R. Kunze]

On 01.02.2016 22:27, Yury Selivanov wrote:

Right now they are private constants in ceval.c.

I will (maybe) expose a private API via the _testcapi module to 
re-define them (set them to 1 or 0), only to write better unittests.  
I have no plans to make those constants public or have a public API to 
tackle them.  IMHO, this is something that almost nobody will ever use.


Alright. I agree with you on that.

What I actually meant was: how can we find the optimal values? I 
understand that 1000 and 20 are some hand-figured/subjective values for now.


Is there standardized/objective way to find out the best values? What 
does best even mean here?


Best,
Sven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Speeding up CPython 5-10%

[Sven R. Kunze]

On 01.02.2016 17:54, Yury Selivanov wrote:
If I were to do some big refactoring of the ceval loop, I'd probably 
consider implementing a register VM.  While register VMs are a bit 
faster than stack VMs (up to 20-30%), they would also allow us to 
apply more optimizations, and even bolt on a simple JIT compiler.


How do JIT and register machine related to each other? :)


Best,
Sven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] More optimisation ideas

[Sven R. Kunze]

On 30.01.2016 19:20, Serhiy Storchaka wrote:
AFAIK the most time is spent in system calls like stat or open. 
Archiving the stdlib into the ZIP file and using zipimport can 
decrease Python startup time (perhaps there is an open issue about this).


Oh, please don't. One thing I love about Python is the ease of access.

I personally think that startup time is not really a big issue; even 
when it comes to microbenchmarks.


Best,
Sven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] More optimisation ideas

[Sven R. Kunze]

On 30.01.2016 21:32, Brett Cannon wrote:
On Sat, Jan 30, 2016, 12:30 Sven R. Kunze <srku...@mail.de 
<mailto:srku...@mail.de>> wrote:


On 30.01.2016 19:20, Serhiy Storchaka wrote:
> AFAIK the most time is spent in system calls like stat or open.
> Archiving the stdlib into the ZIP file and using zipimport can
> decrease Python startup time (perhaps there is an open issue
about this).

Oh, please don't. One thing I love about Python is the ease of access.


It wouldn't be a requirement, just a nootion



That's good. :)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] FAT Python (lack of) performance

[Sven R. Kunze]

On 27.01.2016 11:59, Terry Reedy wrote:

On 1/26/2016 12:35 PM, Sven R. Kunze wrote:

I completely agree with INADA.


I an not sure you do.



I am sure I am. He wants to solve a problem the way that is natural to 
him as a unique human being.



It's like saying, because a specific crossroad features a higher
accident rate, *people need to change their driving behavior*.
*No!* People won't change and it's not necessary either. The crossroad
needs to be changed to be safer.


Safer crossroads tend to be slower unless one switched to alternate 
designs that eliminate crossing streams of traffic.


So Python can be safer AND faster ( = different design) if we try hard 
enough.


Languages that don't have integers but use residue classes (with 
wraparound) or finite integer classes (with overflow) as a (faster) 
substitute have, in practice, lots of accidents (bugs) when used by 
non-experts.  Guido noticed this, gave up on changing coder behavior, 
and put the expert behavior of checking for wraparound/overflow and 
switching to real integers (longs) into the language.  (I forget when 
this was added.)


I am glad he did because it helps humans solve their problems in a 
natural way without artificial boundaries. :)


The purpose of the artificially low input to fib() is to hide and 
avoid the bugginess of most languages.  The analogous trick with 
testing crossroads would be to artificially restrict the density of 
cars to mask the accident-proneness of a 'fast, consenting-adults' 
crossroads with no stop signs and no stop lights.




I am completely with you here, however I disagree about suspected 
hiding/avoiding mentality. You say:


Python -> *no problem with big integers but slow at small integers*
Other Language -> *faster but breaks at big integers*

Yes. That's it.

We haven't solved the human side, however. A human AGAIN would need to 
compromise on either speed or safety.



My point is: it would be insanely great if Python could be more like 
"*fast AND no problem with big integers*". No compromise here (at least 
no noticeable).


So, people could entirely *concentrate on their problem domain* without 
every worrying about such tiny little, nitty-gritty computer science 
details. I love computer science but people of other domains don't have 
the time nor the knowledge to decide properly. That's the reason why 
they might decide by using some weird micro-benchmarks. Just humans.



Same goes for Python. If it's slow using the very same piece of code
(even superficially), you better make the language faster.
Developers won't change and they won't change their code either. Just
not necessary.


Instead of making people rewrite fib to dramatically increase speed, 
we added the lru-cache decorator to get most of the benefit without a 
rewrite.  But Inada rejected this Python speedup.  An ast optimizer 
could potentially do the same speedup without the explicit decorator. 
(No side-effects?  Multiple recursive calls? Add a cache!)




Bingo! That's the spirit.

Why that decorator in the first place? Hey, I mean, if I ever want to 
write some cryptic-looking source code with 3-letters abbreviations 
(LRU), I use Assembler again. But I discovered and love Python and I 
never want to go back when my problem domain does not require me to. So, 
when a machine can detect such an optimization, hell, do it, please. 
It's more likely that I apply it at the wrong function AND only in 10% 
of the correct cases: missing 90% and introducing some wild errors.


Again human stuff.


Btw. it would be a great feature for Python 3 to be faster than Python
2.


We all agree on that.  One way for this to happen is to add optimizers 
that would make Python 'cheat' on micrebenchmarks


Then, we are all set. :)

Best,
Sven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] FAT Python (lack of) performance

[Sven R. Kunze]

On 27.01.2016 12:16, Nick Coghlan wrote:

On 27 January 2016 at 03:35, Sven R. Kunze <srku...@mail.de> wrote:

I completely agree with  INADA.

It's like saying, because a specific crossroad features a higher accident
rate, people need to change their driving behavior.
No! People won't change and it's not necessary either. The crossroad needs
to be changed to be safer.

Umm, no, that's not how this works


That's exactly how it works, Nick.

INADA uses Python as I use crossroads each day. Daily human business.

If you read his post carefully, you can discover that he just presented 
to you his perspective of the world. Moreover, I can assure you that 
he's not alone. As usual with humans it's not about facts or 
mathematically proven theorems but *perception*. It's more about 
marketing, little important details (or unimportant ones depending on 
whom you ask) and so on. Stating that he has a wrong perspective will 
not change anything. Believing that Python is treated unfair will not 
change that either.


Most people believe what they see. When they see a "FUNCTION CALL", it's 
the same in every language. Why? Because it looks like a function call ( 
name + parentheses ), it's called "function call" even if it's 
implemented completely differently. It even doesn't matter if we use 
commas, 'def', return types, etc. Because people understand the bigger 
concept, so that is what people want to compare.


Average Joe doesn't care and does not understand. He looks at the 
benchmarks. That is something he can understand. "While performance is 
not a matter when choosing first language, slowest of three makes bad 
impression and people feel less attractive about Python." << just like that


Not saying that INADA is an Average Joe, but I think you get the idea.

  - developers contribute to
community driven projects for their *own* reasons. Nobody gets to tell
them what to do unless they're paying them.


Bit off-topic.


Micro-optimising a poor algorithm won't deliver macro level
improvements because macro level code uses things like space-speed
trade-offs to improve the algorithmic efficiency (as in the example of
applying functools.lru_cache to a naive recursive fibonacci
implementation).


I completely agree, Nick. :) But that isn't the issue here.


Victor's work on FAT optimiser is interesting because it offers
opportunities to speed up even code that is already algorithmically
efficient, as well as making CPython a better platform for
experimenting with those kinds of changes.


Exactly. :)


More generally though, much larger opportunities for improvement lie
in persuading people to *stop writing code*, and instead spending more
of their time on finding and assembling code other people have
*already written* into solutions to interesting problems. *That's* the
kind of improvement that turns enormously complex problems like facial
recognition into 25 line Python scripts:
https://realpython.com/blog/python/face-recognition-with-python/


Interesting post. :) Thanks.

Btw. I completely agree with you on the "improve programming education", 
but not everybody can do it; and not everybody wants to learn and to 
practice it properly.


Best,
Sven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] FAT Python (lack of) performance

[Sven R. Kunze]

On 27.01.2016 19:33, Brett Cannon wrote:
And this is why this entire email thread has devolved into a 
conversation that isn't really going anywhere. This whole thread has 
completely lost track of the point of Victor's earlier email saying 
"I'm still working on my FAT work and don't take any notice of the 
performance numbers until more stuff gets finished". And this 
discussion of what benchmarks to care about is rather pointless since 
the core team has an implicit understanding that any performance 
improvement is taken into consideration in terms of balancing 
complexity in CPython with how much improvement it gets us. So if 
someone wants to speed up Fibonacci then they are welcome to try, but 
the solution must be maintainable in proportion to the speed increase 
it buys Python as a whole.


+1
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] FAT Python (lack of) performance

[Sven R. Kunze]

Hi,

will look into it soon. :)

Best,
Sven

On 26.01.2016 16:32, Victor Stinner wrote:

Hi,

2016-01-26 3:21 GMT+01:00 INADA Naoki :

How can I help your work?

I don't know exactly yet, but I started to write a documentation to
explain how to contribute:
http://faster-cpython.readthedocs.org/fat_python.html#how-can-you-contribute

You may contact me directly ;-)

Victor


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] FAT Python (lack of) performance

[Sven R. Kunze]

I completely agree with INADA.

It's like saying, because a specific crossroad features a higher 
accident rate, *people need to change their driving behavior*.
*No!* People won't change and it's not necessary either. The crossroad 
needs to be changed to be safer.



Same goes for Python. If it's slow using the very same piece of code 
(even superficially), you better make the language faster.
Developers won't change and they won't change their code either. Just 
not necessary.



Btw. it would be a great feature for Python 3 to be faster than Python 
2. I've heard a lot of complaints of the scientific community that 
Python is slow. Would Python 3 be significantly faster than Python 2, 
that'll be a huge reason to upgrade (and would create pressure to 
upgrade libs as well).
They are satisfied with Python so far, but would there be a language 
equally readable/maintainable and 10x faster (of course proven by some 
weird micro benchmarks - incomprehensible to most nervous subscribers to 
this list), they would readily switch over. I for one hope that *Python 
itself will be that language* in the foreseeable future. This is some 
sort of marketing but also requires hard facts indeed.


Best,
Sven

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] FAT Python (lack of) performance

[Sven R. Kunze]

Hi Victor,

I encourage you to proceed here. I would love to see your PEPs (509-511) 
incorporated into CPython. It's not that I consider Python slow 
(although some folks claim so), but performance improvements are always 
welcome; especially when I glance over diagrams like those: 
http://blog.carlesmateo.com/wp-content/uploads/2014/10/blog-carlesmateo-com-performance-several-languages-php7-phantomjs-nodejs-java-bash-go-perl-luajit-hhvm3_9-scale_mod5.png


So, I join Barry when he says, we want more benchmarking and definite 
results, however, I might be less strict than he is and say:


- even if FAT might not optimize significantly (whatever definition we 
apply), PEP 509 to 511 are a great win for CPython
- they provide a great infrastructure for optimizing CPython AND 
extending/experimenting Python as an ecosystem
- FAT provides interesting insights into the field of optimizing a 
dynamic language


So, keep up the good work. I am eager to see where this goes.

If there's anything I can do, let me know. :)

Best,
Sven

On 25.01.2016 19:16, Victor Stinner wrote:

Hi,

Summary: FAT Python is not faster, but it will be ;-)

--

When I started the FAT Python as a fork of CPython 3.6, I put
everything in the same repository. Last weeks, I focused on splitting
my giant patch (10k lines) into small reviewable patches. I wrote 3
PEP (509 dict version, 510 function specialziation, 511 code
tranformers) and I enhanced the API to make it usable for more use
cases than just FAT Python. I also created fatoptimizer (the AST
optimizer) and fat (runtime dependency of the optimizer) projects on
GitHub to separate clearly what should be outside Python core. For all
links, see:

http://faster-cpython.readthedocs.org/fat_python.html

For the fatoptimizer project, my constraint is to be able to run the
full Python test suite unmodified. In practice, I have to disable some
optimizations by putting a "__fatoptimizer__= {...}" configuration to
some test files. For example, I have to disable constant folding on
test_bool because it tests that False+2 gives 2 at runtime, whereas
the optimizer replaces directly False+2 with 2 during the compilation.
Well, test_bool.py is not the best example because all tests pass with
the constant folding optimization (if I comment my
"__fatoptimizer__={...}" change).

This constraint ensures that the optimizer "works" and doesn't break
(too much ;-)) the Python semantics, but it's more difficult to
implement powerful optimizations.

I also found and fixed various kinds of bugs. In my code obviously,
but also in the Python core, in various places. Some bugs only concern
AST transformers which is a new feature, but I had to fix them. For
example, Python didn't support negative line number delta in
co_lntotab of code objects, and so line number were all wrong on
optimized code. I merged my enhancement in the default branch of
CPython (issue #26107).

In short, I focused on having something working (respecting the Python
semantics), rather than spending time on writing optimizations.

--

When I asked explicitly "Is someone opposed to this PEP 509 [dict
verion] ?", Barry Warsaw answered that a performance analysis is
required. Extract of his mail:

"I still think this is maintenance and potential performance
overhead we don't want to commit to long term unless it enables
significant optimization.  Since you probably can't prove that without
some experimentation, this API should be provisional."

Last week, I ran some benchmarks and I have to admin that I was
disappointed. Not only fatoptimizer doesn't make Python faster, but it
makes it much slower on some tests!

http://fatoptimizer.readthedocs.org/en/latest/benchmarks.html

Quickly, I identified a major performance issue when nested functions
are specialized, especially in Lib/json/encoder.py (tested by
bm_json_v2.py benchmark). I fixed my optimizer to not specialize
nested functions anymore. This simple change fixed the main
performance issue. Reminder: in performance critical code, don't use
nested functions! I will maybe propose patches for Lib/json/encoder.py
to stop using nested functions.

I only ran benchmarks with the optimizer enabled. I now have to
measure the overhead of my patches (PEP 509, 510 and 511) adding the
API fat AST optimizers. The overhead must be negligible. For me, it's
a requirement of the whole project. Changes must not make Python
slower when the optimizer is not used.

fatoptimizer is faster on microbenchmarks, but I had to write manually
some optimizations:

http://fatoptimizer.readthedocs.org/en/latest/microbenchmarks.html

IMHO fatoptimizer is not faster on macro benchmarks because it is not
smart enough (yet) to generate the most interesting optimizations,
like function inlining and specialization for argument types. You can
estimate the speedup if you specialize manually your functions.

--

Barry also wrote: "Did you address my suggestion on python-ideas to
make the new C API optionally 

Re: [Python-Dev] Code formatter bot

[Sven R. Kunze]

Not a core dev, but I would definitely recommend using them.

Best,
Sven

On 19.01.2016 21:59, francismb wrote:

Dear Core-Devs,
what's your opinion about a code-formatter bot for cpython.
Pros, Cons, where could be applicable (new commits, new workflow, it
doesn't make sense), ...


- At least it should follow PEP 7 ;-)
- ...


Thanks in advance,
francis

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/srkunze%40mail.de


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Asynchronous context manager in a typical network server

[Sven R. Kunze]

On 18.12.2015 22:09, Guido van Rossum wrote:
I guess we could make the default arg to sleep() 1e9. Or make it None 
and special-case it. I don't feel strongly about this -- I'm not sure 
how baffling it would be to accidentally leave out the delay and find 
your code sleeps forever rather than raising an error (since if you 
don't expect the infinite default you may not expect this kind of 
behavior). But I do feel it's not important enough to add a new 
function or method.


Why still guessing the best surrogate for infinity?

Seems like python is just missing int('inf'). :/

Best,
Sven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Second milestone of FAT Python

[Sven R. Kunze]

typo: "chance" instead of "change"

On 04.11.2015 21:14, Sven R. Kunze wrote:

Hi Victor,

great to hear. I think everybody here appreciates your efforts.

Do you think there will be any change of merging this back into CPython?

Best,
Sven

On 04.11.2015 09:50, Victor Stinner wrote:

Hi,

I'm writing a new "FAT Python" project to try to implement 
optimizations in CPython (inlining, constant folding, move invariants 
out of loops, etc.) using a "static" optimizer (not a JIT). For the 
background, see the thread on python-ideas:

https://mail.python.org/pipermail/python-ideas/2015-October/036908.html

See also the documentation:
https://hg.python.org/sandbox/fatpython/file/tip/FATPYTHON.rst
https://hg.python.org/sandbox/fatpython/file/tip/ASTOPTIMIZER.rst

I implemented the most basic optimization to test my code: replace 
calls to builtin functions (with constant arguments) with the result. 
For example, len("abc") is replaced with 3. I reached the second 
milestone: it's now possible to run the full Python test suite with 
these optimizations enabled. It confirms that the optimizations don't 
break the Python semantic.


Example:
---
>>> def func():
... return len("abc")
...
>>> import dis
>>> dis.dis(func)
  2   0 LOAD_GLOBAL  0 (len)
  3 LOAD_CONST   1 ('abc')
  6 CALL_FUNCTION1 (1 positional, 0 keyword pair)
  9 RETURN_VALUE

>>> len(func.get_specialized())
1
>>> specialized=func.get_specialized()[0]
>>> dis.dis(specialized['code'])
  2   0 LOAD_CONST   1 (3)
  3 RETURN_VALUE
>>> len(specialized['guards'])
2

>>> func()
3

>>> len=lambda obj: "mock"
>>> func()
'mock'
>>> func.get_specialized()
[]
---

The function func() has specialized bytecode which returns directly 3 
instead of calling len("abc"). The specialized bytecode has two 
guards dictionary keys: builtins.__dict__['len'] and 
globals()['len']. If one of these keys is modified, the specialized 
bytecode is simply removed (when the function is called) and the 
original bytecode is executed.



You cannot expect any speedup at this milestone, it's just to 
validate the implementation. You can only get speedup if you 
implement *manually* optimizations. See for example posixpath.isabs() 
which inlines manually the call to the _get_sep() function. More 
optimizations will be implemented in the third milestone. I don't 
know yet if I will be able to implement constant folding, function 
inlining and/or moving invariants out of loops.



Download, compile and test FAT Python with:

hg clone http://hg.python.org/sandbox/fatpython
./configure && make && ./python -m test test_astoptimizer test_fat


Currently, only 24 functions are specialized in the standard library. 
Calling a builtin function with constant arguments in not common (it 
was expected, it's only the first step for my optimizer). But 161 
functions are specialized in tests.



To be honest, I had to modify some tests to make them pass in FAT 
mode. But most changes are related to the .pyc filename, or to the 
exact size in bytes of dictionary objects.


FAT Python is still experimental. Currently, the main bug is that the 
AST optimizer can optimize a call to a function which is not the 
expected builtin function. I already started to implement code to 
understand namespaces (detect global and local variables), but it's 
not enough yet to detect when a builtin is overriden. See TODO.rst 
for known bugs and limitations.


Victor


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:https://mail.python.org/mailman/options/python-dev/srkunze%40mail.de




___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/srkunze%40mail.de


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Second milestone of FAT Python

[Sven R. Kunze]

Hi Victor,

great to hear. I think everybody here appreciates your efforts.

Do you think there will be any change of merging this back into CPython?

Best,
Sven

On 04.11.2015 09:50, Victor Stinner wrote:

Hi,

I'm writing a new "FAT Python" project to try to implement 
optimizations in CPython (inlining, constant folding, move invariants 
out of loops, etc.) using a "static" optimizer (not a JIT). For the 
background, see the thread on python-ideas:

https://mail.python.org/pipermail/python-ideas/2015-October/036908.html

See also the documentation:
https://hg.python.org/sandbox/fatpython/file/tip/FATPYTHON.rst
https://hg.python.org/sandbox/fatpython/file/tip/ASTOPTIMIZER.rst

I implemented the most basic optimization to test my code: replace 
calls to builtin functions (with constant arguments) with the result. 
For example, len("abc") is replaced with 3. I reached the second 
milestone: it's now possible to run the full Python test suite with 
these optimizations enabled. It confirms that the optimizations don't 
break the Python semantic.


Example:
---
>>> def func():
... return len("abc")
...
>>> import dis
>>> dis.dis(func)
  2   0 LOAD_GLOBAL  0 (len)
  3 LOAD_CONST   1 ('abc')
  6 CALL_FUNCTION1 (1 positional, 0 keyword pair)
  9 RETURN_VALUE

>>> len(func.get_specialized())
1
>>> specialized=func.get_specialized()[0]
>>> dis.dis(specialized['code'])
  2   0 LOAD_CONST   1 (3)
  3 RETURN_VALUE
>>> len(specialized['guards'])
2

>>> func()
3

>>> len=lambda obj: "mock"
>>> func()
'mock'
>>> func.get_specialized()
[]
---

The function func() has specialized bytecode which returns directly 3 
instead of calling len("abc"). The specialized bytecode has two guards 
dictionary keys: builtins.__dict__['len'] and globals()['len']. If one 
of these keys is modified, the specialized bytecode is simply removed 
(when the function is called) and the original bytecode is executed.



You cannot expect any speedup at this milestone, it's just to validate 
the implementation. You can only get speedup if you implement 
*manually* optimizations. See for example posixpath.isabs() which 
inlines manually the call to the _get_sep() function. More 
optimizations will be implemented in the third milestone. I don't know 
yet if I will be able to implement constant folding, function inlining 
and/or moving invariants out of loops.



Download, compile and test FAT Python with:

hg clone http://hg.python.org/sandbox/fatpython
./configure && make && ./python -m test test_astoptimizer test_fat


Currently, only 24 functions are specialized in the standard library. 
Calling a builtin function with constant arguments in not common (it 
was expected, it's only the first step for my optimizer). But 161 
functions are specialized in tests.



To be honest, I had to modify some tests to make them pass in FAT 
mode. But most changes are related to the .pyc filename, or to the 
exact size in bytes of dictionary objects.


FAT Python is still experimental. Currently, the main bug is that the 
AST optimizer can optimize a call to a function which is not the 
expected builtin function. I already started to implement code to 
understand namespaces (detect global and local variables), but it's 
not enough yet to detect when a builtin is overriden. See TODO.rst for 
known bugs and limitations.


Victor


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/srkunze%40mail.de


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com