Re: [Python-Dev] PEP 498: Literal String Interpolation is ready for pronouncement

[haypo s]

2015-09-05 10:44 GMT+02:00 haypo s :
> PEP 498 allows to write >'abc' f'string'< which is replaced with
>>'abc' 'string'.__format__()< whereas str+str is a bad practice.

Oops, you should read which is replaced with >'abc' +
'string'.__format__()< with a '+' between the two strings.

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 498: Literal String Interpolation is ready for pronouncement

[Gustavo Carneiro]

On 5 September 2015 at 09:44, haypo s  wrote:

> 2015-09-05 5:01 GMT+02:00 Guido van Rossum :
> > And I'm ready to accept it. I'll wait until Tuesday night (Monday's a
> > holiday in the US) in case anything unforeseen comes up, but this is
> really
> > the Last Call for this PEP.
>
> String concatenation is inefficient in Python because strings are
> immutable. There is a micro-optimization which tried to reduce the bad
> performances of a+b, but it's better to avoid it.
>
> Python replaces >'abc' 'def'< with a single string >'abcdef'<. It's
> done by the parser, there is no overhead at runtime.
>
> PEP 498 allows to write >'abc' f'string'< which is replaced with
> >'abc' 'string'.__format__()< whereas str+str is a bad practice.
>
> I would prefer to force users to write an explicit '+' to remind them
> that there are more efficient ways to concatenate strings like
> ''.join((str1, str2)) or putting the first string in the f-string:
> >f'abcstring'<.
>
> Section in the PEP:
> https://www.python.org/dev/pepsstring/pep-0498/#concatenating-strings
>
>
That's a very good point!


> Victor
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/gjcarneiro%40gmail.com
>



-- 
Gustavo J. A. M. Carneiro
Gambit Research
"The universe is always one step beyond logic." -- Frank Herbert
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 498: Literal String Interpolation is ready for pronouncement

[haypo s]

2015-09-05 5:01 GMT+02:00 Guido van Rossum :
> And I'm ready to accept it. I'll wait until Tuesday night (Monday's a
> holiday in the US) in case anything unforeseen comes up, but this is really
> the Last Call for this PEP.

Would it be possible to specify a subset of the Python language
allowed in f-string? For example, __import__ or lambda should not be
used in a f-string. I'm not convinced that a loop or
list/dict/set-comprehension is a good idea neither.

I would prefer to keep as much code as possible *outside* f-string because:
- text editor knows well how to color it
- static analyzers know how to analyze it
- it encourage developers to indent and comment their code correctly,
whereas f-string has more restrictons on indentation (is it possible
to indent and comment code inside a f-string?)

For example, for me it's a common practice to write a complex
list-comprehension on two lines for readability:

newlist = [very_complex_expression(item)
for item in oldlist]
# sorry, it's hard to indent correctly in a mail client, especially Gmail

Well, I'm not convinced that we need a larger subset than what is
allowed currently in str.format(), simple expressions like: obj.attr,
obj[index], etc.

I recall horrible examples in the previous mail threads showing how
much complex code you can put inside f-string.

Even the following example from the PEP seems too complex to me:
print(f"Usage: {sys.argv[0]} [{'|'.join('--'+opt for opt in
valid_opts)}]", file=sys.stderr)

Oh, first I read [...] as a list-comprehension :-p But it's part of
the output string, not of the Python code...

I prefer to build the second parameter outside the f-string:
opts = '|'.join('--'+opt for opt in valid_opts)
print(f"Usage: {sys.argv[0]} [{opts}]", file=sys.stderr)

f-string with complex code remembers me PHP where it was possible to
mix PHP and HTML. I have bad memories such... code? template? (I don't
know how to cal them). Text editors and me were unable to identify
quickly the beginning and end of the code. We have a similar issue
with Python unit test embedding Python code to run subprocesses. My
text editor vim is unable to identify if the current code is the
outter code or the code embedded in a long triple-quoted string
(especially when you open the file at the embedded code).

Maybe we should start with a simple f-string, play with it during one
cycle (Python 3.6), and then discuss again if it's necessary to extend
the allowed Python expresions inside f-string?

If you really want to allow *any* Python inside a f-string, can you
please at least explain in th PEP why you consider that it's a good
thing?

IMHO the main advantage of allowing expressions inside f-string is
that it adds something really new compared to the current str.format()
method. Without it, f-string are less interesting.

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 498: Literal String Interpolation is ready for pronouncement

[Chris Angelico]

On Sat, Sep 5, 2015 at 7:10 PM, haypo s  wrote:
> 2015-09-05 5:01 GMT+02:00 Guido van Rossum :
>> And I'm ready to accept it. I'll wait until Tuesday night (Monday's a
>> holiday in the US) in case anything unforeseen comes up, but this is really
>> the Last Call for this PEP.
>
> Would it be possible to specify a subset of the Python language
> allowed in f-string? For example, __import__ or lambda should not be
> used in a f-string. I'm not convinced that a loop or
> list/dict/set-comprehension is a good idea neither.

PEP 8 and other style guides will quite possibly pick up some sort of
guidelines on the subject. The way I see it, though, it's like the
difference between lambda and def - there's a technical difference
(lambda and f-strings are restricted to expressions, so if you need
statements, pull it out of line), but for the rest, it's a judgement
call  as to how big your expression should be. The interpreter doesn't
need to be involved in that decision.

> I would prefer to keep as much code as possible *outside* f-string because:
> - text editor knows well how to color it
> - static analyzers know how to analyze it
> - it encourage developers to indent and comment their code correctly,
> whereas f-string has more restrictons on indentation (is it possible
> to indent and comment code inside a f-string?)

If it can be done inside parentheses, it can be done inside an
f-string. Though I suspect multi-line expressions will generally be a
bad idea - it'll be harder to see how much text is literal and how
much is the expression. But again, question for the style guides not
the interpreter.

Text editors and static analyzers will simply need to be taught that
f-strings are special forms of expression.

> Well, I'm not convinced that we need a larger subset than what is
> allowed currently in str.format(), simple expressions like: obj.attr,
> obj[index], etc.
> ...
> IMHO the main advantage of allowing expressions inside f-string is
> that it adds something really new compared to the current str.format()
> method. Without it, f-string are less interesting.

Putting these two paragraphs together: you're not convinced we need
f-strings, is that correct? I agree with your conclusion paragraph,
and feel that they *are* interesting and useful in their
currently-described state. Having full expression syntax available is
definitely worth while; it's exactly the same benefit as being able to
use lambda inside a function call, or anything else that we can do
inline.

ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 498: Literal String Interpolation is ready for pronouncement

[Chris Angelico]

On Sat, Sep 5, 2015 at 8:57 PM, Chris Angelico  wrote:
>> IMHO the main advantage of allowing expressions inside f-string is
>> that it adds something really new compared to the current str.format()
>> method. Without it, f-string are less interesting.
>
> I agree with your conclusion paragraph,
> and feel that they *are* interesting and useful in their
> currently-described state.

By the way, I've been using this PEP to drum up some excitement about
Python 3. Still trying to get some people to make the jump from 2.7,
but if this lands, it'll be cool and interesting and fun, and maybe
some more weight on the Py3 side of the scale :)

ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 498: Literal String Interpolation is ready for pronouncement

[haypo s]

2015-09-05 5:01 GMT+02:00 Guido van Rossum :
> And I'm ready to accept it. I'll wait until Tuesday night (Monday's a
> holiday in the US) in case anything unforeseen comes up, but this is really
> the Last Call for this PEP.

String concatenation is inefficient in Python because strings are
immutable. There is a micro-optimization which tried to reduce the bad
performances of a+b, but it's better to avoid it.

Python replaces >'abc' 'def'< with a single string >'abcdef'<. It's
done by the parser, there is no overhead at runtime.

PEP 498 allows to write >'abc' f'string'< which is replaced with
>'abc' 'string'.__format__()< whereas str+str is a bad practice.

I would prefer to force users to write an explicit '+' to remind them
that there are more efficient ways to concatenate strings like
''.join((str1, str2)) or putting the first string in the f-string:
>f'abcstring'<.

Section in the PEP:
https://www.python.org/dev/pepsstring/pep-0498/#concatenating-strings

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 501 Shell Command Examples

[Nick Coghlan]

On 5 September 2015 at 12:36, Nikolaus Rath  wrote:
> Hi Nick,
>
> You are giving
>
>   runcommand(sh(i"cat {filename}"))
>
> as an example that avoids injection attacks. While this is true, I think
> this is still a terrible anti-pattern[1] that should not be entombed in
> a PEP as a positive example.
>
> Could you consider removing it?
>
> (It doubly wastes resources by pointlessly calling a shell, and then by
> parsing & quoting the argument only for the shell to do the same in
> reverse).

Any reasonable implementation of that pattern wouldn't actually call a
system shell, it would invoke something like Julia's command system.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Critique of PEP 502 (String Interpolation Redux)

[Mike Miller]

On 09/04/2015 08:15 PM, Guido van Rossum wrote:

The text of the PEP has too much on motivation and rationale: maybe that would
be suitable for an informative PEP.  The proposal itself is under-specified.
But the real weakness cannot be fixed by improving the text: it is in the key…


I'm planning on writing an improved version, but haven't the time to get to it 
this past week; huge project at work.  (Though it may not matter at this point 
as the last sentence indicates.)


Since another thread on the list just started about whether to allow arbitrary 
expressions, I think it is important to have a good explanation of that for 
those that might disagree at first, a place to send them.  That was easier for 
me to write than making interpreter improvements, which explains the current 
state of PEP 502 well... fair enough.



This is initially likable: to recipients who don't know about e-strings they
behave just like regular strings; but recipients who *do* know about them can
extract the template and the values to be interpolated and use these to build an
alternative string, e.g. to be used as a translation.

However, in order to fulfill the former requirement, the type must inherit from
str (it's one of those built-in types for which duck typing doesn't always
work). And at the same time this is a real problem, because it's too easy to..


Yes, was aiming to "split the difference" and produce a helpful object that 
could alleviate various concerns.  The design is somewhat successful at that. 
However, while we could mitigate "losing the object" with docs and 
magic-methods, the end-result is still less compelling than I had hoped.  The 
exercise was helpful in that regard.


There is something quite nice about the simplicity of PEP 498 for the developer. 
 It can also be explained to critics as just "syntactic sugar over existing 
str.format," which is compelling.  Though, can we do any cherry picking from 
these PEPs?


While the 498 implementation is great, I'd argue that the 502 expression-string 
nomenclature is a bit better.  The terminology is not heavily used for strings 
in the stdlib (as format and template are), the e/expression name highlights the 
difference in capability, and in conversations doesn't remind one of the f-word, 
haha.


-Mike
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 498: Literal String Interpolation is ready for pronouncement

[Nathaniel Smith]

On Sep 5, 2015 11:32 AM, "Eric V. Smith"  wrote:
>
> > Actually, my current implementation doesn't use the lexer, although I
> > suppose it could. I'm currently manually scanning the string, keeping
> > track of strings and parens. To find the end of an expression, it looks
> > for a '!', ':', or non-doubled '}', not inside of a string or (), [], or
> > {}. There's a special case for '!=' so the bang isn't seen as ending the
> > expression.
>
> Ignore the part about non-doubled '}'. The actual description is:
>
> To find the end of an expression, it looks for a '!', ':', or '}', not
> inside of a string or (), [], or {}. There's a special case for '!=' so
> the bang isn't seen as ending the expression.

Sounds like you're reimplementing a lot of the lexer... I guess that's
doable, but how confident are you that your definition of "inside a string"
matches the original in all corner cases?

In any case the abstract language definition part should be phrased in
terms of the python lexer -- the expression ends when you encounter the
first } *token* that is not nested inside () [] {} *tokens*, and then you
can implement it however makes sense...

(This is then the same rule that patsy uses to find the end of python
expressions embedded inside patsy formula strings: patsy.readthedocs.org)

-n
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 498: Literal String Interpolation is ready for pronouncement

[Victor Stinner]

> If you really want to allow *any* Python inside a f-string, can you
>> please at least explain in the PEP why you consider that it's a good
>> thing?
>>
>
> Sigh. We've gone over this on python-ideas. Your objection is not new.
>

I'm sure that later others will have exactly the same question than me.
Is''t the purpose of the PEP to summarize a discussion to not have to
repeat it again and again?

Please, can anyone summerize the discussion (on allowing or not complex
expressions in f-string) into a short paragraph in the PEP?

I don't follow python-ideas and I don't want to subscribe.

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 498: Literal String Interpolation is ready for pronouncement

[Brett Cannon]

On Sat, 5 Sep 2015 at 09:19 Guido van Rossum  wrote:

> On Sat, Sep 5, 2015 at 2:10 AM, haypo s  wrote:
>
>> 2015-09-05 5:01 GMT+02:00 Guido van Rossum :
>> > And I'm ready to accept it. I'll wait until Tuesday night (Monday's a
>> > holiday in the US) in case anything unforeseen comes up, but this is
>> really
>> > the Last Call for this PEP.
>>
>> Would it be possible to specify a subset of the Python language
>> allowed in f-string? For example, __import__ or lambda should not be
>> used in a f-string. I'm not convinced that a loop or
>> list/dict/set-comprehension is a good idea neither.
>>
>
> We already went over this. You might as well argue that __import__ or
> lambda should not be used as arguments to print(). It's an expression, and
> it must allow exactly everything that is allowed in other places where
> expressions are allowed.
>
> Of course you don't *have* to write those things in f-string
> interpolations. But that's just a style guide; the language should not try
> to second-guess the user.
>

Right, it's the whole "consenting adults" bit along with trying to minimize
surprise by placing restrictions that are hard to remember. Having the
restriction of "it must be an expression" is pretty minor thing to remember
vs. "it must a specific f-string subset of Python as defined in section
N.N.N of the language spec". One of those fits in my brain and one of them
does not. ;)


>
>
>> I would prefer to keep as much code as possible *outside* f-string
>> because:
>> - text editor knows well how to color it
>> - static analyzers know how to analyze it
>> - it encourage developers to indent and comment their code correctly,
>> whereas f-string has more restrictons on indentation
>
>
> Really, we already went over most of this. You can put whitespace (even
> newlines) exactly where they are allowed in other expressions, as long as
> they don't terminate the string. You probably shouldn't do any of those
> things regularly, but there are lots of other things you that you can do in
> Python that you shouldn't.
>

As Guido said, we're consenting adults and if it makes the code unreadable
then don't do it. Just like we promote refactoring code to have more
functions to be more readable even though it adds overhead, I suspect we
will end up updating PEP 8 to say that f-strings should tend towards simple
expressions and should never compromise the readability of the code, in
which case the expressions should be hoisted out of the f-string and into
the surrounding code.

I also don't think that we should tie our hands in the design of the
language because it makes some work on behalf of the tool providers. If it
made it impossible then I think we should discuss it, but simply requiring
some more work from tools isn't a sufficient condition to restrict what we
do. That's like saying Guido shouldn't have gone with whitespace-delimited
formatting because most editors didn't support it back in the day and it
took some effort to support it since it wasn't such a simple "find the
matching curly brace" algorithm that applied to most languages.

I appreciate wanting to *promote* making code be readable, I don't think it
should come at the *cost* of the simplicity of the solution when coding
practices can easily make up for any ugliness that is possible.

-Brett


>
>>
>> (is it possible to indent and comment code inside a f-string?)
>>
>
> Now that's an interesting question. I think the answer must be No, because
> we don't want to deal with ambiguities like whether a closing curly bracket
> or string quote should be ignored inside such comments. The processing of
> f-strings described by the PEP uses several phases:
>
> - find the end of the string (the final quote[s]) using the same algorithm
> used for all string literals
> - expand \ escapes (e.g. \u)
> - look for single {, then scan ahead to a matching } -- this skips
> matching () [] {}
> - look for optional !a,!r,!s and ! inside each {...} pair
> - take what's left, enclose it in (), parse as expression
>
> The presence of comments would impede several of these stages.
>
>
>> For example, for me it's a common practice to write a complex
>> list-comprehension on two lines for readability:
>>
>> newlist = [very_complex_expression(item)
>> for item in oldlist]
>> # sorry, it's hard to indent correctly in a mail client, especially Gmail
>>
>
> The list comprehension across two lines without the comment would be fine
> in a triple-quoted f-string (or perhaps even in a single-quoted f-string if
> you put a \ before the line break).
>
>
>> Well, I'm not convinced that we need a larger subset than what is
>> allowed currently in str.format(), simple expressions like: obj.attr,
>> obj[index], etc.
>>
>
> The PEP explains why actually.
>
>
>> I recall horrible examples in the previous mail threads showing how
>> much complex code you can put inside f-string.
>>
>
> Yes, that was a very 

Re: [Python-Dev] PEP 498: Literal String Interpolation is ready for pronouncement

[Nathaniel Smith]

On Sep 5, 2015 9:20 AM, "Guido van Rossum"  wrote:
>
> The processing of f-strings described by the PEP uses several phases:
>
> - find the end of the string (the final quote[s]) using the same
algorithm used for all string literals
> - expand \ escapes (e.g. \u)

It might be worth calling out explicitly in the text how this works for
what I suspect will be the most common case: embedded quoting embedded
quotes, like

f"{ table[\"foo\"] }"

> - look for single {, then scan ahead to a matching } -- this skips
matching () [] {}

I assume this part of the algorithm uses the lexer and scans for matching
tokens rather than just scanning for characters? I tried looking at the PEP
to double check but couldn't find the relevant part. It's important for
things like

 f"{ '}' }"

> - look for optional !a,!r,!s and ! inside each {...} pair
> - take what's left, enclose it in (), parse as expression

In fact maybe this whole list that Guido wrote should be copied into the
PEP; it's a good summary :-).

-n
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 498: Literal String Interpolation is ready for pronouncement

[Terry Reedy]

On 9/5/2015 12:18 PM, Guido van Rossum wrote:

On Sat, Sep 5, 2015 at 2:10 AM, haypo s > wrote:



We already went over this. You might as well argue that __import__ or
lambda should not be used as arguments to print(). It's an expression,
and it must allow exactly everything that is allowed in other places
where expressions are allowed.



(is it possible to indent and comment code inside a f-string?)


I cannot think of any expression where indents on continuation lines 
have any significance other than being extra whitespace.  Comments are 
allowed, but cannot cause line continuation by ending with \.



(1 + #\

... 2)
3

1 + #\

  File "", line 1
1 + #\
 ^
SyntaxError: invalid syntax


Now that's an interesting question. I think the answer must be No,


This is a restriction on 'exactly everything'.  The doc should then say 
'any expression without embedded comments' (plus the usual restriction 
on not having embedded strings using the same quotes as the string itself).



because we don't want to deal with ambiguities like whether a closing
curly bracket or string quote should be ignored inside such comments.
The processing of f-strings described by the PEP uses several phases:

- find the end of the string (the final quote[s]) using the same
algorithm used for all string literals
- expand \ escapes (e.g. \u)
- look for single {, then scan ahead to a matching } -- this skips
matching () [] {}
- look for optional !a,!r,!s and ! inside each {...} pair
- take what's left, enclose it in (), parse as expression

The presence of comments would impede several of these stages.


--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 501 Shell Command Examples

[Nikolaus Rath]

On Sep 05 2015, Nick Coghlan  wrote:
> On 5 September 2015 at 12:36, Nikolaus Rath  wrote:
>> Hi Nick,
>>
>> You are giving
>>
>>   runcommand(sh(i"cat {filename}"))
>>
>> as an example that avoids injection attacks. While this is true, I think
>> this is still a terrible anti-pattern[1] that should not be entombed in
>> a PEP as a positive example.
>>
>> Could you consider removing it?
>>
>> (It doubly wastes resources by pointlessly calling a shell, and then by
>> parsing & quoting the argument only for the shell to do the same in
>> reverse).
>
> Any reasonable implementation of that pattern wouldn't actually call a
> system shell, it would invoke something like Julia's command system.

That's obvious to someone like you who thinks about this in terms of the
best implementation.

To someone less experienced, or just coming at from a different angle,
this example just says "writing a shell command is a good way to start
an external program, as long as I take care of quoting".


Best,
-Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] What's New editing

[Guido van Rossum]

Awesome! We need more people with those skills!

--Guido (on mobile)
On Sep 5, 2015 11:07 AM, "Yury Selivanov"  wrote:

> On 2015-09-05 1:27 PM, David Mertz wrote:
>
>> I have to apologize profusely here.  Just after I offered to do this (and
>> work even said it was OK in principle to do it on work time), my work load
>> went through the roof.  And now it's really already later than most of it
>> should have been done.  I'd still very much like to work on this, but I
>> wonder if maybe someone else would like to be co-author since the increased
>> workload doesn't actually seem likely to diminish soon.
>>
>
> I can recommend someone -- my colleague Elvis Pranskevichus.
>
> Elvis is a highly experienced Python developer with deep experience in
> networks programming, databases, compilers etc.  He also has very good
> editing skills, and helped me to better shape PEP 492 (Rationale and
> Abstract sections were written by him).
>
> He isn't a core developer but I can review and commit his patches, as well
> as assist him with other matters, like checking the NEWS file and ensuring
> proper versionadded/changed tags in out documentation.  We can spend 60-70
> hours on this task.
>
> Yury
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEPs and PEP 8 changes

[Jim J. Jewett]

PEP 498 is only the latest PEP where part of the concern is fear that
it may encourage certain types of bad code.

Would it be reasonable to ask PEPs to start including a section on any
recommended changes to PEP8?  (e.g., "If an embedded expression
doesn't fit on a single line, factor it out to a named variable.")

I realize that there will be times when best practices (or common
mistakes) aren't obvious in advance, but I'm a bit uncomfortable with
"PEP 8 will probably grow advice"... if we expect to need such advice,
then we should probably include it from the start.  (PEP 8 is, after
all, only advice.)

-jJ
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] What's New editing

[David Mertz]

I have to apologize profusely here.  Just after I offered to do this (and
work even said it was OK in principle to do it on work time), my work load
went through the roof.  And now it's really already later than most of it
should have been done.  I'd still very much like to work on this, but I
wonder if maybe someone else would like to be co-author since the increased
workload doesn't actually seem likely to diminish soon.

On Wed, Sep 2, 2015 at 7:03 PM, Yury Selivanov 
wrote:

>
>
> On 2015-07-06 11:38 AM, David Mertz wrote:
>
>> Hi Folks,
>>
>> I hereby volunteer to write "What's New for Python 3.5?" if folks on
>> python-dev are fine with me taking the job (i.e. I ran it by Travis, my
>> boss at Continuum, and he's happy to allow me to do that work within my
>> salaried hours... so having time isn't a problem).
>>
>> If this is OK with the powers-that-be, I'll coordinate with David Murray
>> on how best to take over this task from him.
>>
>
> Hi David,
>
> Are you still going to work on what's new for 3.5?
>
> Yury
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/mertz%40gnosis.cx
>



-- 
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] What's New editing

[Yury Selivanov]

On 2015-09-05 1:27 PM, David Mertz wrote:
I have to apologize profusely here.  Just after I offered to do this 
(and work even said it was OK in principle to do it on work time), my 
work load went through the roof.  And now it's really already later 
than most of it should have been done.  I'd still very much like to 
work on this, but I wonder if maybe someone else would like to be 
co-author since the increased workload doesn't actually seem likely to 
diminish soon.


I can recommend someone -- my colleague Elvis Pranskevichus.

Elvis is a highly experienced Python developer with deep experience in 
networks programming, databases, compilers etc.  He also has very good 
editing skills, and helped me to better shape PEP 492 (Rationale and 
Abstract sections were written by him).


He isn't a core developer but I can review and commit his patches, as 
well as assist him with other matters, like checking the NEWS file and 
ensuring proper versionadded/changed tags in out documentation.  We can 
spend 60-70 hours on this task.


Yury

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 498: Literal String Interpolation is ready for pronouncement

[Eric V. Smith]

> Actually, my current implementation doesn't use the lexer, although I
> suppose it could. I'm currently manually scanning the string, keeping
> track of strings and parens. To find the end of an expression, it looks
> for a '!', ':', or non-doubled '}', not inside of a string or (), [], or
> {}. There's a special case for '!=' so the bang isn't seen as ending the
> expression.

Ignore the part about non-doubled '}'. The actual description is:

To find the end of an expression, it looks for a '!', ':', or '}', not
inside of a string or (), [], or {}. There's a special case for '!=' so
the bang isn't seen as ending the expression.

Eric.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 498: Literal String Interpolation is ready for pronouncement

[Eric V. Smith]

On 9/5/2015 3:23 PM, Nathaniel Smith wrote:
> On Sep 5, 2015 11:32 AM, "Eric V. Smith"  > wrote:
>> Ignore the part about non-doubled '}'. The actual description is:
>>
>> To find the end of an expression, it looks for a '!', ':', or '}', not
>> inside of a string or (), [], or {}. There's a special case for '!=' so
>> the bang isn't seen as ending the expression.
> 
> Sounds like you're reimplementing a lot of the lexer... I guess that's
> doable, but how confident are you that your definition of "inside a
> string" matches the original in all corner cases?

Well, this is 35 lines of code (including comments), and it's much
simpler than a lexer (in the sense of "something that generates
tokens"). So I don't think I'm reimplementing a lot of the lexer.

However, your point is valid: if I don't do the same thing the lexer
would do, I could either prematurely find the end of an expression, or
look too far. In either case, when I call ast.parse() I'll get a syntax
error, and/or I'll get an error when parsing/lexing the remainder of the
string.

But it's not like I have to agree with the lexer: no larger error will
occur if I get it wrong. Everything is confined to a single f-string,
since I've already used the lexer to find the f-string in its entirety.
I only need to make sure the users understand how expressions are
extracted from f-strings.

I did look at using the actual lexer (Parser/tokenizer.c) to do this,
but it would require a large amount of surgery. I think it's overkill
for this task.

So far, I've tested it enough to have reasonable confidence that it's
correct. But the implementation could always be swapped out for an
improved version. I'm certainly open to that, if we find cases that the
simple scanner can't deal with.

> In any case the abstract language definition part should be phrased in
> terms of the python lexer -- the expression ends when you encounter the
> first } *token* that is not nested inside () [] {} *tokens*, and then
> you can implement it however makes sense...

I'm not sure that's an improvement on Guido's description when you're
trying to explain it to a user. But when time comes to write the
documentation, we can discuss it then.

> (This is then the same rule that patsy uses to find the end of python
> expressions embedded inside patsy formula strings: patsy.readthedocs.org
> )

I don't see where patsy looks for expressions in parts of strings. Let
me know if I'm missing it.

Eric.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 498: Literal String Interpolation is ready for pronouncement

[Eric V. Smith]

On 9/5/2015 1:28 PM, Nathaniel Smith wrote:
> On Sep 5, 2015 9:20 AM, "Guido van Rossum"  > wrote:
>>
>> The processing of f-strings described by the PEP uses several phases:
>>
>> - find the end of the string (the final quote[s]) using the same
> algorithm used for all string literals
>> - expand \ escapes (e.g. \u)
> 
> It might be worth calling out explicitly in the text how this works for
> what I suspect will be the most common case: embedded quoting embedded
> quotes, like
> 
> f"{ table[\"foo\"] }"
> 
>> - look for single {, then scan ahead to a matching } -- this skips
> matching () [] {}
> 
> I assume this part of the algorithm uses the lexer and scans for
> matching tokens rather than just scanning for characters? I tried
> looking at the PEP to double check but couldn't find the relevant part.
> It's important for things like
> 
>  f"{ '}' }"

Actually, my current implementation doesn't use the lexer, although I
suppose it could. I'm currently manually scanning the string, keeping
track of strings and parens. To find the end of an expression, it looks
for a '!', ':', or non-doubled '}', not inside of a string or (), [], or
{}. There's a special case for '!=' so the bang isn't seen as ending the
expression.

So it does work on this f-string:

>>> f"{ '}' }"
'}'

Although by using the lexer, I'd be worried about multiple levels of
escaping. I'd have to give it some thought.


>> - look for optional !a,!r,!s and ! inside each {...} pair
>> - take what's left, enclose it in (), parse as expression
> 
> In fact maybe this whole list that Guido wrote should be copied into the
> PEP; it's a good summary :-).

I think the PEP does say these things, but maybe it could be tightened up.

Eric.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 498: Literal String Interpolation is ready for pronouncement

[Eric V. Smith]

On 9/5/2015 1:14 PM, Gustavo Carneiro wrote:
> Why not allow string concatenation without plus sign only if/when the
> implementation becomes optimised to allow compile time concatenation? 
> This makes it crystal clear whether the concatenation is compile time or
> runtime. If we allow it now, it's hard to tell without looking at the
> CPython source code...

Although you can make it faster than '+', it's not possible for it to be
a compile-time concatenation. The PEP calls out that it's an expression
that's evaluated at runtime.

I think having strings concatenated with adjacent f-strings follows the
principle of least surprise. I also think it won't get used all that
often, just as compile-time string concatenation isn't seen with great
frequency.

Eric.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 498: Literal String Interpolation is ready for pronouncement

[Eric V. Smith]

On 9/5/2015 12:18 PM, Guido van Rossum wrote:
> On Sat, Sep 5, 2015 at 2:10 AM, haypo s  > wrote:
>
> (is it possible to indent and comment code inside a f-string?)
> 
> 
> Now that's an interesting question. I think the answer must be No,
> because we don't want to deal with ambiguities like whether a closing
> curly bracket or string quote should be ignored inside such comments.
> The processing of f-strings described by the PEP uses several phases:

I'll update the PEP to say comments aren't allowed. It gets especially
wacky when adding () around the expression: f'{2#}' would become the
expression "(2#)", which is a syntax error.

Eric.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] What's New editing

[Yury Selivanov]

On 2015-09-05 2:23 PM, Guido van Rossum wrote:


Awesome! We need more people with those skills!

--Guido (on mobile)



Great, we'll start today!

Thanks,
Yury

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Critique of PEP 501 (General purpose string interpolation)

[Guido van Rossum]

On Sat, Sep 5, 2015 at 5:11 AM, Nick Coghlan  wrote:

> On 5 September 2015 at 13:04, Guido van Rossum  wrote:
> > I think it's too much effort for too little gain.
> >
> > The motivation feels very weak; surely writing
> >
> >   os.system("echo " + message_from_user)
> >
> > is just as easy (as is the %s spelling), so the security issue can
> hardly be
> > blamed on PEP 498.
>
> That's fair - writing PEP 501 actually made *me* a lot more
> comfortable with PEP 498.
>
> Rather than asking for a pronouncement, I've marked 501 as Deferred -
> I still think it's potentially interesting, but I also think it's
> worth taking our time and seeing how far we can get with just eager
> rendering. If cases emerge where the lack of delayed rendering support
> is keenly felt, then it isn't hard to resurrect a deferred PEP.
>

Thank you. I agree with that assessment.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 498: Literal String Interpolation is ready for pronouncement

[Guido van Rossum]

On Sat, Sep 5, 2015 at 2:10 AM, haypo s  wrote:

> 2015-09-05 5:01 GMT+02:00 Guido van Rossum :
> > And I'm ready to accept it. I'll wait until Tuesday night (Monday's a
> > holiday in the US) in case anything unforeseen comes up, but this is
> really
> > the Last Call for this PEP.
>
> Would it be possible to specify a subset of the Python language
> allowed in f-string? For example, __import__ or lambda should not be
> used in a f-string. I'm not convinced that a loop or
> list/dict/set-comprehension is a good idea neither.
>

We already went over this. You might as well argue that __import__ or
lambda should not be used as arguments to print(). It's an expression, and
it must allow exactly everything that is allowed in other places where
expressions are allowed.

Of course you don't *have* to write those things in f-string
interpolations. But that's just a style guide; the language should not try
to second-guess the user.


> I would prefer to keep as much code as possible *outside* f-string because:
> - text editor knows well how to color it
> - static analyzers know how to analyze it
> - it encourage developers to indent and comment their code correctly,
> whereas f-string has more restrictons on indentation


Really, we already went over most of this. You can put whitespace (even
newlines) exactly where they are allowed in other expressions, as long as
they don't terminate the string. You probably shouldn't do any of those
things regularly, but there are lots of other things you that you can do in
Python that you shouldn't.

>
> (is it possible to indent and comment code inside a f-string?)
>

Now that's an interesting question. I think the answer must be No, because
we don't want to deal with ambiguities like whether a closing curly bracket
or string quote should be ignored inside such comments. The processing of
f-strings described by the PEP uses several phases:

- find the end of the string (the final quote[s]) using the same algorithm
used for all string literals
- expand \ escapes (e.g. \u)
- look for single {, then scan ahead to a matching } -- this skips matching
() [] {}
- look for optional !a,!r,!s and ! inside each {...} pair
- take what's left, enclose it in (), parse as expression

The presence of comments would impede several of these stages.


> For example, for me it's a common practice to write a complex
> list-comprehension on two lines for readability:
>
> newlist = [very_complex_expression(item)
> for item in oldlist]
> # sorry, it's hard to indent correctly in a mail client, especially Gmail
>

The list comprehension across two lines without the comment would be fine
in a triple-quoted f-string (or perhaps even in a single-quoted f-string if
you put a \ before the line break).


> Well, I'm not convinced that we need a larger subset than what is
> allowed currently in str.format(), simple expressions like: obj.attr,
> obj[index], etc.
>

The PEP explains why actually.


> I recall horrible examples in the previous mail threads showing how
> much complex code you can put inside f-string.
>

Yes, that was a very poorly thought out argument. All of Python should be
condemned if you took it seriously.


> Even the following example from the PEP seems too complex to me:
> print(f"Usage: {sys.argv[0]} [{'|'.join('--'+opt for opt in
> valid_opts)}]", file=sys.stderr)
>
> Oh, first I read [...] as a list-comprehension :-p But it's part of
> the output string, not of the Python code...
>
> I prefer to build the second parameter outside the f-string:
> opts = '|'.join('--'+opt for opt in valid_opts)
> print(f"Usage: {sys.argv[0]} [{opts}]", file=sys.stderr)
>

The same criticism can be made for the original version (using .format()),
which was lifted from real code.


> f-string with complex code remembers me PHP where it was possible to
> mix PHP and HTML. I have bad memories such... code? template? (I don't
> know how to cal them). Text editors and me were unable to identify
> quickly the beginning and end of the code. We have a similar issue
> with Python unit test embedding Python code to run subprocesses. My
> text editor vim is unable to identify if the current code is the
> outter code or the code embedded in a long triple-quoted string
> (especially when you open the file at the embedded code).
>

Yeah, tools will need to be updated. There are several other languages
(e.g. Ruby) that allow arbitrary expressions in strings and at least some
editors have no problem with them, even vim.


> Maybe we should start with a simple f-string, play with it during one
> cycle (Python 3.6), and then discuss again if it's necessary to extend
> the allowed Python expresions inside f-string?
>
> If you really want to allow *any* Python inside a f-string, can you
> please at least explain in the PEP why you consider that it's a good
> thing?
>

Sigh. We've gone over this on python-ideas. Your objection is not new.


> IMHO the main 

Re: [Python-Dev] Critique of PEP 501 (General purpose string interpolation)

[Nick Coghlan]

On 5 September 2015 at 13:04, Guido van Rossum  wrote:
> I think it's too much effort for too little gain.
>
> The motivation feels very weak; surely writing
>
>   os.system("echo " + message_from_user)
>
> is just as easy (as is the %s spelling), so the security issue can hardly be
> blamed on PEP 498.

That's fair - writing PEP 501 actually made *me* a lot more
comfortable with PEP 498.

Rather than asking for a pronouncement, I've marked 501 as Deferred -
I still think it's potentially interesting, but I also think it's
worth taking our time and seeing how far we can get with just eager
rendering. If cases emerge where the lack of delayed rendering support
is keenly felt, then it isn't hard to resurrect a deferred PEP.

Regards,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 498: Literal String Interpolation is ready for pronouncement

[Guido van Rossum]

On Sat, Sep 5, 2015 at 1:44 AM, haypo s  wrote:

> 2015-09-05 5:01 GMT+02:00 Guido van Rossum :
> > And I'm ready to accept it. I'll wait until Tuesday night (Monday's a
> > holiday in the US) in case anything unforeseen comes up, but this is
> really
> > the Last Call for this PEP.
>
> String concatenation is inefficient in Python because strings are
> immutable. There is a micro-optimization which tried to reduce the bad
> performances of a+b, but it's better to avoid it.
>
> Python replaces >'abc' 'def'< with a single string >'abcdef'<. It's
> done by the parser, there is no overhead at runtime.
>
> PEP 498 allows to write >'abc' f'string'< which is replaced with
> >'abc' 'string'.__format__()< whereas str+str is a bad practice.
>
> I would prefer to force users to write an explicit '+' to remind them
> that there are more efficient ways to concatenate strings like
> ''.join((str1, str2)) or putting the first string in the f-string:
> >f'abcstring'<.
>
> Section in the PEP:
> https://www.python.org/dev/pepsstring/pep-0498/#concatenating-strings
>
> Victor
>

What does the extra + buy users? It's just more punctuation noise. The
implementation is intentionally left unspecified by the PEP; it should be
possible to design an implementation that combines all the parts together
more efficiently than the use of + (e.g. using a "string builder" object
internally).

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 498: Literal String Interpolation is ready for pronouncement

[Barry Warsaw]

On Sep 04, 2015, at 08:01 PM, Guido van Rossum wrote:

>And I'm ready to accept it.

As I've mentioned to Eric, I think it's a great PEP, so congratulations on
getting one through The Gauntlet.  I know that in my own code, I'm looking
forward to using it in however many years it takes for 3.6 to be the base
compatible version of course. :)

I'm okay with it not being appropriate for i18n (IMHO) but no worries,
string.Template and libraries like mine built on top of it all do the trick
nicely and are quite stable.  No reason for them to go away, and PEP 498 is
orthogonal to that.

Still, I read *a lot* of Python code written by other people.  Most of it is
really good (it's nice that there are so many excellent Python developers out
there :).  I also know that the syntax is going to be abused and I rather
dread having to read, comprehend, and debug it.  But oh well.

In another thread it was suggested that a PEP 8 -type of best practices be
included in new PEPs.  I definitely think that any documentation written for
f-strings should include such best practices.  Maybe PEP 8 should also include
some style suggestions or admonitions for f-strings, but I'll let others
propose language for that.

Cheers,
-Barry
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 498: Literal String Interpolation is ready for pronouncement

[Eric V. Smith]

On 9/5/2015 7:12 PM, Nathaniel Smith wrote:
> On Sat, Sep 5, 2015 at 1:00 PM, Eric V. Smith  wrote:
>> On 9/5/2015 3:23 PM, Nathaniel Smith wrote:
>>> On Sep 5, 2015 11:32 AM, "Eric V. Smith" >> > wrote:
 Ignore the part about non-doubled '}'. The actual description is:

 To find the end of an expression, it looks for a '!', ':', or '}', not
 inside of a string or (), [], or {}. There's a special case for '!=' so
 the bang isn't seen as ending the expression.
>>>
>>> Sounds like you're reimplementing a lot of the lexer... I guess that's
>>> doable, but how confident are you that your definition of "inside a
>>> string" matches the original in all corner cases?
>>
>> Well, this is 35 lines of code (including comments), and it's much
>> simpler than a lexer (in the sense of "something that generates
>> tokens"). So I don't think I'm reimplementing a lot of the lexer.
>>
>> However, your point is valid: if I don't do the same thing the lexer
>> would do, I could either prematurely find the end of an expression, or
>> look too far. In either case, when I call ast.parse() I'll get a syntax
>> error, and/or I'll get an error when parsing/lexing the remainder of the
>> string.
>>
>> But it's not like I have to agree with the lexer: no larger error will
>> occur if I get it wrong. Everything is confined to a single f-string,
>> since I've already used the lexer to find the f-string in its entirety.
>> I only need to make sure the users understand how expressions are
>> extracted from f-strings.
>>
>> I did look at using the actual lexer (Parser/tokenizer.c) to do this,
>> but it would require a large amount of surgery. I think it's overkill
>> for this task.
>>
>> So far, I've tested it enough to have reasonable confidence that it's
>> correct. But the implementation could always be swapped out for an
>> improved version. I'm certainly open to that, if we find cases that the
>> simple scanner can't deal with.
>>
>>> In any case the abstract language definition part should be phrased in
>>> terms of the python lexer -- the expression ends when you encounter the
>>> first } *token* that is not nested inside () [] {} *tokens*, and then
>>> you can implement it however makes sense...
>>
>> I'm not sure that's an improvement on Guido's description when you're
>> trying to explain it to a user. But when time comes to write the
>> documentation, we can discuss it then.
> 
> I'm not talking about end-user documentation, I'm talking about the
> formal specification, like in the Python Language Reference.

I think the formal specification can talk about scanning a string
looking for the end of an expression without discussing tokens. Just as
https://docs.python.org/3/reference/lexical_analysis.html#string-and-bytes-literals
uses an EBNF sort of language.

The implementation is not tokenizing anything: it's just trying to find
the end of the string in order to pass it to ast.parse(). That task is
significantly easier than tokenizing.

> I'm pretty sure that just calling the tokenizer will be easier for
> Cython or PyPy than implementing a special purpose scanner :-)

I sincerely doubt that, but I'd be curious how they implemented
_string.formatter_parser(), which is extremely close in functionality.
Sadly, just not close enough to be reusable for this.

But I'm not going to argue about implementation when the PEP hasn't been
accepted yet. Especially for code I'm not going to write in Cython and PyPy!

>>> (This is then the same rule that patsy uses to find the end of python
>>> expressions embedded inside patsy formula strings: patsy.readthedocs.org
>>> )
>>
>> I don't see where patsy looks for expressions in parts of strings. Let
>> me know if I'm missing it.
> 
> Patsy parses strings like
> 
>"np.sin(a + b) + c"
> 
> using a grammar that supports some basic arithmetic-like infix
> operations (+, *, parentheses, etc.), and in which the atoms are
> arbitrary Python expressions. So the above string is parsed into a
> patsy-AST that looks something like:
> 
>   Add(PyExpr("np.sin(a + b)"), PyExpr("c"))
> 
> The rule it uses to do this is that it uses the Python tokenizer,
> counts nesting of () [] {}, and when it sees a valid unnested patsy
> operator, then that's the end of the embedded expression:
> 
>   https://github.com/pydata/patsy/blob/master/patsy/parse_formula.py#L37
> 
> Not tremendously relevant, but that's why I've thought this through before :-)

For the actual parsing of the expressions, I use the equivalent of
ast.parse(). So that phase of the processing I'm not worried about.

Eric.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEPs and PEP 8 changes

[Brett Cannon]

On Sat, 5 Sep 2015 at 12:58 Jim J. Jewett  wrote:

> PEP 498 is only the latest PEP where part of the concern is fear that
> it may encourage certain types of bad code.
>
> Would it be reasonable to ask PEPs to start including a section on any
> recommended changes to PEP8?  (e.g., "If an embedded expression
> doesn't fit on a single line, factor it out to a named variable.")
>
> I realize that there will be times when best practices (or common
> mistakes) aren't obvious in advance, but I'm a bit uncomfortable with
> "PEP 8 will probably grow advice"... if we expect to need such advice,
> then we should probably include it from the start.  (PEP 8 is, after
> all, only advice.)
>

I have no issue suggesting it, but I don't think it needs to be a rule
either.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEPs and PEP 8 changes

[Barry Warsaw]

On Sep 05, 2015, at 03:57 PM, Jim J. Jewett wrote:

>Would it be reasonable to ask PEPs to start including a section on any
>recommended changes to PEP8?  (e.g., "If an embedded expression
>doesn't fit on a single line, factor it out to a named variable.")

Yes, I think it's reasonable to encourage a "best practices" section in
relevant PEPs.  Those best practices can include recommendations for updates
to PEP 8.

>I realize that there will be times when best practices (or common mistakes)
>aren't obvious in advance, but I'm a bit uncomfortable with "PEP 8 will
>probably grow advice"... if we expect to need such advice, then we should
>probably include it from the start.  (PEP 8 is, after all, only advice.)

Except of course that for some projects, it's more than that.  The pep8 (and
related) tools are often used in test suites to force style compliance within
a project.  That's fine except that, to paraphrase Dr. Strangelove, the whole
point of such tests is lost if you don't run those tests!  I've actually had
to fix several packages recently whose pep8 tests fail.

I'm often reluctant to lock PEP 8 down too much because it can sometimes be a
straight jacket.

Cheers,
-Barry
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 498: Literal String Interpolation is ready for pronouncement

[Guido van Rossum]

On Sat, Sep 5, 2015 at 2:22 PM, Victor Stinner 
wrote:

>
> If you really want to allow *any* Python inside a f-string, can you
>>> please at least explain in the PEP why you consider that it's a good
>>> thing?
>>>
>>
>> Sigh. We've gone over this on python-ideas. Your objection is not new.
>>
>
> I'm sure that later others will have exactly the same question than me.
> Is''t the purpose of the PEP to summarize a discussion to not have to
> repeat it again and again?
>
> Please, can anyone summerize the discussion (on allowing or not complex
> expressions in f-string) into a short paragraph in the PEP?
>
> I don't follow python-ideas and I don't want to subscribe.
>

But you can at least read the PEP. There is a whole section "Supporting
full Python expressions" which explains this.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 498: Literal String Interpolation is ready for pronouncement

[Nathaniel Smith]

On Sat, Sep 5, 2015 at 1:00 PM, Eric V. Smith  wrote:
> On 9/5/2015 3:23 PM, Nathaniel Smith wrote:
>> On Sep 5, 2015 11:32 AM, "Eric V. Smith" > > wrote:
>>> Ignore the part about non-doubled '}'. The actual description is:
>>>
>>> To find the end of an expression, it looks for a '!', ':', or '}', not
>>> inside of a string or (), [], or {}. There's a special case for '!=' so
>>> the bang isn't seen as ending the expression.
>>
>> Sounds like you're reimplementing a lot of the lexer... I guess that's
>> doable, but how confident are you that your definition of "inside a
>> string" matches the original in all corner cases?
>
> Well, this is 35 lines of code (including comments), and it's much
> simpler than a lexer (in the sense of "something that generates
> tokens"). So I don't think I'm reimplementing a lot of the lexer.
>
> However, your point is valid: if I don't do the same thing the lexer
> would do, I could either prematurely find the end of an expression, or
> look too far. In either case, when I call ast.parse() I'll get a syntax
> error, and/or I'll get an error when parsing/lexing the remainder of the
> string.
>
> But it's not like I have to agree with the lexer: no larger error will
> occur if I get it wrong. Everything is confined to a single f-string,
> since I've already used the lexer to find the f-string in its entirety.
> I only need to make sure the users understand how expressions are
> extracted from f-strings.
>
> I did look at using the actual lexer (Parser/tokenizer.c) to do this,
> but it would require a large amount of surgery. I think it's overkill
> for this task.
>
> So far, I've tested it enough to have reasonable confidence that it's
> correct. But the implementation could always be swapped out for an
> improved version. I'm certainly open to that, if we find cases that the
> simple scanner can't deal with.
>
>> In any case the abstract language definition part should be phrased in
>> terms of the python lexer -- the expression ends when you encounter the
>> first } *token* that is not nested inside () [] {} *tokens*, and then
>> you can implement it however makes sense...
>
> I'm not sure that's an improvement on Guido's description when you're
> trying to explain it to a user. But when time comes to write the
> documentation, we can discuss it then.

I'm not talking about end-user documentation, I'm talking about the
formal specification, like in the Python Language Reference.

I'm pretty sure that just calling the tokenizer will be easier for
Cython or PyPy than implementing a special purpose scanner :-)

>> (This is then the same rule that patsy uses to find the end of python
>> expressions embedded inside patsy formula strings: patsy.readthedocs.org
>> )
>
> I don't see where patsy looks for expressions in parts of strings. Let
> me know if I'm missing it.

Patsy parses strings like

   "np.sin(a + b) + c"

using a grammar that supports some basic arithmetic-like infix
operations (+, *, parentheses, etc.), and in which the atoms are
arbitrary Python expressions. So the above string is parsed into a
patsy-AST that looks something like:

  Add(PyExpr("np.sin(a + b)"), PyExpr("c"))

The rule it uses to do this is that it uses the Python tokenizer,
counts nesting of () [] {}, and when it sees a valid unnested patsy
operator, then that's the end of the embedded expression:

  https://github.com/pydata/patsy/blob/master/patsy/parse_formula.py#L37

Not tremendously relevant, but that's why I've thought this through before :-)

-n

-- 
Nathaniel J. Smith -- http://vorpus.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 498: Literal String Interpolation is ready for pronouncement

[Gustavo Carneiro]

Why not allow string concatenation without plus sign only if/when the
implementation becomes optimised to allow compile time concatenation?  This
makes it crystal clear whether the concatenation is compile time or
runtime. If we allow it now, it's hard to tell without looking at the
CPython source code...

On Sat, 5 Sep 2015 16:57 Guido van Rossum  wrote:

On Sat, Sep 5, 2015 at 1:44 AM, haypo s  wrote:

2015-09-05 5:01 GMT+02:00 Guido van Rossum :
> And I'm ready to accept it. I'll wait until Tuesday night (Monday's a
> holiday in the US) in case anything unforeseen comes up, but this is
really
> the Last Call for this PEP.

String concatenation is inefficient in Python because strings are
immutable. There is a micro-optimization which tried to reduce the bad
performances of a+b, but it's better to avoid it.

Python replaces >'abc' 'def'< with a single string >'abcdef'<. It's
done by the parser, there is no overhead at runtime.

PEP 498 allows to write >'abc' f'string'< which is replaced with
>'abc' 'string'.__format__()< whereas str+str is a bad practice.

I would prefer to force users to write an explicit '+' to remind them
that there are more efficient ways to concatenate strings like
''.join((str1, str2)) or putting the first string in the f-string:
>f'abcstring'<.

Section in the PEP:
https
://

www.python.org
/dev/

pepsstring

/pep-0498/#concatenating-strings


Victor

What does the extra + buy users? It's just more punctuation noise. The
implementation is intentionally left unspecified by the PEP; it should be
possible to design an implementation that combines all the parts together
more efficiently than the use of + (e.g. using a "string builder" object
internally).


-- 

--Guido van Rossum (python.org /~
guido )

___
Python-Dev mailing list
Python-Dev@python.org
https ://
mail.python.org
/mailman/
listinfo
/
python-dev

Unsubscribe: https

://

mail.python.org

/mailman/options/

python-dev
/

gjcarneiro%40gmail.com

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] What's New editing

[Ryan Gonzalez]

On September 5, 2015 12:27:26 PM CDT, David Mertz  wrote:
>I have to apologize profusely here.  Just after I offered to do this
>(and
>work even said it was OK in principle to do it on work time), my work
>load
>went through the roof.  And now it's really already later than most of
>it
>should have been done.  I'd still very much like to work on this, but I
>wonder if maybe someone else would like to be co-author since the
>increased
>workload doesn't actually seem likely to diminish soon.

So...

I also have a lot of stuff right now, but it's not that bad...could I help out 
here?


>
>On Wed, Sep 2, 2015 at 7:03 PM, Yury Selivanov
>
>wrote:
>
>>
>>
>> On 2015-07-06 11:38 AM, David Mertz wrote:
>>
>>> Hi Folks,
>>>
>>> I hereby volunteer to write "What's New for Python 3.5?" if folks on
>>> python-dev are fine with me taking the job (i.e. I ran it by Travis,
>my
>>> boss at Continuum, and he's happy to allow me to do that work within
>my
>>> salaried hours... so having time isn't a problem).
>>>
>>> If this is OK with the powers-that-be, I'll coordinate with David
>Murray
>>> on how best to take over this task from him.
>>>
>>
>> Hi David,
>>
>> Are you still going to work on what's new for 3.5?
>>
>> Yury
>>
>>
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/mertz%40gnosis.cx
>>

-- 
Sent from my Nexus 5 with K-9 Mail. Please excuse my brevity.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com