Re: [Python-ideas] Let’s make escaping in f-literals impossible

[Philipp A.]

Hi Guido, thanks for calling me out.

Yikes, I'm terribly sorry that it came over that way!

I'll write the RFC. Should I expand the existing one (this would need
Chris’ pending changes though) or write a new one?

My goals were to sound factual and terse, not to insult anyone. And I don't
see the flaws in my phrasing, so it seems I'm still sometimes bad at
written communication.

@everyone who perceived it as Guido did: It would be really nice if you
could pinpoint the phrases and reasons that make it seem that I mean it
that way. (In a private mail to me)

Best, Philipp

Guido van Rossum  schrieb am Di., 30. Aug. 2016, 18:43:

> Philipp, you need to stop debating this issue *now*.
>
> You need to write a PEP that can go into Python 3.7. Further debate at
> the current level (a hair-width close to name-calling) is not going to
> sway anyone.
>
> (This actually goes for Chris too -- nothing is obviously going to
> change Philipp's mind, so you might as well stop debating and save all
> the onlookers the embarrassment.)
>
> --
> --Guido van Rossum (python.org/~guido)
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Let’s make escaping in f-literals impossible

[Guido van Rossum]

Philipp, you need to stop debating this issue *now*.

You need to write a PEP that can go into Python 3.7. Further debate at
the current level (a hair-width close to name-calling) is not going to
sway anyone.

(This actually goes for Chris too -- nothing is obviously going to
change Philipp's mind, so you might as well stop debating and save all
the onlookers the embarrassment.)

-- 
--Guido van Rossum (python.org/~guido)
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Let’s make escaping in f-literals impossible

[Guido van Rossum]

Please just call it f-string and move on, we've had the naming debate
previously, it's no longer productive.

Regarding eventually supporting f'{'x'}', that will have to be a new
PEP to extend PEP 498. (I previously thought it would be an
incompatibility, but since f'{' is currently invalid, it's not.
However it's a huge change conceptually and implementation-wise, and I
don't blame Eric if he doesn't want to be responsible for it. So it
has to be a new PEP, to be introduced in 3.7 at the earliest.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Let’s make escaping in f-literals impossible

[Philipp A.]

Sorry, but I'm afraid you are projecting your thinking onto others.

The syntactical constructs are called “string interpolations”, not
“interpolated strings”. I.e. they're interpolations (a certain type of
action) on strings.

Strings are the objects, not the subjects. Strings are data, we have
code/expressions that look like strings with holes, but in reality, only
the parts outside of the braces are strings.

I hope I explained my semantics here adequately. Even if they're internally
post-processed strings in the CPython code: that's an implementation
detail, not a description of the way they work for Python users.

Best, Philipp

Chris Angelico  schrieb am Di., 30. Aug. 2016, 15:43:

> On Tue, Aug 30, 2016 at 10:56 PM, Philipp A.  wrote:
> > My issue is just that it’s as much of a string as a call of a (string
> > returning) function/method or an expression concatenating strings:
> >
> > ''.join(things)   # would you call this a string?
> > '{!r}'.format(x)  # or this? it’s basically the same as this “f-string”:
> > f'{x!r}'
> > 'start' + 'end'   # or this? it’s a concatenation of two strings, just
> like
> > f'start{ "end" }'
>
> Yes, an f-string is really a form of expression, not a literal. But
> prior art has generally been to have similar constructs referred to as
> "interpolated strings" or similar terms:
>
> https://en.wikipedia.org/wiki/String_interpolation
>
> Plenty of languages have some such feature, and it's usually
> considered a type of string. Notice the close parallels between actual
> string literals used as format strings ("I have %d apples" % apples)
> and f-strings (f"I have {apples} apples"), and how this same parallel
> can be seen in many other languages. Yes, it may be a join expression
> to the compiler, but it's a string to the programmer.
>
> ChrisA
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Let’s make escaping in f-literals impossible

[Chris Angelico]

On Tue, Aug 30, 2016 at 10:56 PM, Philipp A.  wrote:
> My issue is just that it’s as much of a string as a call of a (string
> returning) function/method or an expression concatenating strings:
>
> ''.join(things)   # would you call this a string?
> '{!r}'.format(x)  # or this? it’s basically the same as this “f-string”:
> f'{x!r}'
> 'start' + 'end'   # or this? it’s a concatenation of two strings, just like
> f'start{ "end" }'

Yes, an f-string is really a form of expression, not a literal. But
prior art has generally been to have similar constructs referred to as
"interpolated strings" or similar terms:

https://en.wikipedia.org/wiki/String_interpolation

Plenty of languages have some such feature, and it's usually
considered a type of string. Notice the close parallels between actual
string literals used as format strings ("I have %d apples" % apples)
and f-strings (f"I have {apples} apples"), and how this same parallel
can be seen in many other languages. Yes, it may be a join expression
to the compiler, but it's a string to the programmer.

ChrisA
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Let’s make escaping in f-literals impossible

[Steve Dower]

On 29Aug2016 1433, Eric V. Smith wrote:

On 8/29/2016 5:26 PM, Ethan Furman wrote:

Update the PEP, then it's a bugfix.  ;)


Heh. I guess that's true. But it's sort of a big change, so shipping
beta 1 with the code not agreeing with the PEP rubs me the wrong way.

Or, I could stop worrying and typing emails, and instead just get on
with it!


I like this approach :)

But I agree. Release Manager Ned has the final say, but I think this 
change can comfortably go in during the beta period. (I also disagree 
that it's a big change - nobody could agree on the 'obvious' behaviour 
of backslashes anyway, so chances are people would avoid them anyway, 
and there was strong consensus on advising people to avoid them.)


Cheers,
Steve
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Let’s make escaping in f-literals impossible

[Eric V. Smith]

On 8/29/2016 5:40 PM, Steve Dower wrote:

On 29Aug2016 1433, Eric V. Smith wrote:

On 8/29/2016 5:26 PM, Ethan Furman wrote:

Update the PEP, then it's a bugfix.  ;)


Heh. I guess that's true. But it's sort of a big change, so shipping
beta 1 with the code not agreeing with the PEP rubs me the wrong way.

Or, I could stop worrying and typing emails, and instead just get on
with it!


I like this approach :)

But I agree. Release Manager Ned has the final say, but I think this
change can comfortably go in during the beta period. (I also disagree
that it's a big change - nobody could agree on the 'obvious' behaviour
of backslashes anyway, so chances are people would avoid them anyway,
and there was strong consensus on advising people to avoid them.)


By "big", I meant "a lot of C code changes". And you'd be surprised by 
the percentage of the tests that are devoted to backslashes inside braces!


Eric.


___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Let’s make escaping in f-literals impossible

[Ethan Furman]

On 08/29/2016 02:16 PM, Eric V. Smith wrote:


I've been looking at this, and I agree it's the best thing to do, for
now (and possibly forever).

I'm just not convinced I can get it done before alpha 1.


Isn't the f-string feature already in place?

Update the PEP, then it's a bugfix.  ;)

--
~Ethan~
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Let’s make escaping in f-literals impossible

[Eric V. Smith]

Oops, I meant beta 1 where I said alpha 1.

Eric.

On 8/29/2016 5:12 PM, Eric V. Smith wrote:

On 8/23/2016 8:18 AM, Nick Coghlan wrote:

On 21 August 2016 at 03:32, Eric V. Smith  wrote:

If anything, I'd make it an error to have any backslashes inside the
brackets of an f-string for 3.6. We could always remove this
restriction at
a later date.


+1 for this if you can find a way to do it - it eliminates the
problematic cases where the order of evaluation makes a difference,
and ensures the parts within the braces can be reliably processed as
normal Python code.


I've been looking at this, and I agree it's the best thing to do, for
now (and possibly forever).

I'm just not convinced I can get it done before alpha 1. Assuming I can
get the coding done, I think I should update PEP 498 to say there can be
no backslashes inside the curly braces. That's my preferred outcome.

If I can't get it done by alpha 1, then I think the options are:
1. Leave f-strings as they are now, and that's how they'll always be.
2. Leave f-strings as they are now, but mark them as provisional and
   warn people that the backslash restrictions will show up in an
   upcoming release.
3. Disallow any backslashes anywhere in f-strings for 3.6, and relax the
   restriction in 3.7 to make it only inside braces where the
   restriction is enforced.
4. Remove f-strings from 3.6, and add them in 3.7 with the "no backslash
   inside braces" restriction.

I'm not wild about 2: people will ignore this and will write code that
will break in 3.7. I'm also not wild about 3, since it's too restrictive.

I'm open to suggestions.

Eric.



___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Let’s make escaping in f-literals impossible

[Eric V. Smith]

On 8/23/2016 8:18 AM, Nick Coghlan wrote:

On 21 August 2016 at 03:32, Eric V. Smith  wrote:

If anything, I'd make it an error to have any backslashes inside the
brackets of an f-string for 3.6. We could always remove this restriction at
a later date.


+1 for this if you can find a way to do it - it eliminates the
problematic cases where the order of evaluation makes a difference,
and ensures the parts within the braces can be reliably processed as
normal Python code.


I've been looking at this, and I agree it's the best thing to do, for 
now (and possibly forever).


I'm just not convinced I can get it done before alpha 1. Assuming I can 
get the coding done, I think I should update PEP 498 to say there can be 
no backslashes inside the curly braces. That's my preferred outcome.


If I can't get it done by alpha 1, then I think the options are:
1. Leave f-strings as they are now, and that's how they'll always be.
2. Leave f-strings as they are now, but mark them as provisional and
   warn people that the backslash restrictions will show up in an
   upcoming release.
3. Disallow any backslashes anywhere in f-strings for 3.6, and relax the
   restriction in 3.7 to make it only inside braces where the
   restriction is enforced.
4. Remove f-strings from 3.6, and add them in 3.7 with the "no backslash
   inside braces" restriction.

I'm not wild about 2: people will ignore this and will write code that 
will break in 3.7. I'm also not wild about 3, since it's too restrictive.


I'm open to suggestions.

Eric.

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Let’s make escaping in f-literals impossible

[Nick Coghlan]

On 21 August 2016 at 03:32, Eric V. Smith  wrote:
> If anything, I'd make it an error to have any backslashes inside the
> brackets of an f-string for 3.6. We could always remove this restriction at
> a later date.

+1 for this if you can find a way to do it - it eliminates the
problematic cases where the order of evaluation makes a difference,
and ensures the parts within the braces can be reliably processed as
normal Python code.

> In any event, I'll take a look at adding this restriction, just to get an
> estimate of the magnitude of work involved. The easiest thing to do might be
> to disallow backslashes in any part of an f-string for 3.6, although that
> seems to be going too far.

Disallowing \t, \n, etc even in the plain text parts of the f-string
would indeed be problematic.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Let’s make escaping in f-literals impossible

[Chris Angelico]

On Sun, Aug 21, 2016 at 5:51 PM, Franklin? Lee
 wrote:
> Speaking of which, how is this parsed?
> f"{'\n'}"
> If escape-handling is done first, the expression is a string literal holding
> an actual newline character (normally illegal), rather than an escape
> sequence which resolves to a newline character.

It's illegal.

> If that one somehow works, how about this?
> f"{r'\n'}"

Also illegal.

> I guess you'd have to write one of these:
> f"{'\\n'}"
> f"{'''\n''')"
> rf"{'\n'}"

Modulo the typo in the second one, these all result in the same code:

>>> dis.dis(lambda: f"{'\\n'}")
  1   0 LOAD_CONST   1 ('\n')
  2 FORMAT_VALUE 0
  4 RETURN_VALUE
>>> f"{'\\n'}"
'\n'

ChrisA
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Let’s make escaping in f-literals impossible

[Eric V. Smith]

On 8/19/2016 2:57 PM, Guido van Rossum wrote:

I don't think we should take action now.

Would it make sense, as a precaution, to declare the PEP provisional for
one release? Then we can develop a sense of whether the current approach
causes real problems.

We could also emit some kind of warning if the expression part contains
an escaped quote, since that's where a potential change would cause
breakage. (Or we could leave that to the linters.)


If anything, I'd make it an error to have any backslashes inside the 
brackets of an f-string for 3.6. We could always remove this restriction 
at a later date.


I say this because as soon as f-strings make it into the wild, we're 
going to have a hard time breaking code in say 3.7 by saying "well, we 
told you that f-strings might change".


Although frankly, other than be executive fiat (which I'm okay with), I 
don't see us ever resolving the issue if f-strings are strings first, or 
if the brackets put you into "non-string mode". There are good arguments 
on both sides.


Moving to the implementation details, I'm not sure how easy it would be 
to even find backslashes, though. IIRC, backslashes are replaced early, 
before the f-string parser really gets to process the string. It might 
require a new implementation of the f-string parser independent of 
regular strings, which I likely would not have time for before beta 1. 
Although since this would be a reduction in functionality, maybe it 
doesn't have to get done by then.


I also haven't thought of how this would affect raw f-strings.

In any event, I'll take a look at adding this restriction, just to get 
an estimate of the magnitude of work involved. The easiest thing to do 
might be to disallow backslashes in any part of an f-string for 3.6, 
although that seems to be going too far.


Eric.


___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Let’s make escaping in f-literals impossible

[Paul Moore]

On 20 August 2016 at 04:57, C Anthony Risinger  wrote:
> The two string parts are string-colored and the x.partition bits would look 
> like
> any other code in the file. It won't look like concatenation at that point.

That's entirely subjective and theoretical (unless you've implemented
it and reviewed the resulting look of the code). In my (equally
subjective and theoretical) opinion it would still look like
concatenation, and would confuse me. I made a deliberate point of
saying that *to me* it looked like concatenation.

YMMV - remember this tangent was started by people stating their
opinions. Saying that your opinion differs doesn't invalidate their
(my) view.

> tl;dr, UX is weaker when the editor implies a free-form expression in every
> possible way, but the writer can't use the quote they always use, and I
> think strings will be common in f-string expression sections.

FWIW I would instantly reject any code passed to me for review which
used the same quote within an f-string as was used to delimit it,
should this proposal be accepted.

Also, a lot of code is read on media that doesn't do syntax
highlighting - email, books, etc. A construct that needs syntax
highlighting to be readable is problematic because of this.

Paul
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Let’s make escaping in f-literals impossible

[Stephen J. Turnbull]

C Anthony Risinger writes:

 > The only time I personally use a different quote is when it somehow
 > makes the data more amenable to the task at hand. The data! The
 > literal data! Not the expressions I'm conveniently inlining with
 > the help of f-strings.

You do *not* know that yet!  *Nobody* does.  Nobody has yet written an
f-string in production code, let alone read thousands and written
hundreds.  Can you be sure that after you write a couple dozen
f-strings you won't find that such "quote context" is carried over
naturally from the way you write other strings?  (Eg, because "I'm
still in a string" is signaled by the highlighting of the surrounding
stringish text.)

I think the proposed changes to the PEP fall into the "Sometimes never
is better than *right* now" category.  The arguments I've seen so far
are plausible but not founded in experience: it could easily go the
other way, and I don't see potential for true disaster.

 > If I have to water it down for people to find it acceptable (such
 > as creating simpler variables ahead-of-time) I'd probably just keep
 > using .format(...). Because what I have gained with an f-string?

I don't see a problem if you choose not to write f-strings.  Would
other people using that convention be hard for you to *read*?

 > Not just because it's at odds with other languages, but because
 > it's at odds with what the editor is telling the user (free-form
 > expression).

There are no editors that will tell you such a thing yet.

And if you trust an editor that *does* tell you that it's a free-form
expression and use the same kind of quote that delimits the f-string,
you won't actually create a syntax error.  You're merely subject to
the same kind of "problem" that you have if you don't write PEP8
conforming code.

Regards,

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Let’s make escaping in f-literals impossible

[Brendan Barnwell]

On 2016-08-19 20:57, C Anthony Risinger wrote:

In a quality editor, everything about the {...} will tell me I'm writing a
Python expression. It'll be colored like an expression. It'll do fancy
completion like an expression. Aw shucks, it*IS*  a Python expression!
Except for one tiny detail: I'm not allowed to use the quote I use in 95%
of all my Python code--without thinking--because I already used it at the
string start :-( It's like this weird invisible ruh-roh-still-in-a-string
state hangs over you despite everything else suggesting otherwise


	But it IS inside a string.  That's why it's an f-string.  The essence 
of your argument seems to be that you want expressions inside f-strings 
to act just like expressions outside of f-strings.  But there's already 
a way to do that: just write the expression outside of the f-string. 
Then you can assign it to a variable, and refer to the variable in the 
f-string.  The whole point of f-strings is that they allow expressions 
*inside strings*.  It doesn't make sense to pretend those expressions 
are not inside strings.  It's true that the string itself "isn't really 
a string" in the sense that it's put together at runtime rather than 
being a constant, but again, the point of f-strings is to make things 
like that writable as strings in source code.  If you don't want to 
write them as strings, you can still concatenate separate string values 
or use various other solutions.


--
Brendan Barnwell
"Do not follow where the path may lead.  Go, instead, where there is no 
path, and leave a trail."

   --author unknown
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Let’s make escaping in f-literals impossible

[Brendan Barnwell]

On 2016-08-19 13:11, C Anthony Risinger wrote:

It might be harder to find the end of an f-string in one shot, but I
think that's the crux of the issue: to a reader/developer, is an
f-string conceptually one thing or a compound thing?

To me (someone who would like to see f-string expressions appear like
normal expressions, without extra quoting, and proper syntax
highlighting *always*, just like shell), this argument is essentially
the same as trying to use a regex to find a closing bracket or brace or
parse HTML. It's only hard (disregarding any underlying impl details)
because that view regards f-strings as singular things with only one
"end", when in reality an f-string is much much more like a compound
expression that just happens to look like a string.


	Personally I think that is a dangerous road to go down.  It seems it 
would lead to the practice of doing all sorts of complicated things 
inside f-strings, which seems like a bad idea to me.  In principle you 
could write your entire program in an f-string, but that doesn't mean we 
need to accommodate the sort of syntax highlighting that would 
facilitate that.


	To me it seems more prudent to just say that f-strings are (as the name 
implies) strings, and leave it at that.  If I ever get to the point 
where what I'm doing in the f-string is so complicated that I really 
need syntax highlighting for it to look good, I'd take that as a sign 
that I should move some of that code out of the f-string into ordinary 
expressions.


--
Brendan Barnwell
"Do not follow where the path may lead.  Go, instead, where there is no 
path, and leave a trail."

   --author unknown
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Let’s make escaping in f-literals impossible

[Ethan Furman]

On 08/19/2016 08:57 PM, C Anthony Risinger wrote:

[...]


The appeal of f-strings is the rapid inlining of whatever plus string data. "Whatever" is 
typically more complex than a simple attribute access or variable reference, though not much more 
complex eg. `object.method(key, "default")`. If I have to water it down for people to 
find it acceptable (such as creating simpler variables ahead-of-time) I'd probably just keep using 
.format(...). Because what I have gained with an f-string?


I suspect f-strings are in the same category as lambda -- if it's that complex, 
use the other tools instead.

At this point I don't see this changing.  If you want to make any headway 
you're going to have to do it with a complete alternate implementation, and 
even then I don't think you have good odds.

--
~Ethan~
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Let’s make escaping in f-literals impossible

[Paul Moore]

On 19 August 2016 at 21:50, C Anthony Risinger  wrote:
> The only real point I'm trying to make is that expressions within an
> f-string are an *escape*. They escape the normal semantics of a string
> literal and instead do something else for a while. Therefore, the escaped
> sections should not look like (or need to conform to) the rest of the string
> and they should not require quoting as if it were still within the string,
> because I escaped it already!

So, to me

 f'{x.partition(' + ')[0]}'

reads as a string concatenation. I'm not sure how you'd expect a
syntax highlighter to make it look like anything else, to be honest
(given that you're arguing *not* to highlight the whole of the content
of the f-string the same way).

The *real* solution is not to write something like this, instead write

 f"{x.partition(' + ')[0]}"

That makes it easy for *humans* to read. Computers parsing it is
irrelevant. Once you do that, the proposal here (that unescaped quotes
can be used in an f-string) also becomes irrelevant - this expression
parses exactly the same way under both the current code and the
proposed approach. And that's generally true - code that is clearly
written should, in my mind, work the same way regardless. So the
proposal ends up being merely "choose your preference as to which form
of badly-written code is a syntax error".

So the only relevance of syntax highlighting is how badly it fails
when handling badly-written or syntactically incorrect code. And
detecting an f-string just like you detect any other string, is *much*
better behaved in that situation. Detecting a closing quote is simple,
and isn't thrown off by incorrect nesting. If you want the *content*
of an f-string to be highlighted as an expression, Vim can do that, it
can apply specific syntax when in another syntax group such as an
f-string, and I'm sure other editors can do this as well - but you
should do that once you've determined where the f-string starts and
ends.

Paul
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Let’s make escaping in f-literals impossible

[David Mertz]

Ok. My .vimrc is probably different from yours. I'm sure you are right I
*could* make that happen. But I haven't so far.

On Aug 19, 2016 1:50 PM, "C Anthony Risinger"  wrote:

> On Fri, Aug 19, 2016 at 3:39 PM, David Mertz  wrote:
>
>> I don't think I've ever used a syntax highlighter than changed color of
>> \n in a string. I get the concept, but I haven't suffered for the absence
>> of that.
>>
>> Moreover, although I haven't yet used them, I really doubt I want extra
>> syntax highlighting in f-strings beyond simply the color strings appear as.
>> Well, maybe a uniform distinction for f-string vs. some other kind of
>> string, but nothing internal to the string.
>>
>> YMMV, but that would be my preference in my text editor. Curly braces are
>> perfectly good visual distinction to me.
>>
> At least vim does this and so does Sublime Text IIRC. Maybe I spend a lot
> of time writing shell code too, but I very much appreciate the extra visual
> cue.
>
> The only real point I'm trying to make is that expressions within an
> f-string are an *escape*. They escape the normal semantics of a string
> literal and instead do something else for a while. Therefore, the escaped
> sections should not look like (or need to conform to) the rest of the
> string and they should not require quoting as if it were still within the
> string, because I escaped it already!
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Let’s make escaping in f-literals impossible

[Chris Angelico]

On Sat, Aug 20, 2016 at 4:43 AM, Eric V. Smith  wrote:
> Maybe I'm the odd man out, but I really don't care if my editor ever syntax
> highlights within f-strings. I don't plan on putting anything more
> complicated than variable names in my f-strings, and I think PEP 8 should
> recommend something similar.

I'd go further than "variable names", and happily include attribute
access, subscripting (item access), etc, including a couple of levels
of same:

def __repr__(self):
return f"{self.__class__.__name__}(foo={self.foo!r}, spam={self.spam!r})"

But yes, I wouldn't put arbitrarily complex expressions into
f-strings. Whether PEP 8 needs to explicitly say so or not, I would
agree with you that it's a bad idea.

ChrisA
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Let’s make escaping in f-literals impossible

[Guido van Rossum]

I don't think we should take action now.

Would it make sense, as a precaution, to declare the PEP provisional for
one release? Then we can develop a sense of whether the current approach
causes real problems.

We could also emit some kind of warning if the expression part contains an
escaped quote, since that's where a potential change would cause breakage.
(Or we could leave that to the linters.)

-- 
--Guido van Rossum (python.org/~guido)
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Let’s make escaping in f-literals impossible

[Eric V. Smith]

On 8/18/2016 11:05 AM, Philipp A. wrote:

Hi, I originially posted this via google groups, which didn’t make it
through to the list proper, sorry! Read it here please:
https://groups.google.com/forum/#!topic/python-ideas/V1U6DGL5J1s


Hi, Philipp.

I'm including your original proposal here, so that it's archived
properly. Here's what you'd like to have work:
f'foo{repr('bar\n') * 3}baz' == 'foo"bar\n""bar\n""bar\n"baz'

We've had this discussion before, but I can't find it in the
archives. It's possible it happened off-list.

The problem I have with your proposal is that it greatly complicates
the implementation and it makes it harder for humans to reason about
the code (for the same reasons). This is not just an ease of 
implementation issue: it's a cognitive burden issue.


As it currently stands, Python strings, of all 'flavors' (raw, unicode, 
binary, and f-), are parsed the same way: an optional prefix, one or 
three quote characters, the body of the string, then matching quote 
characters. This is sufficiently simple that it can be (and often is) 
implemented with regular expressions. You also need to support 
combinations of prefixes, like raw f-strings (fr or rf).


With your proposal, it's much more difficult to find the end of an
f-string. I do not think this is a reasonable requirement.

For example, consider the following:
f'a{func({'a{':1,'3}':[{}, ')', '{']})}b'

A regex will not be able to deal with the matching braces needed to
find the end of the expression. You need to keep track of the nesting
level of parens, braces, brackets, and quotes (at least those, I might
have left something off).

The way this currently is written in Python 3.6a4:
f"a{func({'a{':1,'3}':[{}, ')', '{']})}b"

It's trivially easy to find the end of the string. It's easy for both
humans and the parsers.

Now admittedly in order to execute or syntax highlight the existing
f-strings, you need to perform this parsing. But of the 3 parsers that
ship with Python (ast.c, tokenize.py, IDLE), only ast.c needs to do
that currently. I don't think tokenize.py ever will, and IDLE might
(but could use the ast module).

I think many parsers (e.g. python-mode.el, etc.) will just be able to 
simply consume the f-strings without looking inside them and move one, 
and we shouldn't unnecessarily complicate them.



My arguments are basically:

 1. f-literals are semantically not strings, but expressions.
 2. Their escape sequences in the code parts are fundamentally both
detrimental and superfluous (they’re only in for convenience, as
confirmed by Guido in the quote below)


I disagree that they're detrimental and superfluous. I'd say they're 
consistent with all other strings.



 3. They’re detrimental because Syntax highlighters are (by design)
unable to handle this part of Python 3.6a4’s grammar. This will
cause code to be highlighted as parts of a string and therefore
overlooked. i’m very sure this will cause bugs.


I disagree.


 4. The fact that people see the embedded expressions as somehow “part
of the string” is confusing.

My poposal is to redo their grammar:
They shouldn’t be parsed as strings and post-processed, but be their own
thing. This also opens the door to potentially extend to with something
like JavaScript’s tagged templates)

Without the limitations of the string tokenization code/rules, only the
string parts would have escape sequences, and the expression parts would
be regular python code (“holes” in the literal).

Below the mentioned quote and some replies to the original thread:

Guido van Rossum > schrieb am
Mi., 17. Aug. 2016 um 20:11 Uhr:

The explanation is honestly that the current approach is the most
straightforward for the implementation (it's pretty hard to
intercept the string literal before escapes have been processed) and
nobody cares enough about the edge cases to force the implementation
to jump through more hoops.

I really don't think this discussion should be reopened. If you
disagree, please start a new thread on python-ideas.


I really think it should. Please look at python code with f-literals. if
they’re highlighted as strings throughout, you won’t be able to spot
which parts are code. if they’re highlighted as code, the escaping rules
guarantee that most highlighters can’t correctly highlight python
anymore. i think that’s a big issue for readability.


Maybe I'm the odd man out, but I really don't care if my editor ever 
syntax highlights within f-strings. I don't plan on putting anything 
more complicated than variable names in my f-strings, and I think PEP 8 
should recommend something similar.



Brett Cannon > schrieb am
Mi., 17. Aug. 2016 um 20:28 Uhr:

They are still strings, there is just post-processing on the string
itself to do the interpolation.


Sounds hacky to me. I’d rather see a proper parser for them, which of
course would 

Re: [Python-ideas] Let’s make escaping in f-literals impossible

[Nick Coghlan]

On 19 August 2016 at 18:27, Eric V. Smith  wrote:
> For something else that would become significantly more complicated to
> implement, you need look no further than the stdlib's own tokenizer module.
> So Python itself would require changes to parsers/lexers in Python/ast.c,
> IDLE, and Lib/tokenizer.py. In addition it would require adding tokens to
> Include/tokens.h and the generated Lib/token.py, and everyone using those
> files would need to adapt.
>
> Not that it's impossible, of course. But don't underestimate the amount of
> work this proposal would cause to the many places in and outside of Python
> that examine Python code.

And if folks want to do something more clever than regex based single
colour string highlighting, Python's own AST module is available to
help them out:

>>> tree = ast.parse("f'example{parsing:formatting}and trailer'")
>>> ast.dump(tree)
"Module(body=[Expr(value=JoinedStr(values=[Str(s='example'),
FormattedValue(value=Name(id='parsing', ctx=Load()), conversion=-1,
format_spec=Str(s='formatting')), Str(s='and trailer')]))])"

Extracting the location of the field expression for syntax highlighting:

>>> ast.dump(tree.body[0].value.values[1].value)
"Name(id='parsing', ctx=Load())"

(I haven't shown it in the example, but AST nodes have lineno and
col_offset fields so you can relocate the original source code for
processing)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Let’s make escaping in f-literals impossible

[Eric V. Smith]

On 8/19/2016 1:16 AM, Terry Reedy wrote:

On 8/18/2016 8:27 PM, Eric V. Smith wrote:

So something that parses or scans a Python file and currently
understands u, b, and r to be string prefixes, just needs to add f to
the prefixes it uses, and it can now at least understand f-strings (and
fr-strings). It doesn't need to implement a full-blown expression parser
just to find out where the end of a f-string is.


Indeed, IDLE has one prefix re, which has changed occasionally and which
I need to change for 3.6, and 4 res for the 4 unprefixed strings, which
have been the same, AFAIK, for decades.  It that prefixes all 4 string
res with the prefix re and o or's the results together to get the
'string' re.


For something else that would become significantly more complicated to 
implement, you need look no further than the stdlib's own tokenizer 
module. So Python itself would require changes to parsers/lexers in 
Python/ast.c, IDLE, and Lib/tokenizer.py. In addition it would require 
adding tokens to Include/tokens.h and the generated Lib/token.py, and 
everyone using those files would need to adapt.


Not that it's impossible, of course. But don't underestimate the amount 
of work this proposal would cause to the many places in and outside of 
Python that examine Python code.


Eric.

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Let’s make escaping in f-literals impossible

[Steven D'Aprano]

On Thu, Aug 18, 2016 at 08:27:50PM -0400, Eric V. Smith wrote:

> Right. Because all strings (regardless of prefixes) are first parsed as 
> strings, and then have their prefix "operator" applied, it's easy for a 
> parser to ignore any sting prefix character.

Is that why raw strings can't end with a backspace?

If so, that's the first time I've seen an explanation of that fact which 
makes sense!


-- 
Steve
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Let’s make escaping in f-literals impossible

[Terry Reedy]

On 8/18/2016 8:27 PM, Eric V. Smith wrote:

On 8/18/2016 3:15 PM, Terry Reedy wrote:



Without the escapes, existing f-unaware highlighters like IDLE's will be
broken in that they will highlight the single f-string as two strings
with differently highlighted content in the middle.  For
f'{x.partition('if')[0]}', the 'if' is and will be erroneously
highlighted as a keyword.  I consider this breakage unacceptible.


Right. Because all strings (regardless of prefixes) are first parsed as
strings, and then have their prefix "operator" applied, it's easy for a
parser to ignore any sting prefix character.

So something that parses or scans a Python file and currently
understands u, b, and r to be string prefixes, just needs to add f to
the prefixes it uses, and it can now at least understand f-strings (and
fr-strings). It doesn't need to implement a full-blown expression parser
just to find out where the end of a f-string is.


Indeed, IDLE has one prefix re, which has changed occasionally and which 
I need to change for 3.6, and 4 res for the 4 unprefixed strings, which 
have been the same, AFAIK, for decades.  It that prefixes all 4 string 
res with the prefix re and o or's the results together to get the 
'string' re.


--
Terry Jan Reedy

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Let’s make escaping in f-literals impossible

[Terry Reedy]

On 8/18/2016 3:30 PM, Steve Dower wrote:

On 18Aug2016 1215, Terry Reedy wrote:

On 8/18/2016 12:50 PM, Steve Dower wrote:

I don't think f'{x.partition('-')[0]}' is any less readable as a result
of the reused quotes,


Why are you reusing the single quote', which needs the escaping that you 
don't like, instead of any of at least 6 alternatives that do not need 
any escaping?


f'{x.partition("-")[0]}'
f'{x.partition("""-""")[0]}'
f"{x.partition('-')[0]}"
f'''{x.partition('-')[0]}'''
f"""{x.partition('-')[0]}"""
f"""{x.partition('''-''')[0]}"""

It seems to me that that this is at least somewhat a strawman issue.

If you want to prohibit backslashed quote reuse in expressions, as in 
f'{x.partition(\'-\')[0]}', that is okay with me, as this is 
unnecessary* and arguably bad.  The third alternative above is better. 
What breaks colorizers, and what I therefore object to, is the 
innovation of adding magical escaping of ' or " without \.


Or add a new style rule to PEP 8.

F-strings: avoid unnecessary escaping in the expression part of f-strings.
Good: f"{x.partition('-')[0]}"
Bad: f'{x.partition(\'-\')[0]}'

Then PEP-8 checkers will flag such usage.

*I am sure that there are possible complex expressions that would be 
prohibited by the rule that would be otherwise possible.  But they 
should be extremely rare and possibly not the best solution anyway.



I find it hard to not read f'{x.partition(' + ')[0]}' as string
concatenation.



That's a fair counter-example. Though f'{x.partition(\' + \')[0]}' still
reads like string concatenation to me at first glance. YMMV.


When the outer and inner quotes are no longer the same, the effect is 
greatly diminished if not eliminated.



and it will certainly be easier for highlighters
to handle (assuming they're doing anything more complicated than simply
displaying the entire expression in a different colour).


Without the escapes, existing f-unaware highlighters like IDLE's will be
broken in that they will highlight the single f-string as two strings
with differently highlighted content in the middle.  For
f'{x.partition('if')[0]}', the 'if' is and will be erroneously
highlighted as a keyword.  I consider this breakage unacceptible.


Won't it be broken anyway because of the new prefix?


No.  IDLE currently handles f-strings just fine other than not coloring 
the 'f'. This is a minor issue and easily fixed by adding '|f' and if 
allowed, '|F' at the end of the current stringprefix re.



I'm sure there's a fairly straightforward way for a regex to say that a
closing quote must not be preceded immediately by a backslash or by an
open brace at all without a closing brace in between.


I do not know that this is possible.
Here is IDLE's current re for an unprefixed single quoted string.
   r"'[^'\\\n]*(\\.[^'\\\n]*)*'?"
The close quote is optional because it must match a string that is in 
the process of being typed and is not yet closed.  I consider providing 
a tested augmented re to be required for this proposal.


Even then, making the parsing out of strings in Python code for 
colorizing version dependent is a problem in itself for colorizers not 
tied to a particular x.y version.  Leaving prefixes aside, I can't 
remember string delimiter syntax changing since I learned it in 1.3.



Not having escapes within the expression makes it harder for everyone
except the Python developer, in my opinion, and the rest of us ought to
go out of our way for them.


I am not sure that this says what you mean.

--
Terry Jan Reedy

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Let’s make escaping in f-literals impossible

[Chris Angelico]

On Fri, Aug 19, 2016 at 10:18 AM, Steven D'Aprano  wrote:
> On Fri, Aug 19, 2016 at 02:17:29AM +1000, Chris Angelico wrote:
>
>> Format codes are just text,
>
> I really think that is wrong. They're more like executable code.
>
> https://www.python.org/dev/peps/pep-0498/#expression-evaluation
>
> "Just text" implies it is data:
>
> result = "function(arg)"
>
> like the string on the right hand side of the = is data. You wouldn't
> say that a function call was data (although it may *return* data):
>
> result = function(arg)
>
> or that it was "just text", and you shouldn't say the same about:
>
> result = f"{function(arg)}"
>
> either since they are functionally equivalent. Format codes are "just
> text" only in the sense that source code is "just text". Its technically
> correct and horribly misleading.
>

By "format code", I'm talking about the bit after the colon, which
isn't executable code, but is a directive that says how the result is
to be formatted. These have existed since str.format() was introduced,
and have always been text, not code.

ChrisA
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Let’s make escaping in f-literals impossible

[Eric V. Smith]

On 8/18/2016 3:15 PM, Terry Reedy wrote:

On 8/18/2016 12:50 PM, Steve Dower wrote:
I find it hard to not read f'{x.partition(' + ')[0]}' as string
concatenation.


and it will certainly be easier for highlighters
to handle (assuming they're doing anything more complicated than simply
displaying the entire expression in a different colour).


Without the escapes, existing f-unaware highlighters like IDLE's will be
broken in that they will highlight the single f-string as two strings
with differently highlighted content in the middle.  For
f'{x.partition('if')[0]}', the 'if' is and will be erroneously
highlighted as a keyword.  I consider this breakage unacceptible.


Right. Because all strings (regardless of prefixes) are first parsed as 
strings, and then have their prefix "operator" applied, it's easy for a 
parser to ignore any sting prefix character.


So something that parses or scans a Python file and currently 
understands u, b, and r to be string prefixes, just needs to add f to 
the prefixes it uses, and it can now at least understand f-strings (and 
fr-strings). It doesn't need to implement a full-blown expression parser 
just to find out where the end of a f-string is.


Eric.


___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Let’s make escaping in f-literals impossible

[Steven D'Aprano]

On Thu, Aug 18, 2016 at 12:26:26PM -0400, Random832 wrote:

> There's a precedent. "$()" works this way in bash - call it a recursive
> parser context or whatever you like, but the point is that "$(command
> "argument with spaces")" works fine, and humans don't seem to have any
> trouble with it.

This is the first time I've ever seen anyone claim that humans don't 
have any trouble with bash escaping and evaluation rules.


-- 
Steve
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Let’s make escaping in f-literals impossible

[Greg Ewing]

Chris Angelico wrote:


f"This is a number: {13:0\u07c4}"


If I understand correctly, the proposal intends to make
it easier for a syntax hightlighter to treat

   f"This is a number: {foo[42]:0\u07c4}"

as

   f"This is a number: {foo[42] :0\u07c4}"
   ---- --
   highlight as string hightlight   highlight as string
  as
 code

I'm not sure an RE-based syntax hightlighter would
have any easier a time with that, because for the
second part it would need to recognise ':' as starting
a string, but only if it followed some stuff that was
preceded by the beginning of an f-string.

I'm not very familiar with syntax higlighters, so I
don't know if they're typically smart enought to cope
with things like that.

--
Greg
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Let’s make escaping in f-literals impossible

[Random832]

On Thu, Aug 18, 2016, at 15:15, Terry Reedy wrote:
> This is the crux of this thread.  Is an f-string a single string that 
> contains magically handled code, or interleaved strings using { and } as 
> closing and opening quotes (which is backwards from their normal 
> function of being opener and closer)

I'd rather conceptualize it as a sequence of two* kinds of thing:
literal character sequences [as sequences of characters other than {]
and expressions [started with {, and ended with a } that is not
otherwise part of the expression] rather than treating { as a closing
quote.

In particular, treating } as an opening quote doesn't really work, since
expressions can contain both strings (which may contain an unbalanced })
and dictionary/set literals (which contain balanced }'s which are not in
quotes) - what ends the expression is a } at the top level.

*or three, considering that escapes are used in the non-expression
parts.

> and expressions?  The latter view 
> makes the grammar context sensitive, I believe, as } could only open a 
> string if there is a previous f-tagged string an indefinite number of 
> alternations back.

} at the top level is otherwise a syntax error. I don't know enough
about the theoretical constructs involved to know if this makes it
formally 'context sensitive' or not - I don't know that it's any more
context sensitive than ) being valid if there is a matching (. Honestly,
I'd be more worried about : than }.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Let’s make escaping in f-literals impossible

[Random832]

On Thu, Aug 18, 2016, at 12:17, Chris Angelico wrote:
> The trouble with that way of thinking is that, to a human, the braces
> contain something. They don't "uncontain" it. Those braced expressions
> are still part of a string; they just have this bit of magic that gets
> them evaluated. Consider this:

There's a precedent. "$()" works this way in bash - call it a recursive
parser context or whatever you like, but the point is that "$(command
"argument with spaces")" works fine, and humans don't seem to have any
trouble with it. Really it all comes down to what exactly the "bit of
magic" is and how magical it is.

> IMO it doesn't matter that much either way - people will have to
> figure stuff out anyway. I like the idea that everything in the quotes
> is a string (and then parts of it get magically evaluated), but could
> live with there being some non-stringy parts in it. My suspicion is
> that what's easiest to code (ie easiest for the CPython parser) is
> also going to be easiest for all or most other tools (eg syntax
> highlighters).

Except the parser has to actually parse string literals into what string
they represent (so it can apply a further transformation to the result).
Syntax highlighters generally don't.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/