Re: [swift-evolution] Prohibit invisible characters in identifier names

I've submitted a draft of the proposal on the thread "Normalize Unicode Identifiers ". Please make any comments and recommendations there. Sincerely, João Pinheiro > On 23 Jun 2016, at 18:30, Chris Lattner

Re: [swift-evolution] Prohibit invisible characters in identifier names

Yes. See earlier in the discussion about the Unicode confusables list. The security issues that arise from confusable URLs aren't the same as those for identifiers in Swift, and I think the short version of the previous discussion is that prohibiting the use of Greek nu and mathematical bold

Re: [swift-evolution] Prohibit invisible characters in identifier names

Doesn't Unicode have a standard for this that specified which characters are look-alikes? Russ > On Jun 21, 2016, at 7:48 AM, Vladimir.S via swift-evolution > wrote: > > >> On 21.06.2016 7:37, Charlie Monroe via swift-evolution wrote: >> >>> On Jun 21, 2016, at

Re: [swift-evolution] Prohibit invisible characters in identifier names

It’s leaving a lot of Linux users out to dry; a better option may be a sort of hybrid approach with a “middleman” tool that does the translation, which people could simply add as a build step if they needed translation. On Sun, Jun 26, 2016 at 11:08 PM Charlie Monroe

Re: [swift-evolution] Prohibit invisible characters in identifier names

Sure, but if you want to have translated identifiers, there's really no other (better) option unless you want to create ABI incompatible code (given that Swift 4 has a finalized ABI) that only runs on your localized system. > On Jun 27, 2016, at 7:59 AM, Saagar Jha

Re: [swift-evolution] Prohibit invisible characters in identifier names

The problem with depending on the IDE is that not everyone is using Xcode…or even a modern IDE. There are those that are using basic text editors, which must be considered as well. On Sun, Jun 26, 2016 at 9:25 PM Charlie Monroe via swift-evolution < swift-evolution@swift.org> wrote: > > > On Jun

Re: [swift-evolution] Prohibit invisible characters in identifier names

> On Jun 25, 2016, at 7:12 AM, David Sweeris wrote: > > >> On Jun 24, 2016, at 23:13, Charlie Monroe via swift-evolution >> wrote: >> >> BTW how far along with programming do you think you'd get without the >> knowledge of English? All

Re: [swift-evolution] Prohibit invisible characters in identifier names

> 25 June 2016 06:13 Charlie Monroe wrote: > >> >> On Jun 25, 2016, at 12:21 AM, Magnus Ahltorp wrote: >> >> As far as I can see, forcing the programmer to write identifiers in an >> only-ASCII language, and requiring that identifier names have to be

Re: [swift-evolution] Prohibit invisible characters in identifier names

> On Jun 24, 2016, at 08:27, Vladimir.S via swift-evolution > wrote: > > (Personally I really don't understand why we need anything than ASCII codes > for identifiers. This could solve all the problems with invisible > space/left-to-right-flags/complicated

Re: [swift-evolution] Prohibit invisible characters in identifier names

> On Jun 24, 2016, at 23:13, Charlie Monroe via swift-evolution > wrote: > > BTW how far along with programming do you think you'd get without the > knowledge of English? All libraries, SDKs use English identifiers. The > documentation is in English. For one to

Re: [swift-evolution] Prohibit invisible characters in identifier names

> On Jun 24, 2016, at 8:27 AM, Vladimir.S via swift-evolution > wrote: > > (Personally I really don't understand why we need anything than ASCII codes > for identifiers. This could solve all the problems with invisible > space/left-to-right-flags/complicated

Re: [swift-evolution] Prohibit invisible characters in identifier names

> On Jun 25, 2016, at 12:21 AM, Magnus Ahltorp wrote: > >> 21 June 2016 16:48 Vladimir.S via swift-evolution wrote: >> >> On 21.06.2016 7:37, Charlie Monroe via swift-evolution wrote: >>> On Jun 21, 2016, at 2:23 AM, Brent Royal-Gordon via swift-evolution

Re: [swift-evolution] Prohibit invisible characters in identifier names

On 24.06.2016 0:57, João Pinheiro via swift-evolution wrote: Indeed, the case shown in Josh's example was the motivation for this thread and will be solved by the proposal. The current discussion has been around whether it should be solved by ignoring invisible characters or prohibiting them

Re: [swift-evolution] Prohibit invisible characters in identifier names

Indeed, the case shown in Josh's example was the motivation for this thread and will be solved by the proposal. The current discussion has been around whether it should be solved by ignoring invisible characters or prohibiting them and explicitly highlighting them as an error. I originally

Re: [swift-evolution] Prohibit invisible characters in identifier names

+1 on this. Josh Wisenbaker’s example says enough. Yikes! > On Jun 23, 2016, at 3:18 PM, David Sweeris via swift-evolution > wrote: > > +1 > I didn't even know there were any invisible characters until this thread came > up. > > - Dave Sweeris > > On Jun 23, 2016,

Re: [swift-evolution] Prohibit invisible characters in identifier names

+1 I didn't even know there were any invisible characters until this thread came up. - Dave Sweeris > On Jun 23, 2016, at 15:13, Xiaodi Wu via swift-evolution > wrote: > >> On Thu, Jun 23, 2016 at 2:54 PM, João Pinheiro wrote: >> >> > On 23

Re: [swift-evolution] Prohibit invisible characters in identifier names

> On Jun 23, 2016, at 3:54 PM, João Pinheiro via swift-evolution > wrote: > >> On 23 Jun 2016, at 20:43, Xiaodi Wu > > wrote: >> That's cool, although my preferred solution would be more closely aligned >> with UAX

Re: [swift-evolution] Prohibit invisible characters in identifier names

> On 23 Jun 2016, at 20:43, Xiaodi Wu wrote: > That's cool, although my preferred solution would be more closely aligned > with UAX #31: overtly disallow the glyphs in Table 4 (instead of ignoring > them) except in the specific scenarios for ZWJ and ZWNJ identified in UAX

Re: [swift-evolution] Prohibit invisible characters in identifier names

On Thu, Jun 23, 2016 at 2:29 PM, João Pinheiro wrote: > > I think we're using terminology differently here. What you call > "character normalization" is what I'm calling canonicalization. NFC is > described in UAX #15 as "canonical decomposition followed by canonical >

Re: [swift-evolution] Prohibit invisible characters in identifier names

> I think we're using terminology differently here. What you call "character > normalization" is what I'm calling canonicalization. NFC is described in UAX > #15 as "canonical decomposition followed by canonical composition" and I'm > just using the word "canonicalization" because it's shorter.

Re: [swift-evolution] Prohibit invisible characters in identifier names

On Thu, Jun 23, 2016 at 12:56 PM, Xiaodi Wu wrote: > On Thu, Jun 23, 2016 at 12:41 PM, João Pinheiro > wrote: > >> There are two different issues here, individual character normalisation >> and identifier canonicalisation. NFC handles character

Re: [swift-evolution] Prohibit invisible characters in identifier names

I think this issue is bigger than that. As UAX #31 suggests, the most appropriate approach is canonicalizing identifiers by NFC, with specific treatment of ZWJ and ZWNJ by allowing them in three contexts, which will require thought as to how to implement. Given that there is a specifically

Re: [swift-evolution] Prohibit invisible characters in identifier names

> On Jun 23, 2016, at 9:17 AM, João Pinheiro via swift-evolution > wrote: > > >> On 21 Jun 2016, at 20:15, Xiaodi Wu via swift-evolution >> > wrote: >> >> On Tue, Jun 21, 2016 at 1:16 PM, Joe Groff

Re: [swift-evolution] Prohibit invisible characters in identifier names

I’m no unicode expert, but this sounds like the way to go to me. l8r Sean > On Jun 23, 2016, at 11:17 AM, João Pinheiro via swift-evolution > wrote: > > >> On 21 Jun 2016, at 20:15, Xiaodi Wu via swift-evolution >> wrote: >> >> On

Re: [swift-evolution] Prohibit invisible characters in identifier names

> On 21 Jun 2016, at 20:15, Xiaodi Wu via swift-evolution > wrote: > > On Tue, Jun 21, 2016 at 1:16 PM, Joe Groff > wrote: > Any discussion about this ought to start from UAX #31, the Unicode > consortium's

Re: [swift-evolution] Prohibit invisible characters in identifier names

On Jun 21, 2016, at 12:15 PM, Xiaodi Wu via swift-evolution wrote: > > Any discussion about this ought to start from UAX #31, the Unicode > consortium's recommendations on identifiers in programming languages: > > http://unicode.org/reports/tr31/

Re: [swift-evolution] Prohibit invisible characters in identifier names

On Tue, Jun 21, 2016 at 1:16 PM, Joe Groff wrote: > > > On Jun 21, 2016, at 8:47 AM, John McCall via swift-evolution < > swift-evolution@swift.org> wrote: > > > >> On Jun 20, 2016, at 7:07 PM, Xiaodi Wu wrote: > >> On Mon, Jun 20, 2016 at 8:58 PM, John

Re: [swift-evolution] Prohibit invisible characters in identifier names

> On Jun 21, 2016, at 8:47 AM, John McCall via swift-evolution > wrote: > >> On Jun 20, 2016, at 7:07 PM, Xiaodi Wu wrote: >> On Mon, Jun 20, 2016 at 8:58 PM, John McCall via swift-evolution >> wrote: >>> On Jun 20,

Re: [swift-evolution] Prohibit invisible characters in identifier names

> On Jun 20, 2016, at 7:07 PM, Xiaodi Wu wrote: > On Mon, Jun 20, 2016 at 8:58 PM, John McCall via swift-evolution > > wrote: >> On Jun 20, 2016, at 5:22 PM, Jordan Rose via swift-evolution >>

Re: [swift-evolution] Prohibit invisible characters in identifier names

On 21.06.2016 7:37, Charlie Monroe via swift-evolution wrote: On Jun 21, 2016, at 2:23 AM, Brent Royal-Gordon via swift-evolution wrote: Perhaps stupid but: why was Swift designed to accept most Unicode characters in identifier names? Wouldn’t it be simpler to go

Re: [swift-evolution] Prohibit invisible characters in identifier names

> On Jun 21, 2016, at 2:23 AM, Brent Royal-Gordon via swift-evolution > wrote: > >> Perhaps stupid but: why was Swift designed to accept most Unicode characters >> in identifier names? Wouldn’t it be simpler to go back to a model where only >> standard ascii

Re: [swift-evolution] Prohibit invisible characters in identifier names

On Mon, Jun 20, 2016 at 8:58 PM, John McCall via swift-evolution < swift-evolution@swift.org> wrote: > On Jun 20, 2016, at 5:22 PM, Jordan Rose via swift-evolution < > swift-evolution@swift.org> wrote: > IIRC, some languages *require* zero-width joiners (though not zero-width > spaces, which are

Re: [swift-evolution] Prohibit invisible characters in identifier names

> On Jun 20, 2016, at 5:22 PM, Jordan Rose via swift-evolution > wrote: > IIRC, some languages require zero-width joiners (though not zero-width > spaces, which are distinct) to properly encode some of their characters. I'd > be very leery of having Swift land on a

Re: [swift-evolution] Prohibit invisible characters in identifier names

> Perhaps stupid but: why was Swift designed to accept most Unicode characters > in identifier names? Wouldn’t it be simpler to go back to a model where only > standard ascii characters are accepted in identifier names? I assume it has something to do with the fact that 94.6% of the world's

Re: [swift-evolution] Prohibit invisible characters in identifier names

IIRC, some languages require zero-width joiners (though not zero-width spaces, which are distinct) to properly encode some of their characters. I'd be very leery of having Swift land on a model where identifiers can be used with some languages and not others; that smacks of ethnocentrism.

Re: [swift-evolution] Prohibit invisible characters in identifier names

Perhaps stupid but: why was Swift designed to accept most Unicode characters in identifier names? Wouldn’t it be simpler to go back to a model where only standard ascii characters are accepted in identifier names? > On 20 Jun 2016, at 20:26, Vladimir.S via swift-evolution >

Re: [swift-evolution] Prohibit invisible characters in identifier names

On Mon, Jun 20, 2016 at 5:20 PM, João Pinheiro wrote: > > > On 20 Jun 2016, at 22:44, Dave Abrahams via swift-evolution < > swift-evolution@swift.org> wrote: > > I'm a little concerned about cases like these: > > > > 1D6CE ; 0076 ; MA # ( 훎 → v )

Re: [swift-evolution] Prohibit invisible characters in identifier names

On Mon, Jun 20, 2016 at 4:44 PM, Dave Abrahams via swift-evolution < swift-evolution@swift.org> wrote: > > on Mon Jun 20 2016, Xiaodi Wu wrote: > > > On Mon, Jun 20, 2016 at 2:42 PM, João Pinheiro > > wrote: > > > >> I agree that treating

Re: [swift-evolution] Prohibit invisible characters in identifier names

> On 20 Jun 2016, at 22:44, Dave Abrahams via swift-evolution > wrote: > I'm a little concerned about cases like these: > > 1D6CE ; 0076 ; MA # ( 훎 → v ) MATHEMATICAL BOLD SMALL NU → LATIN > SMALL LETTER V # →ν→ > 1D6D2 ; 0070 ; MA # ( 훒 →

Re: [swift-evolution] Prohibit invisible characters in identifier names

on Mon Jun 20 2016, Xiaodi Wu wrote: > On Mon, Jun 20, 2016 at 2:42 PM, João Pinheiro > wrote: > >> I agree that treating zero-width spaces as non-existent would be a >> possible solution, but I think it would make more sense to consider it as

Re: [swift-evolution] Prohibit invisible characters in identifier names

On 20 Jun 2016, at 21:07, Xiaodi Wu wrote: > > On Mon, Jun 20, 2016 at 2:42 PM, João Pinheiro > wrote: > I agree that treating zero-width spaces as non-existent would be a possible > solution, but I think it would make

Re: [swift-evolution] Prohibit invisible characters in identifier names

On Mon, Jun 20, 2016 at 2:42 PM, João Pinheiro wrote: > I agree that treating zero-width spaces as non-existent would be a > possible solution, but I think it would make more sense to consider it as > white space and thus not admissible in identifier names. > If you treat

Re: [swift-evolution] Prohibit invisible characters in identifier names

I agree that treating zero-width spaces as non-existent would be a possible solution, but I think it would make more sense to consider it as white space and thus not admissible in identifier names. I'm not sure of what the best way to handle left-to-right and right-to-left markers would be.

Re: [swift-evolution] Prohibit invisible characters in identifier names

On Mon, Jun 20, 2016 at 2:17 PM, João Pinheiro wrote: > Nice feature in the IBM Swift Sandbox. Xcode doesn't display zero-width > spaces either so the identifier names look exactly the same. > > The issue with left-to-right and right-to-left markers is interesting and

Re: [swift-evolution] Prohibit invisible characters in identifier names

Very interesting. Btw, IBM Swift Sandbox shows these spaces: https://swiftlang.ng.bluemix.net/ But my mail client does not - i.e. I saw exactly the same "test"&"abc" Also, I read about some issues with left-to-right and right-to-left markers that also somehow change the actual text of source -

Re: [swift-evolution] Prohibit invisible characters in identifier names

+1 l8r Sean > On Jun 20, 2016, at 12:51 PM, João Pinheiro via swift-evolution > wrote: > > Recently there has been a screenshot going around Twitter about C++ allowing > zero-width spaces in variable names. Swift also suffers from this problem > which can be

[swift-evolution] Prohibit invisible characters in identifier names

Recently there has been a screenshot going around Twitter about C++ allowing zero-width spaces in variable names. Swift also suffers from this problem which can be abused to create ambiguous, misleading, and potentially obfuscate nefarious code. I would like to propose a change to prohibit the