Re: Bidi paragraph direction in terminal emulators BiDi in terminal emulators

2019-02-07 Thread Eli Zaretskii via Unicode
> From: Egmont Koblinger > Date: Thu, 7 Feb 2019 19:01:33 +0100 > Cc: Richard Wordingham , > unicode Unicode Discussion > > On Thu, Feb 7, 2019 at 6:53 PM Eli Zaretskii wrote: > > > No, it needs no interaction. Unless the regexp doesn't work for you, > > which you should then report as

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-07 Thread Egmont Koblinger via Unicode
Hi Philippe, On Thu, Feb 7, 2019 at 3:21 PM Philippe Verdy wrote: > "Rules" are not formally written, they are just a sense of best practices. When it comes to BiDi in terminals, I haven't seen anything that I consider reasonably okay, let alone "best practice". It's a mess. That's why I decide

Re: Bidi paragraph direction in terminal emulators BiDi in terminal emulators

2019-02-07 Thread Egmont Koblinger via Unicode
On Thu, Feb 7, 2019 at 6:53 PM Eli Zaretskii wrote: > No, it needs no interaction. Unless the regexp doesn't work for you, > which you should then report as a bug in Emacs. Do you mean you aim to maintain a regex that matches everyone's prompt in the world, without a significant amount of false

Re: Bidi paragraph direction in terminal emulators BiDi in terminal emulators

2019-02-07 Thread Eli Zaretskii via Unicode
> From: Egmont Koblinger > Date: Thu, 7 Feb 2019 18:20:02 +0100 > Cc: Richard Wordingham , > unicode Unicode Discussion > > > It uses a regular expression, see term-prompt-regexp. > > So, it's not automatic, needs user interaction No, it needs no interaction. Unless the regexp doesn't

Re: Bidi paragraph direction in terminal emulators

2019-02-07 Thread Egmont Koblinger via Unicode
On Thu, Feb 7, 2019 at 6:33 PM Eli Zaretskii wrote: > Well, let's just say that Emacs uses the HL1 rule, and determines the > base direction for the entire chunk of text between empty lines. Exactly! Now it's my turn to figure out how to add this behavior to terminals, preferably stopping befor

Re: Bidi paragraph direction in terminal emulators

2019-02-07 Thread Eli Zaretskii via Unicode
> From: Egmont Koblinger > Date: Thu, 7 Feb 2019 18:12:37 +0100 > Cc: Richard Wordingham , > unicode Unicode Discussion > > I believe it's not my mental model that's weird, but your use of > terminology that doesn't match UBA's that confused me. Well, let's just say that Emacs uses the H

Re: Bidi paragraph direction in terminal emulators BiDi in terminal emulators

2019-02-07 Thread Egmont Koblinger via Unicode
Hi, On Thu, Feb 7, 2019 at 3:27 PM Eli Zaretskii wrote: > It uses a regular expression, see term-prompt-regexp. So, it's not automatic, needs user interaction, and for that reason, may not have worked for me. (I have other weird things in my prompt, like 256-color sequences that Emacs didn't re

Re: Bidi paragraph direction in terminal emulators

2019-02-07 Thread Egmont Koblinger via Unicode
On Thu, Feb 7, 2019 at 3:14 PM Eli Zaretskii wrote: > Not a bug, a feature. Emacs doesn't remove the bidi controls from > display (that's another deviation allowed by the UBA, see section > 5.2). On GUI displays, these controls are displayed as thin 1-pixel > spaces, but on text-mode terminals

Re: Two more ellispis-type interpunctations: ?.. and !..

2019-02-07 Thread Serik Serikbay via Unicode
Khakass language is much close to Kyrgyz .. On Thu, Feb 7, 2019 at 8:54 PM "Jörg Knappen" via Unicode < unicode@unicode.org> wrote: > While working on a corpus of Kyrgyz language, a Turkic language written in > the Cyrilic script, > I encountered two ellipsis-type

Re: Bidi paragraph direction in terminal emulators BiDi in terminal emulators

2019-02-07 Thread Eli Zaretskii via Unicode
> Date: Thu, 7 Feb 2019 00:45:55 +0100 > Cc: unicode Unicode Discussion > From: Egmont Koblinger via Unicode > > > Not necessarily. One could allow the first strong character in the > > prompt to determine the paragraph directions > > How does Emacs know what&

Re: Bidi paragraph direction in terminal emulators BiDi in terminal emulators

2019-02-07 Thread Eli Zaretskii via Unicode
> Date: Wed, 6 Feb 2019 23:32:43 + > From: Richard Wordingham via Unicode > > > You define paragraphs as emptyline-separated blocks on which you > > perform autodetection of the paragraph direction. This is great! As > > I've mentioned, I'd love to ha

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-07 Thread Philippe Verdy via Unicode
Le jeu. 7 févr. 2019 à 13:29, Egmont Koblinger a écrit : > Hi Philippe, > > > There's some rules for correct display including with Bidi: > > In what sense are these "rules"? Where are these written, in what kind > of specification or existing practice? > "Rules" are not formally written, they a

Re: Bidi paragraph direction in terminal emulators

2019-02-07 Thread Eli Zaretskii via Unicode
> From: Egmont Koblinger > Date: Wed, 6 Feb 2019 22:01:59 +0100 > Cc: Richard Wordingham , unicode@unicode.org > > - Emacs running in a terminal shows an underscore wherever there's a > BiDi control in the source file – while the graphical one doesn't. > This looks like a simple bug to me, right?

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-07 Thread Egmont Koblinger via Unicode
Hi Philippe, > There's some rules for correct display including with Bidi: In what sense are these "rules"? Where are these written, in what kind of specification or existing practice? > - Separate paragraphs that need a different default Bidi by double newlines > (to force a hard break) There

Re: Bidi paragraph direction in terminal emulators BiDi in terminal emulators

2019-02-07 Thread Richard Wordingham via Unicode
On Thu, 7 Feb 2019 00:45:55 +0100 Egmont Koblinger via Unicode wrote: > Hi Richard, > > > Not necessarily. One could allow the first strong character in the > > prompt to determine the paragraph directions > > How does Emacs know what's a prompt? How can it te

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-06 Thread Philippe Verdy via Unicode
d one, but in most > terminals, a newline is not such a control function.) > > Anyway, please also see my previous email; I hope that clarifies a lot > for you, too. > > > cheers, > egmont > > On Tue, Feb 5, 2019 at 5:53 PM Philippe Verdy via Unicode > wrote: > >

Re: Bidi paragraph direction in terminal emulators BiDi in terminal emulators

2019-02-06 Thread Egmont Koblinger via Unicode
Hi Richard, > Not necessarily. One could allow the first strong character in the > prompt to determine the paragraph directions How does Emacs know what's a prompt? How can it tell it from the previous and next command's output? Whatever it does to know where the prompt is, can it be made into

Re: Bidi paragraph direction in terminal emulators BiDi in terminal emulators

2019-02-06 Thread Richard Wordingham via Unicode
On Wed, 6 Feb 2019 22:01:59 +0100 Egmont Koblinger via Unicode wrote: > Hi Eli, > > (I'm getting lost where to reply, and how the subject gets mangled and > the thread split into different ones.) > > > I've thought about it a lot, experimented with Emacs's

Re: Bidi paragraph direction in terminal emulators BiDi in terminal emulators

2019-02-06 Thread Egmont Koblinger via Unicode
Hi, I was loose with my terminology once again, which is not a wise thing when you're trying to clarify misunderstandings :) > But once you have > decided on a direction, each _line_ within that data is passed > separately to the BiDi algorithm to get reshuffled; this is what Emacs > does, this i

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-06 Thread Egmont Koblinger via Unicode
nctions turn them back to hard one, but in most terminals, a newline is not such a control function.) Anyway, please also see my previous email; I hope that clarifies a lot for you, too. cheers, egmont On Tue, Feb 5, 2019 at 5:53 PM Philippe Verdy via Unicode wrote: > > I think that bef

Re: Bidi paragraph direction in terminal emulators BiDi in terminal emulators

2019-02-06 Thread Egmont Koblinger via Unicode
27;t know where they stand now, I'll investigate if it's important, but I don't think it is.) Luckily both Emacs and my specification shuffles the contents separately within both lines (using LTR paragraph for both lines, as it's guessed from the union of them), resulting in the d

Re: mildly OT from bidi - curious email

2019-02-06 Thread Arthur Reutenauer via Unicode
On Wed, Feb 06, 2019 at 02:30:24PM +, Julian Bradfield via Unicode wrote: > So far, so common. The curious thing is that the (entirely > ASCII) company name was enclosed in a left-to-right direction, thus: > > Subject: Your Aaa Ltd receipt [#-] > > w

mildly OT from bidi - curious email

2019-02-06 Thread Julian Bradfield via Unicode
The current bidi discussion prompts me to post a curiosity I received today. I ordered something from a (UK) company, and the payment receipt came via Stripe. So far, so common. The curious thing is that the (entirely ASCII) company name was enclosed in a left-to-right direction, thus: Subject: Y

Re: Encoding italic

2019-02-05 Thread Richard Wordingham via Unicode
On Tue, 5 Feb 2019 16:01:41 + Andrew West via Unicode wrote: > You would > have to first convert any text to be italicized to NFD, then apply > VS14 to each non-combining character. This alone would make a VS > solution unacceptable in my opinion. What is so unacceptable about

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-05 Thread Philippe Verdy via Unicode
cimal or octal digits. There's a wide variety of escaping mechanisms used by various higher-layer protocols (including transport protocols or encoding syntaxes used just below the plain-text layer, in a lower layer than the transport protocol layer). Le lun. 4 févr. 2019 à 21:46, Eli Zaretskii

Re: Bidi paragraph direction in terminal emulators

2019-02-05 Thread Eli Zaretskii via Unicode
> From: Egmont Koblinger > Date: Tue, 5 Feb 2019 02:28:50 +0100 > Cc: unicode@unicode.org > > I have to admit, I'm not an Emacs user, I only have some vague ideas > how powerful a tool it is. But in its very core I still believe it's a > text editor – is it fair to say this? It could be used for

Re: Encoding italic

2019-02-05 Thread Andrew West via Unicode
On Tue, 5 Feb 2019 at 15:34, wjgo_10...@btinternet.com via Unicode wrote: > > italic version of a glyph in plain text, including a suggestion of to > which characters it could apply, would test whether such a proposal > would be accepted to go into the Document Register for

Re: Bidi paragraph direction in terminal emulators

2019-02-05 Thread Eli Zaretskii via Unicode
> From: Egmont Koblinger > Date: Tue, 5 Feb 2019 01:32:34 +0100 > Cc: unicode@unicode.org > > On the other hand, it's not unreasonable for higher level stuff (e.g. > shell scripts, or tools like "zip") to use such control characters. Yes, but most of them won't ever do that. > > No, this simple

Re: Bidi paragraph direction in terminal emulators BiDi in terminal emulators

2019-02-05 Thread Eli Zaretskii via Unicode
> Date: Tue, 5 Feb 2019 00:05:47 + > From: Richard Wordingham via Unicode > > > > Actually, UAX#9 defines "paragraph" as the chunk of text delimited > > > by paragraph separator characters. This means characters whose bidi > > > category i

Re: Bidi paragraph direction in terminal emulators BiDi in terminal emulators)

2019-02-05 Thread Eli Zaretskii via Unicode
> From: Egmont Koblinger > Date: Tue, 5 Feb 2019 00:08:10 +0100 > Cc: unicode@unicode.org > > every single newline character starts a new paragraph. The result of > printf "Hello\nWorld\n" > world.txt > is a text file consisting of two paragraphs, with 5 characters in each. > Correct? Yes. > >

Re: Encoding italic

2019-02-05 Thread wjgo_10...@btinternet.com via Unicode
James Kass wrote: William’s suggestion of floating a proposal for handling italics with VS14 might be an example of the old saying about “putting the cart before the horse”. Well, a proposal just about using VS14 to indicate a request for an italic version of a glyph in plain text, including

Re: Encoding italic

2019-02-05 Thread James Kass via Unicode
William Overington wrote, > Well, a proposal just about using VS14 to indicate a request for an > italic version of a glyph in plain text, including a suggestion of to > which characters it could apply, would test whether such a proposal > would be accepted to go into the Document Register for

Re: Ancient Greek apostrophe marking elision

2019-02-05 Thread James Tauber via Unicode
On Tue, Feb 5, 2019 at 12:23 AM James Kass via Unicode wrote: > Text a man has JOINED together, let not algorithm put asunder. > I was hoping so much that ὃ οὖν ὁ θεὸς συνέζευξεν ἄνθρωπος μὴ χωριζέτω would have an apostrophe but alas no.

Re: Encoding italic

2019-02-04 Thread James Kass via Unicode
Philippe Verdy responded to William Overington, > the proposal would contradict the goals of variation selectors and would > pollute ther variation sequences registry (possibly even creating conflicts). > And if we admit it for italics, than another VSn will be dedicated to bold, > and anoth

Re: Ancient Greek apostrophe marking elision

2019-02-04 Thread James Kass via Unicode
On 2019-01-28 8:58 PM, Richard Wordingham wrote: > On Mon, 28 Jan 2019 03:48:52 + > James Kass via Unicode wrote: > >> It’s been said that the text segmentation rules seem over-complicated >> and are probably non-trivial to implement properly.  I tried your >>

Re: Proposal for BiDi in terminal emulators

2019-02-04 Thread Eli Zaretskii via Unicode
> Date: Mon, 4 Feb 2019 21:00:55 + > From: Richard Wordingham via Unicode > > > The definition is trivial: the order of characters on > > display, from left to right. The only possible reason to split hairs > > here could be when some characters don't app

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-04 Thread Richard Wordingham via Unicode
On Mon, 4 Feb 2019 22:27:39 +0100 Egmont Koblinger via Unicode wrote: > Hi Richard, > > > The concept appears to exist in the form of the fields of the > > fifth edition of ECMA-48. Have you digested this ambitious > > standard? > > To be honest: No, I haven&#x

Re: Bidi paragraph direction in terminal emulators BiDi in terminal emulators)

2019-02-04 Thread Egmont Koblinger via Unicode
Hi Eli, > IME, this is a grave mistake. I hope I explained why; it is now up to > you to decide what to do about that. Let me share one more thought. I have to admit, I'm not an Emacs user, I only have some vague ideas how powerful a tool it is. But in its very core I still believe it's a text

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-04 Thread Egmont Koblinger via Unicode
Hi Eli, > I think it's unreasonable and impractical to expect 'echo', 'cat', and > its ilk to emit bidi controls (or any other controls) to force > paragraph direction. For starters, they won't know what direction to > force, because they don't understand the text they are processing. I agree, i

Re: Bidi paragraph direction in terminal emulators BiDi in terminal emulators)

2019-02-04 Thread Richard Wordingham via Unicode
On Tue, 5 Feb 2019 00:08:10 +0100 Egmont Koblinger via Unicode wrote: > Hi Eli, > > > Actually, UAX#9 defines "paragraph" as the chunk of text delimited > > by paragraph separator characters. This means characters whose bidi > > category is B, which includes

Re: Bidi paragraph direction in terminal emulators BiDi in terminal emulators)

2019-02-04 Thread Egmont Koblinger via Unicode
Hi Eli, > Actually, UAX#9 defines "paragraph" as the chunk of text delimited by > paragraph separator characters. This means characters whose bidi > category is B, which includes Newline, the CR-LF pair on Windows, > U+0085 NEL, and U+2029 PARAGRAPH SEPARATOR. Indeed, this was an oversight on my

Re: Proposal for BiDi in terminal emulators

2019-02-04 Thread Egmont Koblinger via Unicode
Hi, > To me, 'visual order' means in the dominant order of the script. This is not a definition I've come across anywhere else, nor matches my intuition of "visual order" : the exact visual order (recursive definition, yay!) of how you see the glyphs being displayed in the row. > So, > if one ta

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-04 Thread Richard Wordingham via Unicode
On Mon, 04 Feb 2019 22:39:07 +0200 Eli Zaretskii via Unicode wrote: > > Date: Mon, 4 Feb 2019 19:45:13 + > > From: Richard Wordingham via Unicode > > > > Yes. If one has a text composed of LTR and RTL paragraphs, one has > > to choose how far apart their

Re: Proposal for BiDi in terminal emulators

2019-02-04 Thread Asmus Freytag via Unicode
On 2/4/2019 1:00 PM, Richard Wordingham via Unicode wrote: To me, 'visual order' means in the dominant order of the script. Visual order is a term of art, meaning the characters are ordered in memory in the same order as they are displa

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-04 Thread Egmont Koblinger via Unicode
> > Yes. If one has a text composed of LTR and RTL paragraphs, one has to > > choose how far apart their starting margins are. I think that could > > get complicated for plain text if the terminal has unbounded width. > > But no real-life terminal does. The width is always bounded. Allegedly th

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-04 Thread Egmont Koblinger via Unicode
Hi Richard, > That split is wrong if you want the non-HTML text to lay out reasonably > well in anything but a higher order protocol forcing RTL. You need to > it split as: > > lorem ipsum ABC > <[ DEF foobar Okay, so you should use LRMs or other similar tricks when wrapping a human-perceived pa

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-04 Thread Egmont Koblinger via Unicode
Hi Richard, > The concept appears to exist in the form of the fields of the > fifth edition of ECMA-48. Have you digested this ambitious standard? To be honest: No, I haven't. And I have no idea what those "fields" are. I spent (read: wasted) way too much time studying ECMA TR/53 to get to unde

Re: Proposal for BiDi in terminal emulators

2019-02-04 Thread Richard Wordingham via Unicode
On Sun, 3 Feb 2019 20:50:03 + Richard Wordingham via Unicode wrote: > On Sun, 03 Feb 2019 20:07:51 +0200 > Eli Zaretskii via Unicode wrote: > Which is why I try to remember to issue the emacs command 'M-x shell' > command and issue grep commands from the buffer

Re: Does "endian-ness" apply to UTF-8 characters that use multiple bytes?

2019-02-04 Thread Doug Ewell via Unicode
http://www.unicode.org/faq/utf_bom.html#utf8-2 -- Doug Ewell | Thornton, CO, US | ewellic.org

Re: Proposal for BiDi in terminal emulators

2019-02-04 Thread Richard Wordingham via Unicode
On Sun, 03 Feb 2019 18:03:37 +0200 Eli Zaretskii via Unicode wrote: > > Date: Sun, 3 Feb 2019 03:02:13 +0100 > > Cc: unicode@unicode.org > > From: Egmont Koblinger via Unicode > > > > > All I am saying is that your proposal should define what it means >

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-04 Thread Eli Zaretskii via Unicode
> Date: Mon, 4 Feb 2019 19:45:13 + > From: Richard Wordingham via Unicode > > Yes. If one has a text composed of LTR and RTL paragraphs, one has to > choose how far apart their starting margins are. I think that could > get complicated for plain text if the terminal h

Re: Does "endian-ness" apply to UTF-8 characters that use multiple bytes?

2019-02-04 Thread James Tauber via Unicode
, endian-ness doesn't effect the ordering of each 16-bit unit that makes up the pair, only the two bytes within each of the units. James On Mon, Feb 4, 2019 at 2:25 PM Costello, Roger L. via Unicode < unicode@unicode.org> wrote: > Hello Unicode Experts! > > As I understand it, en

Re: Does "endian-ness" apply to UTF-8 characters that use multiple bytes?

2019-02-04 Thread Clive Hohberger via Unicode
the Ultracode Bar Code Symbology* *2017 Label Industry Global Award for Innovation* On Mon, Feb 4, 2019 at 1:29 PM Asmus Freytag via Unicode < unicode@unicode.org> wrote: > On 2/4/2019 11:21 AM, Costello, Roger L. via Unicode wrote: > > Hello Unicode Experts! > > As I under

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-04 Thread Richard Wordingham via Unicode
On Mon, 04 Feb 2019 18:53:22 +0200 Eli Zaretskii via Unicode wrote: > Date: Mon, 4 Feb 2019 01:19:21 + > From: Richard Wordingham via Unicode >> If you look at it in Notepad, all >> lines will be LTR or all lines will be RTL. > That's because Notepad implemen

Re: Does "endian-ness" apply to UTF-8 characters that use multiple bytes?

2019-02-04 Thread Asmus Freytag via Unicode
On 2/4/2019 11:21 AM, Costello, Roger L. via Unicode wrote: Hello Unicode Experts! As I understand it, endian-ness applies to multi-byte words. Endian-ness does not apply to ASCII characters because each character is a single byte. Endian-ness does apply to UTF

Does "endian-ness" apply to UTF-8 characters that use multiple bytes?

2019-02-04 Thread Costello, Roger L. via Unicode
Hello Unicode Experts! As I understand it, endian-ness applies to multi-byte words. Endian-ness does not apply to ASCII characters because each character is a single byte. Endian-ness does apply to UTF-16BE (Big-Endian), UTF-16LE (Little-Endian), UTF-32BE and UTF32-LE because each character us

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-04 Thread Eli Zaretskii via Unicode
> Date: Mon, 4 Feb 2019 01:19:21 + > From: Richard Wordingham via Unicode > > On Sun, 03 Feb 2019 19:50:50 +0200 > Eli Zaretskii via Unicode wrote: > > > Do you see how this is carefully formatted to avoid overflowing an > > 80-column line of a typical

Re: Bidi paragraph direction in terminal emulators BiDi in terminal emulators)

2019-02-04 Thread Eli Zaretskii via Unicode
> From: Egmont Koblinger > Date: Mon, 4 Feb 2019 00:36:23 +0100 > Cc: unicode@unicode.org > > The Unicode BiDi algorithm states that it operates on paragraphs of > text, and leaves it up to a higher protocol to define what a paragraph > exactly is. > > What's the definition of "paragraph" in the

Re: Proposal for BiDi in terminal emulators

2019-02-04 Thread Eli Zaretskii via Unicode
> Date: Mon, 04 Feb 2019 05:25:43 +0200 > Cc: unicode@unicode.org > From: Eli Zaretskii via Unicode > > Try customizing scroll-conservatively, it sounds like you want that. Ignore me: I misunderstood what you were looking for. You are right: Emacs doesn't support such scrolling method.

Re: Proposal for BiDi in terminal emulators

2019-02-03 Thread Eli Zaretskii via Unicode
> Date: Sun, 3 Feb 2019 20:35:18 + > From: Richard Wordingham via Unicode > > > What is "screen overwriting" in this context? > > When instead of adding lines to the bottom, new lines are added on top > of and replace existing lines. I prefer the scro

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-03 Thread Richard Wordingham via Unicode
On Mon, 4 Feb 2019 00:36:23 +0100 Egmont Koblinger via Unicode wrote: > Now, back to terminals. > > The smallest possible viable definition of a "paragraph" in terminal > emulators is stuff between one newline and the next one. > > It would require a h

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-03 Thread Richard Wordingham via Unicode
On Mon, 4 Feb 2019 00:36:23 +0100 Egmont Koblinger via Unicode wrote: > I wish to store and deliver the following text, as it's layed out here > in logical order. That is, the order as the bytes appear in the text > file, as I typed them from the keyboard, is laid out here strict

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-03 Thread Richard Wordingham via Unicode
On Sun, 03 Feb 2019 19:50:50 +0200 Eli Zaretskii via Unicode wrote: > Do you see how this is carefully formatted to avoid overflowing an > 80-column line of a typical terminal? Now suppose this is translated > into a RTL language, which causes the Copyright line to start with a &

Re: Proposal for BiDi in terminal emulators

2019-02-03 Thread Richard Wordingham via Unicode
On Sun, 03 Feb 2019 18:13:06 +0200 Eli Zaretskii via Unicode wrote: > Actually, you pass the characters to be shaped in logical order, and > then display the produced grapheme clusters in visual order. Some early systems supporting computerised Hebrew script did pass characters in left-to

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-03 Thread Egmont Koblinger via Unicode
Hi Eli, (I'm responding in multiple emails.) The Unicode BiDi algorithm states that it operates on paragraphs of text, and leaves it up to a higher protocol to define what a paragraph exactly is. What's the definition of "paragraph" in the context of plain text files? I don't think there's a s

Re: Proposal for BiDi in terminal emulators

2019-02-03 Thread Richard Wordingham via Unicode
On Sun, 03 Feb 2019 20:07:51 +0200 Eli Zaretskii via Unicode wrote: > > Date: Sun, 3 Feb 2019 17:45:06 + > > From: Richard Wordingham via Unicode > > > > > > So, what do you recommend I run grep from for Hebrew or Tai > > > > Lue? > >

Re: Proposal for BiDi in terminal emulators

2019-02-03 Thread Richard Wordingham via Unicode
On Sun, 03 Feb 2019 18:05:49 +0200 Eli Zaretskii via Unicode wrote: > > Date: Sat, 2 Feb 2019 21:49:40 + > > From: Richard Wordingham via Unicode > > > > Eli will probably tell me I'm behind the times, but there are a few > > places where a Gnome-terminal

Re: Proposal for BiDi in terminal emulators

2019-02-03 Thread Eli Zaretskii via Unicode
> Date: Sun, 3 Feb 2019 17:45:06 + > From: Richard Wordingham via Unicode > > > > So, what do you recommend I run grep from for Hebrew or Tai Lue? > > > > Inside Emacs, of course: "M-x grep RET" etc. > > That assumes you like using bindings

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-03 Thread Eli Zaretskii via Unicode
> From: Egmont Koblinger > Date: Sun, 3 Feb 2019 17:54:25 +0100 > Cc: unicode@unicode.org > > I'm arguing, although my reasons are not rock solid, that IMHO the > default should be the strict direction as set by SCP, without > autodetection. I think it's unreasonable and impractical to expect 'e

Re: Proposal for BiDi in terminal emulators

2019-02-03 Thread Richard Wordingham via Unicode
On Sun, 03 Feb 2019 18:14:53 +0200 Eli Zaretskii via Unicode wrote: > > Date: Sun, 3 Feb 2019 02:43:06 + > > Cc: Kent Karlsson > > From: Richard Wordingham via Unicode > > > > So, what do you recommend I run grep from for Hebrew or Tai Lue? > > In

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-03 Thread Egmont Koblinger via Unicode
Hi Eli, > The document cited at the beginning of the parent thread states that > "simple" text-mode utilities, such as 'echo', 'cat', 'ls' etc. should > use the "implicit" mode of bidi reordering, with automatic guessing of > the base paragraph direction. Not exactly. I take the SCP escape sequen

Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-03 Thread Eli Zaretskii via Unicode
> Date: Sun, 03 Feb 2019 18:10:15 +0200 > Cc: richard.wording...@ntlworld.com, unicode@unicode.org > From: Eli Zaretskii via Unicode > > I think there are hard problems even for such "simple" utilities, and > I will start a separate thread about this. I think we

Re: Proposal for BiDi in terminal emulators

2019-02-03 Thread Eli Zaretskii via Unicode
> Date: Sun, 3 Feb 2019 02:43:06 + > Cc: Kent Karlsson > From: Richard Wordingham via Unicode > > So, what do you recommend I run grep from for Hebrew or Tai Lue? Inside Emacs, of course: "M-x grep RET" etc.

Re: Proposal for BiDi in terminal emulators

2019-02-03 Thread Eli Zaretskii via Unicode
> Date: Sun, 3 Feb 2019 01:30:26 + > From: Richard Wordingham via Unicode > > Shaping for RTL scripts happens on strings stored in logical order. > These are then laid out right to left, though the dominant usage of > the term 'advance width' for right-t

Re: Proposal for BiDi in terminal emulators

2019-02-03 Thread Eli Zaretskii via Unicode
> Date: Sat, 2 Feb 2019 23:02:10 +0100 > Cc: unicode@unicode.org > From: Egmont Koblinger via Unicode > > On top of this, I make the clarification that combining marks need to > be reordered to be sent out to the terminal emulator _after_ their > base letter That is true

Re: Proposal for BiDi in terminal emulators

2019-02-03 Thread Eli Zaretskii via Unicode
> Date: Sat, 2 Feb 2019 21:49:40 + > From: Richard Wordingham via Unicode > > Eli will probably tell me I'm behind the times, but there are a few > places where a Gnome-terminal is better than an Emacs GUI window. One > is colour highlighting of text found by gre

Re: Proposal for BiDi in terminal emulators

2019-02-03 Thread Eli Zaretskii via Unicode
> Date: Sun, 3 Feb 2019 03:02:13 +0100 > Cc: unicode@unicode.org > From: Egmont Koblinger via Unicode > > > All I am saying is that your proposal should define what it means by > > visual order. > > Are you nitpicking on me not giving a precise definition on the &g

Re: Proposal for BiDi in terminal emulators

2019-02-02 Thread Richard Wordingham via Unicode
On Sun, 03 Feb 2019 02:01:18 +0100 Kent Karlsson via Unicode wrote: > Den 2019-02-02 16:12, skrev "Richard Wordingham via Unicode" > : > > Doesn't Jerusalem in biblical Hebrew sometime have 3 marks below the > > lamedh? The depth then is the maximum dep

Re: Proposal for BiDi in terminal emulators

2019-02-02 Thread Egmont Koblinger via Unicode
Hi Richard, On Sun, Feb 3, 2019 at 2:32 AM Richard Wordingham via Unicode wrote: > That first reference doesn't even use the word 'visual'. The Unicode BiDi algorithm does speak about "visual positions for display", "reordering for display" etc. > All

Re: Proposal for BiDi in terminal emulators

2019-02-02 Thread Richard Wordingham via Unicode
On Sat, 2 Feb 2019 23:02:10 +0100 Egmont Koblinger via Unicode wrote: > Hi Richard, > > On Sat, Feb 2, 2019 at 9:57 PM Richard Wordingham > wrote: > > > Seriously, you need to give a definition of 'visual order' for this > > context. Not everyone shar

Re: Proposal for BiDi in terminal emulators

2019-02-02 Thread Kent Karlsson via Unicode
Den 2019-02-02 16:12, skrev "Richard Wordingham via Unicode" : > On Sat, 02 Feb 2019 14:01:46 +0100 > Kent Karlsson via Unicode wrote: > >> Well, I guess you may need to put some (practical) limit to the number >> of non-spacing marks (like max two above + max o

Re: Proposal for BiDi in terminal emulators

2019-02-02 Thread Benjamin Riefenstahl via Unicode
Hi Richard, > Benjamin Riefenstahl wrote: >> the severe limitations of that environment. Richard Wordingham writes: > Eli will probably tell me I'm behind the times, but there are a few > places where a Gnome-terminal is better than an Emacs GUI window. One > is colour highlighting of text foun

Re: Proposal for BiDi in terminal emulators

2019-02-02 Thread Egmont Koblinger via Unicode
Hi Richard, > My main interest in this, though, is in improving the general run of > Indic terminal cell editors. If we can get Gnome-terminal working for > Kharoshthi, things should improve for LTR Indic. Even working on the > false assumption that Indic scripts are like Devanagari would be an

Re: Proposal for BiDi in terminal emulators

2019-02-02 Thread Egmont Koblinger via Unicode
Hi Richard, On Sat, Feb 2, 2019 at 9:57 PM Richard Wordingham wrote: > Seriously, you need to give a definition of 'visual order' for this > context. Not everyone shares your chiralist view. When I look at the Unicode BiDi algorithm, or go to an online demo at https://unicode.org/cldr/utility/

Re: Proposal for BiDi in terminal emulators

2019-02-02 Thread Richard Wordingham via Unicode
On Sat, 02 Feb 2019 20:58:06 +0100 Benjamin Riefenstahl via Unicode wrote: > Hi Egmont, hi all, > > > This is a interesting discussion here. If only because I would have > thought that there is only minimal interest by the actual target > audience in supporting these scr

Re: Proposal for BiDi in terminal emulators

2019-02-02 Thread Richard Wordingham via Unicode
On Sat, 2 Feb 2019 12:54:16 +0100 Egmont Koblinger via Unicode wrote: > Hi Richard, > > > > Are they okay to be present in visual order (the terminal's > > > explicit mode, what we're discussing now) too? > > > > Where do you define the order f

Re: Proposal for BiDi in terminal emulators

2019-02-02 Thread Doug Ewell via Unicode
Richard Wordingham wrote: > Unicode may not deprecate the tag characters, but the characters of > Plane 14 are widely deplored, despised or abhorred. That is why I > think of it as the deprecated plane. Think of it as the deplored plane, then, or the despised plane or the abhorred plane or the P

Use of tag characters in emoji sequences (was: Re: Proposal for BiDi in terminal emulators)

2019-02-02 Thread Doug Ewell via Unicode
Philippe Verdy wrote: > Actually not all U+E0020 through U+E007E are "un-deprecated" for this > use. Characters in Unicode are not "deprecated" for some purposes and not for others. "Deprecated" is a clearly defined property in Unicode. The only reference that matters here is in PropList.txt:

Re: Proposal for BiDi in terminal emulators

2019-02-02 Thread Benjamin Riefenstahl via Unicode
Hi Egmont, hi all, This is a interesting discussion here. If only because I would have thought that there is only minimal interest by the actual target audience in supporting these scripts in a terminal, given the severe limitations of that environment. The most important limitation seems to me

Re: Proposal for BiDi in terminal emulators

2019-02-02 Thread Richard Wordingham via Unicode
On Sat, 2 Feb 2019 13:18:03 +0100 Egmont Koblinger via Unicode wrote: > Hi Richard, > > On Sat, Feb 2, 2019 at 12:43 PM Richard Wordingham via Unicode > wrote: > > > I'm not conversant with the details of terminal controls and I > > haven't used fields.

Re: Proposal for BiDi in terminal emulators

2019-02-02 Thread Richard Wordingham via Unicode
On Sat, 02 Feb 2019 14:01:46 +0100 Kent Karlsson via Unicode wrote: > Den 2019-02-02 12:17, skrev "Egmont Koblinger" : > > Most terminal emulators handle non-spacing combining marks, it's a > > piece of cake. (Spacing marks are more problematic.) > Well,

Re: Proposal for BiDi in terminal emulators

2019-02-02 Thread Philippe Verdy via Unicode
the existing specification for tags. So only E+E0020 through U+E0040, and U+E005B through U+E007E remain deprecated. Le ven. 1 févr. 2019 à 23:26, Doug Ewell via Unicode a écrit : > Richard Wordingham wrote: > > > Language tagging is already available in Unicode, via the tag >

Re: Proposal for BiDi in terminal emulators

2019-02-02 Thread Kent Karlsson via Unicode
Den 2019-02-02 12:17, skrev "Egmont Koblinger" : > the font. It's taken from EastAsianWidth (or other means, which we're > working on: https://gitlab.freedesktop.org/terminal-wg/specifications/issues/9 Yes, that too: FE0F ? VARIATION SELECTOR-16 = emoji variation selector But the issue you ref

Re: Proposal for BiDi in terminal emulators

2019-02-02 Thread Egmont Koblinger via Unicode
Hi Richard, On Sat, Feb 2, 2019 at 12:43 PM Richard Wordingham via Unicode wrote: > I'm not conversant with the details of terminal controls and I haven't > used fields. However, where I spoke of lines above, I believe you can > simply translate it to fields. I don't k

Re: Proposal for BiDi in terminal emulators

2019-02-02 Thread Egmont Koblinger via Unicode
Hi Richard, > Not all terminal emulators can deal with non-spacing combining > characters. Both Hebrew and Arabic seem to use non-spacing combining characters, presumably other Arabic-like scripts too. I forgot to state explicitly in my docs, but let's just say that handling non-spacing combinin

Re: Proposal for BiDi in terminal emulators

2019-02-02 Thread Egmont Koblinger via Unicode
Hi Richard, > > Are they okay to be present in visual order (the terminal's explicit > > mode, what we're discussing now) too? > > Where do you define the order for explicit mode? In explicit mode, the application (Emacs, Vim, whatever) reorders the characters, and passes visual order (left to ri

Re: Proposal for BiDi in terminal emulators

2019-02-02 Thread Richard Wordingham via Unicode
On Fri, 1 Feb 2019 15:15:53 +0100 Egmont Koblinger via Unicode wrote: > Hi Richard, > > On Fri, Feb 1, 2019 at 12:19 AM Richard Wordingham via Unicode > wrote: > > > Cropped why? If the problem is the truncation of lines, one can > > simple store the next character

Re: Proposal for BiDi in terminal emulators

2019-02-02 Thread Egmont Koblinger via Unicode
Hi Kent, On Sat, Feb 2, 2019 at 12:41 AM Kent Karlsson via Unicode wrote: > [...] neither of which > should directly consult the font [...] > But terminals > (read terminal emulators) can deal with mixed single width and double > width characters (which is, IIUC, the motivation f

Re: Proposal for BiDi in terminal emulators

2019-02-01 Thread Richard Wordingham via Unicode
On Fri, 1 Feb 2019 15:15:53 +0100 Egmont Koblinger via Unicode wrote: > Hi Richard, > > On Fri, Feb 1, 2019 at 12:19 AM Richard Wordingham via Unicode > wrote: > > > Cropped why? If the problem is the truncation of lines, one can > > simple store the next character

Re: Proposal for BiDi in terminal emulators

2019-02-01 Thread Richard Wordingham via Unicode
On Sat, 02 Feb 2019 00:38:04 +0100 Kent Karlsson via Unicode wrote: > Den 2019-02-01 19:57, skrev "Richard Wordingham via Unicode" > : > "Monospaced font" is really a concept with modification. Even for > "plain old ASCII" there are two advance widths

<    3   4   5   6   7   8   9   10   11   12   >