Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-18 Thread Egmont Koblinger via Unicode
On Sun, Feb 17, 2019 at 1:59 PM Philippe Verdy wrote: > Resist this idea, I've not been impolite. I didn't say a word about you being impolite. I said I might be impolite for not wishing to continue this discussion in that direction. > I just want to show you that terminals are legacy

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-17 Thread Philippe Verdy via Unicode
Le ven. 8 févr. 2019 à 13:56, Egmont Koblinger a écrit : > Philippe, I hate do say it, but at the risk of being impolite, I just > have to. > Resist this idea, I've not been impolite. I just want to show you that terminals are legacy environments that are far behind what is needed for proper

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-08 Thread Egmont Koblinger via Unicode
On Fri, Feb 8, 2019 at 10:36 PM Eli Zaretskii wrote: > No one in their right minds will run Emacs inside the Emacs terminal > emulator. And even for other applications, disabling bidi will almost > always needed only for full-screen programs, which use curses-like > libraries to address the

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-08 Thread Eli Zaretskii via Unicode
> From: Egmont Koblinger > Date: Fri, 8 Feb 2019 17:44:53 +0100 > Cc: Richard Wordingham , > unicode Unicode Discussion > > For certain apps, one of the modes is required (e.g. for cat it's the > implicit mode). For other tasks it's the other mode (e.g. for emacs > the explicit mode).

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-08 Thread Egmont Koblinger via Unicode
Hi Eli, > Why would they want to toggle it back and forth? What are the use > cases where it makes sense to mix both modes? IME, you either need > one or the other, never both. (Back to the basics, which are mentioned pretty clearly in my specification, I believe, and I've also described here

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-08 Thread Eli Zaretskii via Unicode
> From: Egmont Koblinger > Date: Fri, 8 Feb 2019 15:42:51 +0100 > Cc: Richard Wordingham , > unicode Unicode Discussion > > On Fri, Feb 8, 2019 at 3:28 PM Eli Zaretskii wrote: > > > You can have what you call the "explicit mode" if you set the variable > > bidi-display-reordering to

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-08 Thread Egmont Koblinger via Unicode
On Fri, Feb 8, 2019 at 3:28 PM Eli Zaretskii wrote: > You can have what you call the "explicit mode" if you set the variable > bidi-display-reordering to nil. So, if someone is running a mixture of applications requiring implicit vs. explicit modes, they'll have to continuously toggle the

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-08 Thread Eli Zaretskii via Unicode
> From: Egmont Koblinger > Date: Fri, 8 Feb 2019 14:57:56 +0100 > Cc: Richard Wordingham , > unicode Unicode Discussion > > According to the description you give, Emacs's terminal always applies > the BiDi algorithm, therefore by its design only implements what I > call "implicit mode",

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-08 Thread Egmont Koblinger via Unicode
Hi Eli, > Emacs implements the latest UBA from Unicode 11; and the Emacs > terminal emulator inserts all the text into a "normal" Emacs buffer, > and displays that buffer as any other buffer. So yes, you have there > full UBA support. One of the essentials of my work is that there's much more

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-08 Thread Eli Zaretskii via Unicode
> From: Egmont Koblinger > Date: Fri, 8 Feb 2019 13:30:42 +0100 > Cc: Richard Wordingham , > unicode Unicode Discussion > > Hi Eli, > > > Not sure why. There are terminal emulators out there which support > > proportional fonts. > > Well, of course, a terminal emulator can load any

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-08 Thread Egmont Koblinger via Unicode
Hi Philippe, > Adding a single bit of protection in cell attributes to indicate they are > either protected or become transparent (and the rest of the > attributes/character field indicates the id of another terminal grid or > rendering plugin crfeating its own layer and having its own

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-08 Thread Egmont Koblinger via Unicode
Hi Eli, > Not sure why. There are terminal emulators out there which support > proportional fonts. Well, of course, a terminal emulator can load any font, even proportional, but as it places them in the grid, it will look ugly as hell (like this one: https://askubuntu.com/q/781327/398785 ).

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-08 Thread Eli Zaretskii via Unicode
> Date: Fri, 8 Feb 2019 06:40:44 + > From: Richard Wordingham via Unicode > > > I, for one, am not to the slightest bit interested in abandoning the > > character grid and allowing for proportional fonts. This would just > > break a gazillion of things. > > The message I take from that and

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-07 Thread Richard Wordingham via Unicode
On Fri, 8 Feb 2019 00:38:24 +0100 Egmont Koblinger via Unicode wrote: > I, for one, am not to the slightest bit interested in abandoning the > character grid and allowing for proportional fonts. This would just > break a gazillion of things. The message I take from that and this thread in

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-07 Thread Philippe Verdy via Unicode
Adding a single bit of protection in cell attributes to indicate they are either protected or become transparent (and the rest of the attributes/character field indicates the id of another terminal grid or rendering plugin crfeating its own layer and having its own scrolling state and dimensions)

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-07 Thread Egmont Koblinger via Unicode
Hi Philippe, > I have never said anything about your work because I don't know where you > spoke about it or where you made some proposals. I must have missed one of > your messages (did it reach this list?). This entire conversation started by me announcing here my work, aiming to bring

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-07 Thread Philippe Verdy via Unicode
Le jeu. 7 févr. 2019 à 19:38, Egmont Koblinger a écrit : > As you can see from previous discussions, there's a whole lot of > confusion about the terminology. And it was exactly the subject of my first message sent to this thread ! you probably missed it. > Philippe, with all due respect, I

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-07 Thread Egmont Koblinger via Unicode
Hi Philippe, On Thu, Feb 7, 2019 at 3:21 PM Philippe Verdy wrote: > "Rules" are not formally written, they are just a sense of best practices. When it comes to BiDi in terminals, I haven't seen anything that I consider reasonably okay, let alone "best practice". It's a mess. That's why I

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-07 Thread Philippe Verdy via Unicode
Le jeu. 7 févr. 2019 à 13:29, Egmont Koblinger a écrit : > Hi Philippe, > > > There's some rules for correct display including with Bidi: > > In what sense are these "rules"? Where are these written, in what kind > of specification or existing practice? > "Rules" are not formally written, they

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-07 Thread Egmont Koblinger via Unicode
Hi Philippe, > There's some rules for correct display including with Bidi: In what sense are these "rules"? Where are these written, in what kind of specification or existing practice? > - Separate paragraphs that need a different default Bidi by double newlines > (to force a hard break)

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-06 Thread Philippe Verdy via Unicode
I read your email, you spoke for example about how a typical Unix/Linux tool shows its usage option (e.g. "anycommand --help") with a leading line then syntaxes and tabulated lists of options followed by translated help on the same line. There's some rules for correct display including with Bidi:

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-06 Thread Egmont Koblinger via Unicode
Hi Philippe, Thanks a lot for your input! Another fundamental difficulty with terminal emulators is: These controls (CR, LF...) are control instructions that move the cursor in some ways, and then are forgotten. You cannot do BiDi on the instructions the terminal receives. You can only do BiDi

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-05 Thread Philippe Verdy via Unicode
I think that before making any decision we must make some decision about what we mean by "newlines". There are in fact 3 different functions: - (1) soft line breaks (which are used to enforce a maximum display width between paragraph margins): these are equivalent to breakable and compressible

Re: Proposal for BiDi in terminal emulators

2019-02-04 Thread Eli Zaretskii via Unicode
> Date: Mon, 4 Feb 2019 21:00:55 + > From: Richard Wordingham via Unicode > > > The definition is trivial: the order of characters on > > display, from left to right. The only possible reason to split hairs > > here could be when some characters don't appear on display, like > > control

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-04 Thread Richard Wordingham via Unicode
On Mon, 4 Feb 2019 22:27:39 +0100 Egmont Koblinger via Unicode wrote: > Hi Richard, > > > The concept appears to exist in the form of the fields of the > > fifth edition of ECMA-48. Have you digested this ambitious > > standard? > > To be honest: No, I haven't. And I have no idea what those

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-04 Thread Egmont Koblinger via Unicode
Hi Eli, > I think it's unreasonable and impractical to expect 'echo', 'cat', and > its ilk to emit bidi controls (or any other controls) to force > paragraph direction. For starters, they won't know what direction to > force, because they don't understand the text they are processing. I agree,

Re: Proposal for BiDi in terminal emulators

2019-02-04 Thread Egmont Koblinger via Unicode
Hi, > To me, 'visual order' means in the dominant order of the script. This is not a definition I've come across anywhere else, nor matches my intuition of "visual order" : the exact visual order (recursive definition, yay!) of how you see the glyphs being displayed in the row. > So, > if one

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-04 Thread Richard Wordingham via Unicode
On Mon, 04 Feb 2019 22:39:07 +0200 Eli Zaretskii via Unicode wrote: > > Date: Mon, 4 Feb 2019 19:45:13 + > > From: Richard Wordingham via Unicode > > > > Yes. If one has a text composed of LTR and RTL paragraphs, one has > > to choose how far apart their starting margins are. I think

Re: Proposal for BiDi in terminal emulators

2019-02-04 Thread Asmus Freytag via Unicode
On 2/4/2019 1:00 PM, Richard Wordingham via Unicode wrote: To me, 'visual order' means in the dominant order of the script. Visual order is a term of art, meaning the characters are ordered in memory in the same order as they are displayed on the

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-04 Thread Egmont Koblinger via Unicode
> > Yes. If one has a text composed of LTR and RTL paragraphs, one has to > > choose how far apart their starting margins are. I think that could > > get complicated for plain text if the terminal has unbounded width. > > But no real-life terminal does. The width is always bounded. Allegedly

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-04 Thread Egmont Koblinger via Unicode
Hi Richard, > That split is wrong if you want the non-HTML text to lay out reasonably > well in anything but a higher order protocol forcing RTL. You need to > it split as: > > lorem ipsum ABC > <[ DEF foobar Okay, so you should use LRMs or other similar tricks when wrapping a human-perceived

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-04 Thread Egmont Koblinger via Unicode
Hi Richard, > The concept appears to exist in the form of the fields of the > fifth edition of ECMA-48. Have you digested this ambitious standard? To be honest: No, I haven't. And I have no idea what those "fields" are. I spent (read: wasted) way too much time studying ECMA TR/53 to get to

Re: Proposal for BiDi in terminal emulators

2019-02-04 Thread Richard Wordingham via Unicode
On Sun, 3 Feb 2019 20:50:03 + Richard Wordingham via Unicode wrote: > On Sun, 03 Feb 2019 20:07:51 +0200 > Eli Zaretskii via Unicode wrote: > Which is why I try to remember to issue the emacs command 'M-x shell' > command and issue grep commands from the buffer created thereby. The >

Re: Proposal for BiDi in terminal emulators

2019-02-04 Thread Richard Wordingham via Unicode
On Sun, 03 Feb 2019 18:03:37 +0200 Eli Zaretskii via Unicode wrote: > > Date: Sun, 3 Feb 2019 03:02:13 +0100 > > Cc: unicode@unicode.org > > From: Egmont Koblinger via Unicode > > > > > All I am saying is that your proposal should define what it means > > > by visual order. > > > > Are

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-04 Thread Eli Zaretskii via Unicode
> Date: Mon, 4 Feb 2019 19:45:13 + > From: Richard Wordingham via Unicode > > Yes. If one has a text composed of LTR and RTL paragraphs, one has to > choose how far apart their starting margins are. I think that could > get complicated for plain text if the terminal has unbounded width.

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-04 Thread Richard Wordingham via Unicode
On Mon, 04 Feb 2019 18:53:22 +0200 Eli Zaretskii via Unicode wrote: > Date: Mon, 4 Feb 2019 01:19:21 + > From: Richard Wordingham via Unicode >> If you look at it in Notepad, all >> lines will be LTR or all lines will be RTL. > That's because Notepad implements _only_ the higher-level

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-04 Thread Eli Zaretskii via Unicode
> Date: Mon, 4 Feb 2019 01:19:21 + > From: Richard Wordingham via Unicode > > On Sun, 03 Feb 2019 19:50:50 +0200 > Eli Zaretskii via Unicode wrote: > > > Do you see how this is carefully formatted to avoid overflowing an > > 80-column line of a typical terminal? Now suppose this is

Re: Proposal for BiDi in terminal emulators

2019-02-04 Thread Eli Zaretskii via Unicode
> Date: Mon, 04 Feb 2019 05:25:43 +0200 > Cc: unicode@unicode.org > From: Eli Zaretskii via Unicode > > Try customizing scroll-conservatively, it sounds like you want that. Ignore me: I misunderstood what you were looking for. You are right: Emacs doesn't support such scrolling method.

Re: Proposal for BiDi in terminal emulators

2019-02-03 Thread Eli Zaretskii via Unicode
> Date: Sun, 3 Feb 2019 20:35:18 + > From: Richard Wordingham via Unicode > > > What is "screen overwriting" in this context? > > When instead of adding lines to the bottom, new lines are added on top > of and replace existing lines. I prefer the scrollable terminal > behaviour to the

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-03 Thread Richard Wordingham via Unicode
On Mon, 4 Feb 2019 00:36:23 +0100 Egmont Koblinger via Unicode wrote: > Now, back to terminals. > > The smallest possible viable definition of a "paragraph" in terminal > emulators is stuff between one newline and the next one. > > It would require a hell lot of work, redesigning

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-03 Thread Richard Wordingham via Unicode
On Mon, 4 Feb 2019 00:36:23 +0100 Egmont Koblinger via Unicode wrote: > I wish to store and deliver the following text, as it's layed out here > in logical order. That is, the order as the bytes appear in the text > file, as I typed them from the keyboard, is laid out here strictly > from left

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-03 Thread Richard Wordingham via Unicode
On Sun, 03 Feb 2019 19:50:50 +0200 Eli Zaretskii via Unicode wrote: > Do you see how this is carefully formatted to avoid overflowing an > 80-column line of a typical terminal? Now suppose this is translated > into a RTL language, which causes the Copyright line to start with a > strong R

Re: Proposal for BiDi in terminal emulators

2019-02-03 Thread Richard Wordingham via Unicode
On Sun, 03 Feb 2019 18:13:06 +0200 Eli Zaretskii via Unicode wrote: > Actually, you pass the characters to be shaped in logical order, and > then display the produced grapheme clusters in visual order. Some early systems supporting computerised Hebrew script did pass characters in left-to-right

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-03 Thread Egmont Koblinger via Unicode
Hi Eli, (I'm responding in multiple emails.) The Unicode BiDi algorithm states that it operates on paragraphs of text, and leaves it up to a higher protocol to define what a paragraph exactly is. What's the definition of "paragraph" in the context of plain text files? I don't think there's a

Re: Proposal for BiDi in terminal emulators

2019-02-03 Thread Richard Wordingham via Unicode
On Sun, 03 Feb 2019 20:07:51 +0200 Eli Zaretskii via Unicode wrote: > > Date: Sun, 3 Feb 2019 17:45:06 + > > From: Richard Wordingham via Unicode > > > > > > So, what do you recommend I run grep from for Hebrew or Tai > > > > Lue? > > > > > > Inside Emacs, of course: "M-x grep RET"

Re: Proposal for BiDi in terminal emulators

2019-02-03 Thread Richard Wordingham via Unicode
On Sun, 03 Feb 2019 18:05:49 +0200 Eli Zaretskii via Unicode wrote: > > Date: Sat, 2 Feb 2019 21:49:40 + > > From: Richard Wordingham via Unicode > > > > Eli will probably tell me I'm behind the times, but there are a few > > places where a Gnome-terminal is better than an Emacs GUI

Re: Proposal for BiDi in terminal emulators

2019-02-03 Thread Eli Zaretskii via Unicode
> Date: Sun, 3 Feb 2019 17:45:06 + > From: Richard Wordingham via Unicode > > > > So, what do you recommend I run grep from for Hebrew or Tai Lue? > > > > Inside Emacs, of course: "M-x grep RET" etc. > > That assumes you like using bindings for all the commands; I don't. What bindings?

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-03 Thread Eli Zaretskii via Unicode
> From: Egmont Koblinger > Date: Sun, 3 Feb 2019 17:54:25 +0100 > Cc: unicode@unicode.org > > I'm arguing, although my reasons are not rock solid, that IMHO the > default should be the strict direction as set by SCP, without > autodetection. I think it's unreasonable and impractical to expect

Re: Proposal for BiDi in terminal emulators

2019-02-03 Thread Richard Wordingham via Unicode
On Sun, 03 Feb 2019 18:14:53 +0200 Eli Zaretskii via Unicode wrote: > > Date: Sun, 3 Feb 2019 02:43:06 + > > Cc: Kent Karlsson > > From: Richard Wordingham via Unicode > > > > So, what do you recommend I run grep from for Hebrew or Tai Lue? > > Inside Emacs, of course: "M-x grep RET"

Re: Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-03 Thread Egmont Koblinger via Unicode
Hi Eli, > The document cited at the beginning of the parent thread states that > "simple" text-mode utilities, such as 'echo', 'cat', 'ls' etc. should > use the "implicit" mode of bidi reordering, with automatic guessing of > the base paragraph direction. Not exactly. I take the SCP escape

Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

2019-02-03 Thread Eli Zaretskii via Unicode
> Date: Sun, 03 Feb 2019 18:10:15 +0200 > Cc: richard.wording...@ntlworld.com, unicode@unicode.org > From: Eli Zaretskii via Unicode > > I think there are hard problems even for such "simple" utilities, and > I will start a separate thread about this. I think we spent enough time discussing

Re: Proposal for BiDi in terminal emulators

2019-02-03 Thread Eli Zaretskii via Unicode
> Date: Sun, 3 Feb 2019 02:43:06 + > Cc: Kent Karlsson > From: Richard Wordingham via Unicode > > So, what do you recommend I run grep from for Hebrew or Tai Lue? Inside Emacs, of course: "M-x grep RET" etc.

Re: Proposal for BiDi in terminal emulators

2019-02-03 Thread Eli Zaretskii via Unicode
> Date: Sun, 3 Feb 2019 01:30:26 + > From: Richard Wordingham via Unicode > > Shaping for RTL scripts happens on strings stored in logical order. > These are then laid out right to left, though the dominant usage of > the term 'advance width' for right-to-left glyph sequences feels >

Re: Proposal for BiDi in terminal emulators

2019-02-03 Thread Eli Zaretskii via Unicode
> Date: Sat, 2 Feb 2019 23:02:10 +0100 > Cc: unicode@unicode.org > From: Egmont Koblinger via Unicode > > On top of this, I make the clarification that combining marks need to > be reordered to be sent out to the terminal emulator _after_ their > base letter That is true in general regarding

Re: Proposal for BiDi in terminal emulators

2019-02-03 Thread Eli Zaretskii via Unicode
> Date: Sat, 2 Feb 2019 21:49:40 + > From: Richard Wordingham via Unicode > > Eli will probably tell me I'm behind the times, but there are a few > places where a Gnome-terminal is better than an Emacs GUI window. One > is colour highlighting of text found by grep. ??? The Emacs 'grep'

Re: Proposal for BiDi in terminal emulators

2019-02-03 Thread Eli Zaretskii via Unicode
> Date: Sun, 3 Feb 2019 03:02:13 +0100 > Cc: unicode@unicode.org > From: Egmont Koblinger via Unicode > > > All I am saying is that your proposal should define what it means by > > visual order. > > Are you nitpicking on me not giving a precise definition on the > otherwise IMO freaking obvious

Re: Proposal for BiDi in terminal emulators

2019-02-02 Thread Richard Wordingham via Unicode
On Sun, 03 Feb 2019 02:01:18 +0100 Kent Karlsson via Unicode wrote: > Den 2019-02-02 16:12, skrev "Richard Wordingham via Unicode" > : > > Doesn't Jerusalem in biblical Hebrew sometime have 3 marks below the > > lamedh? The depth then is the maximum depth, not the sum of the > > depths. >

Re: Proposal for BiDi in terminal emulators

2019-02-02 Thread Egmont Koblinger via Unicode
Hi Richard, On Sun, Feb 3, 2019 at 2:32 AM Richard Wordingham via Unicode wrote: > That first reference doesn't even use the word 'visual'. The Unicode BiDi algorithm does speak about "visual positions for display", "reordering for display" etc. > All I am saying is that your proposal should

Re: Proposal for BiDi in terminal emulators

2019-02-02 Thread Richard Wordingham via Unicode
On Sat, 2 Feb 2019 23:02:10 +0100 Egmont Koblinger via Unicode wrote: > Hi Richard, > > On Sat, Feb 2, 2019 at 9:57 PM Richard Wordingham > wrote: > > > Seriously, you need to give a definition of 'visual order' for this > > context. Not everyone shares your chiralist view. > > When I

Re: Proposal for BiDi in terminal emulators

2019-02-02 Thread Kent Karlsson via Unicode
Den 2019-02-02 16:12, skrev "Richard Wordingham via Unicode" : > On Sat, 02 Feb 2019 14:01:46 +0100 > Kent Karlsson via Unicode wrote: > >> Well, I guess you may need to put some (practical) limit to the number >> of non-spacing marks (like max two above + max one below; overstrikes >> are an

Re: Proposal for BiDi in terminal emulators

2019-02-02 Thread Benjamin Riefenstahl via Unicode
Hi Richard, > Benjamin Riefenstahl wrote: >> the severe limitations of that environment. Richard Wordingham writes: > Eli will probably tell me I'm behind the times, but there are a few > places where a Gnome-terminal is better than an Emacs GUI window. One > is colour highlighting of text

Re: Proposal for BiDi in terminal emulators

2019-02-02 Thread Egmont Koblinger via Unicode
Hi Richard, > My main interest in this, though, is in improving the general run of > Indic terminal cell editors. If we can get Gnome-terminal working for > Kharoshthi, things should improve for LTR Indic. Even working on the > false assumption that Indic scripts are like Devanagari would be an

Re: Proposal for BiDi in terminal emulators

2019-02-02 Thread Egmont Koblinger via Unicode
Hi Richard, On Sat, Feb 2, 2019 at 9:57 PM Richard Wordingham wrote: > Seriously, you need to give a definition of 'visual order' for this > context. Not everyone shares your chiralist view. When I look at the Unicode BiDi algorithm, or go to an online demo at

Re: Proposal for BiDi in terminal emulators

2019-02-02 Thread Richard Wordingham via Unicode
On Sat, 2 Feb 2019 12:54:16 +0100 Egmont Koblinger via Unicode wrote: > Hi Richard, > > > > Are they okay to be present in visual order (the terminal's > > > explicit mode, what we're discussing now) too? > > > > Where do you define the order for explicit mode? > > In explicit mode, the

Re: Proposal for BiDi in terminal emulators

2019-02-02 Thread Doug Ewell via Unicode
Richard Wordingham wrote: > Unicode may not deprecate the tag characters, but the characters of > Plane 14 are widely deplored, despised or abhorred. That is why I > think of it as the deprecated plane. Think of it as the deplored plane, then, or the despised plane or the abhorred plane or the

Use of tag characters in emoji sequences (was: Re: Proposal for BiDi in terminal emulators)

2019-02-02 Thread Doug Ewell via Unicode
Philippe Verdy wrote: > Actually not all U+E0020 through U+E007E are "un-deprecated" for this > use. Characters in Unicode are not "deprecated" for some purposes and not for others. "Deprecated" is a clearly defined property in Unicode. The only reference that matters here is in PropList.txt:

Re: Proposal for BiDi in terminal emulators

2019-02-02 Thread Benjamin Riefenstahl via Unicode
Hi Egmont, hi all, This is a interesting discussion here. If only because I would have thought that there is only minimal interest by the actual target audience in supporting these scripts in a terminal, given the severe limitations of that environment. The most important limitation seems to

Re: Proposal for BiDi in terminal emulators

2019-02-02 Thread Richard Wordingham via Unicode
On Sat, 2 Feb 2019 13:18:03 +0100 Egmont Koblinger via Unicode wrote: > Hi Richard, > > On Sat, Feb 2, 2019 at 12:43 PM Richard Wordingham via Unicode > wrote: > > > I'm not conversant with the details of terminal controls and I > > haven't used fields. However, where I spoke of lines above,

Re: Proposal for BiDi in terminal emulators

2019-02-02 Thread Richard Wordingham via Unicode
On Sat, 02 Feb 2019 14:01:46 +0100 Kent Karlsson via Unicode wrote: > Den 2019-02-02 12:17, skrev "Egmont Koblinger" : > > Most terminal emulators handle non-spacing combining marks, it's a > > piece of cake. (Spacing marks are more problematic.) > Well, I guess you may need to put some

Re: Proposal for BiDi in terminal emulators

2019-02-02 Thread Philippe Verdy via Unicode
Actually not all U+E0020 through U+E007E are "un-deprecated" for this use. For now emoji flags only use: - U+E0041 through U+E005A (mapping to ASCII letters A through Z used in 2-letter ISO3166-1 codes). These are usable in pairs, without requiring any modifier (and only for ISO3166-1 registered

Re: Proposal for BiDi in terminal emulators

2019-02-02 Thread Kent Karlsson via Unicode
Den 2019-02-02 12:17, skrev "Egmont Koblinger" : > the font. It's taken from EastAsianWidth (or other means, which we're > working on: https://gitlab.freedesktop.org/terminal-wg/specifications/issues/9 Yes, that too: FE0F ? VARIATION SELECTOR-16 = emoji variation selector But the issue you

Re: Proposal for BiDi in terminal emulators

2019-02-02 Thread Egmont Koblinger via Unicode
Hi Richard, On Sat, Feb 2, 2019 at 12:43 PM Richard Wordingham via Unicode wrote: > I'm not conversant with the details of terminal controls and I haven't > used fields. However, where I spoke of lines above, I believe you can > simply translate it to fields. I don't know how one best handles

Re: Proposal for BiDi in terminal emulators

2019-02-02 Thread Egmont Koblinger via Unicode
Hi Richard, > Not all terminal emulators can deal with non-spacing combining > characters. Both Hebrew and Arabic seem to use non-spacing combining characters, presumably other Arabic-like scripts too. I forgot to state explicitly in my docs, but let's just say that handling non-spacing

Re: Proposal for BiDi in terminal emulators

2019-02-02 Thread Egmont Koblinger via Unicode
Hi Richard, > > Are they okay to be present in visual order (the terminal's explicit > > mode, what we're discussing now) too? > > Where do you define the order for explicit mode? In explicit mode, the application (Emacs, Vim, whatever) reorders the characters, and passes visual order (left to

Re: Proposal for BiDi in terminal emulators

2019-02-02 Thread Richard Wordingham via Unicode
On Fri, 1 Feb 2019 15:15:53 +0100 Egmont Koblinger via Unicode wrote: > Hi Richard, > > On Fri, Feb 1, 2019 at 12:19 AM Richard Wordingham via Unicode > wrote: > > > Cropped why? If the problem is the truncation of lines, one can > > simple store the next character. > > Yup, trancation of

Re: Proposal for BiDi in terminal emulators

2019-02-02 Thread Egmont Koblinger via Unicode
Hi Kent, On Sat, Feb 2, 2019 at 12:41 AM Kent Karlsson via Unicode wrote: > [...] neither of which > should directly consult the font [...] > But terminals > (read terminal emulators) can deal with mixed single width and double > width characters (which is, IIUC, the motivation for the datafile

Re: Proposal for BiDi in terminal emulators

2019-02-01 Thread Richard Wordingham via Unicode
On Fri, 1 Feb 2019 15:15:53 +0100 Egmont Koblinger via Unicode wrote: > Hi Richard, > > On Fri, Feb 1, 2019 at 12:19 AM Richard Wordingham via Unicode > wrote: > > > Cropped why? If the problem is the truncation of lines, one can > > simple store the next character. > > Yup, trancation of

Re: Proposal for BiDi in terminal emulators

2019-02-01 Thread Richard Wordingham via Unicode
On Sat, 02 Feb 2019 00:38:04 +0100 Kent Karlsson via Unicode wrote: > Den 2019-02-01 19:57, skrev "Richard Wordingham via Unicode" > : > "Monospaced font" is really a concept with modification. Even for > "plain old ASCII" there are two advance widths, not just one: 0 for > control characters

Re: Proposal for BiDi in terminal emulators

2019-02-01 Thread Richard Wordingham via Unicode
On Fri, 01 Feb 2019 15:18:13 -0700 Doug Ewell via Unicode wrote: > Richard Wordingham wrote: > > > Language tagging is already available in Unicode, via the tag > > characters in the deprecated plane. > > Plane 14 isn't deprecated -- that isn't a property of planes -- and > the tag

Re: Proposal for BiDi in terminal emulators

2019-02-01 Thread Khaled Hosny via Unicode
On Fri, Feb 01, 2019 at 06:57:43PM +, Richard Wordingham via Unicode wrote: > On Fri, 1 Feb 2019 13:02:45 +0200 > Khaled Hosny via Unicode wrote: > > > On Thu, Jan 31, 2019 at 11:17:19PM +, Richard Wordingham via > > Unicode wrote: > > > On Thu, 31 Jan 2019 12:46:48 +0100 > > > Egmont

Re: Proposal for BiDi in terminal emulators

2019-02-01 Thread Kent Karlsson via Unicode
Den 2019-02-01 19:57, skrev "Richard Wordingham via Unicode" : > On Fri, 1 Feb 2019 13:02:45 +0200 > Khaled Hosny via Unicode wrote: > >> On Thu, Jan 31, 2019 at 11:17:19PM +, Richard Wordingham via >> Unicode wrote: >>> On Thu, 31 Jan 2019 12:46:48 +0100 >>> Egmont Koblinger wrote: >>>

Re: Proposal for BiDi in terminal emulators

2019-02-01 Thread Ken Whistler via Unicode
Richard, On 2/1/2019 1:30 PM, Richard Wordingham via Unicode wrote: Language tagging is already available in Unicode, via the tag characters in the deprecated plane. Recte: 1. Plane 14 is not a "deprecated plane". 2. The tag characters in Tag Character block (U+E..U+E007F) are not

Re: Proposal for BiDi in terminal emulators

2019-02-01 Thread Andrew West via Unicode
On Fri, 1 Feb 2019 at 22:20, Doug Ewell via Unicode wrote: > > Richard Wordingham wrote: > > > Language tagging is already available in Unicode, via the tag > > characters in the deprecated plane. > > Plane 14 isn't deprecated -- that isn't a property of planes -- and the > tag characters U+E0020

Re: Proposal for BiDi in terminal emulators

2019-02-01 Thread Doug Ewell via Unicode
Richard Wordingham wrote: > Language tagging is already available in Unicode, via the tag > characters in the deprecated plane. Plane 14 isn't deprecated -- that isn't a property of planes -- and the tag characters U+E0020 through U+E007E have been un-deprecated for use with emoji flags. Only

Re: Proposal for BiDi in terminal emulators

2019-02-01 Thread Richard Wordingham via Unicode
On Fri, 1 Feb 2019 14:47:22 +0100 Egmont Koblinger via Unicode wrote: > Hi Ken, > > > [language tag] > > That is a complete non-starter for the Unicode Standard. > > Thanks for your input! > > (I hope it was clear that I just started throwing in random ideas, as > in a brainstorming

Re: Proposal for BiDi in terminal emulators

2019-02-01 Thread Richard Wordingham via Unicode
On Fri, 1 Feb 2019 13:02:45 +0200 Khaled Hosny via Unicode wrote: > On Thu, Jan 31, 2019 at 11:17:19PM +, Richard Wordingham via > Unicode wrote: > > On Thu, 31 Jan 2019 12:46:48 +0100 > > Egmont Koblinger wrote: > > > > No. How many cells do CJK ideographs occupy? We've had a strong > >

Re: Proposal for BiDi in terminal emulators

2019-02-01 Thread Egmont Koblinger via Unicode
Hi, I'm trying to respond to every question, but I'm having a hard time keeping up :-) Thanks a lot for all the precious input about shaping! Here's my suggestion, for version 0.2 of the recommendation: - No longer encourage any use of presentation form characters. - State that it's the

Re: Proposal for BiDi in terminal emulators

2019-02-01 Thread Eli Zaretskii via Unicode
> From: Egmont Koblinger > Date: Fri, 1 Feb 2019 14:35:35 +0100 > Cc: Frédéric Grosshans , > unicode@unicode.org > > > You could do that, but it will require a lot of non-trivial processing > > from the applications. Text-mode applications don't want any complex > > tinkering, they want

Re: Proposal for BiDi in terminal emulators

2019-02-01 Thread Egmont Koblinger via Unicode
Hi Richard, On Fri, Feb 1, 2019 at 12:19 AM Richard Wordingham via Unicode wrote: > Cropped why? If the problem is the truncation of lines, one can simple > store the next character. Yup, trancation of line for example. I agree that one could "store the next character". We could extend the

Re: Proposal for BiDi in terminal emulators

2019-02-01 Thread Egmont Koblinger via Unicode
Hi Ken, > [language tag] > That is a complete non-starter for the Unicode Standard. Thanks for your input! (I hope it was clear that I just started throwing in random ideas, as in a brainstorming session. This one is ruled out, then.) cheers, egmont

Re: Proposal for BiDi in terminal emulators

2019-02-01 Thread Egmont Koblinger via Unicode
On Thu, Jan 31, 2019 at 4:26 PM Eli Zaretskii wrote: > > Yes, I do argue that emacs will need to print a new escape sequence. > > Which is much-much-much-much-much better than having to tell users to > > go into the settings of their macOS Terminal / Konsole / > > gnome-terminal etc. and disable

Re: Proposal for BiDi in terminal emulators

2019-02-01 Thread Eli Zaretskii via Unicode
> From: Egmont Koblinger > Date: Fri, 1 Feb 2019 14:16:03 +0100 > Cc: Adam Borowski , unicode@unicode.org > > There's absolutely no way we could reorder first, and then handle > TAB's cursor movement. TAB's cursor movement happens in the lower > layer, reordering happens in the upper one. But

Re: Proposal for BiDi in terminal emulators

2019-02-01 Thread Egmont Koblinger via Unicode
Hi, On Thu, Jan 31, 2019 at 4:14 PM Eli Zaretskii wrote:> > I suggest that you show the result to someone who does read Arabic. I contacted one guy who is pretty knowledgeable in Arabic scripts, as well as terminal emulation, I sent out an early unpublished version of the proposal to him, but

Re: Proposal for BiDi in terminal emulators

2019-02-01 Thread Eli Zaretskii via Unicode
> From: Egmont Koblinger > Date: Fri, 1 Feb 2019 13:54:02 +0100 > Cc: Adam Borowski , unicode@unicode.org > > For this behavior, the only feature you need from a terminal emulator > is to have a mode where it doesn't shuffle the characters. Currently > every emulator I'm aware of has such a

Re: Proposal for BiDi in terminal emulators

2019-02-01 Thread Eli Zaretskii via Unicode
> From: Egmont Koblinger > Date: Fri, 1 Feb 2019 13:40:48 +0100 > Cc: unicode@unicode.org > > I now understand that presentation forms isn't an ideal possible > approach, and the recommendation should be improved here. > > Until it happens, I'm uncertain whether using presentation form >

Re: Proposal for BiDi in terminal emulators

2019-02-01 Thread Egmont Koblinger via Unicode
Hi, On Thu, Jan 31, 2019 at 4:10 PM Eli Zaretskii wrote: > The reordering happens before TABs are converted to cursor motion, > does it not? No, not at all. You cannot "mix" handling the input and reordering, since the input is not available as a single step but arrives continuously in a

Re: Proposal for BiDi in terminal emulators

2019-02-01 Thread Egmont Koblinger via Unicode
Hi Eli, > So we will some day have one such terminal emulator. That's good, but > a text-mode application that needs to support bidi cannot rely on its > users all having access to that single terminal. No. A text-mode application that needs to support BiDi must do the BiDi itself and pass

Re: Proposal for BiDi in terminal emulators

2019-02-01 Thread Egmont Koblinger via Unicode
Hi Eli, > Arabic presentation forms are more like an exception than a rule, I > hope you understand this by now. Most languages/scripts don't have > such forms, and even for Arabic they cover only a part of what needs > to be done to present correctly shaped text. Complex script shaping > is

Re: Proposal for BiDi in terminal emulators

2019-02-01 Thread Khaled Hosny via Unicode
On Thu, Jan 31, 2019 at 11:17:19PM +, Richard Wordingham via Unicode wrote: > On Thu, 31 Jan 2019 12:46:48 +0100 > Egmont Koblinger wrote: > > No. How many cells do CJK ideographs occupy? We've had a strong hint > that a medial BEH should occupy one cell, while an isolated BEH should >

Re: Proposal for BiDi in terminal emulators

2019-01-31 Thread Eli Zaretskii via Unicode
> Date: Thu, 31 Jan 2019 23:17:19 + > From: Richard Wordingham via Unicode > > Emacs needs a lot of help - I can't write a generic Tai Tham > OpenType .flt file :-( Which is why Emacs is migrating towards HarfBuzz.

  1   2   >