Re: Emacs Hebrew footnotes [CODE SNIPPET]

[Shay Gover]

On Sun, Dec 17, 2017 at 11:40 PM, Boruch Baum  wrote:

> ref: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=29759
>
> I honestly don't remember if ever in the past I've used Hebrew footnote
> numbering in emacs, but the current state of affairs is that although the
> documentation indicates that the option is available, the file seems
> nowhere to be found. The above reference reports the issue, and includes
> the following code snippet, so if anyone has an improvement, please post
> it there first.
>
> (defconst footnote-hebrew-regex "[אבגדהוזחטיכלמנסעפצקרשת]+")
> (defconst footnote-hebrew '(
>   ("א" "ב" "ג" "ד" "ה" "ו" "ז" "ח" "ט")
>   ("י" "כ" "ל" "מ" "נ" "ס" "ע" "פ" "צ")
>   ("ק" "ר" "ש" "ת" "תק" "תר"" תש" "תת" "תתק")))
> (defun Footnote-hebrew(n)
>   "Supports  footnotes, then rolls over."
>   (let*
> ((n (+ (mod n 1) (/ n 1)))
>  (thousands (/ n 1000))
>  (hundreds (/ (mod n 1000) 100))
>  (tens (/ (mod n 100) 10))
>  (units (mod n 10))
>  (special (if (not (= tens 1)) nil
>(or (when (= units 5) "טו")
>(when (= units 6) "טז")
>(concat
>  (when (/= 0 thousands) (concat (nth (1- thousands) (nth 0
> footnote-hebrew)) "'"))
>  (when (/= 0 hundreds) (nth (1- hundreds) (nth 2 footnote-hebrew)))
>  (if special special
>   (concat
> (when (/= 0 tens) (nth (1- tens) (nth 1 footnote-hebrew)))
> (when (/= 0 units) (nth (1- units) (nth 0 footnote-hebrew
> (add-to-list 'footnote-style-alist `(hebrew Footnote-hebrew
> ,footnote-hebrew-regex) t)
>
>
> --
> hkp://keys.gnupg.net
> CA45 09B5 5351 7C11 A9D1  7286 0036 9E45 1595 8BC0
>
> ___
> Linux-il mailing list
> Linux-il@cs.huji.ac.il
> http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
>
___
Linux-il mailing list
Linux-il@cs.huji.ac.il
http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il

Re: Emacs Hebrew

[Eli Zaretskii]

 From: w...@zak.co.il (Omer Zak)
 Date: Sun, 10 Jun 2012 19:11:51 +0300
 
 On Sun, 2012-06-10 at 18:56 +0300, Nadav Har'El wrote:
  On Sun, Jun 10, 2012, Tzafrir Cohen wrote about Re: Emacs  Hebrew:
   The Bidi has landed!
   
   Quoting https://www.gnu.org/software/emacs/NEWS.24.1 :
  
  Do you know if there's an option to NOT do Bidi, and just show
  everything left-to-right as previously?
 
 +1
 
 It is a very important option.  Sometimes I open a mixed language file
 in Emacs just to see the logical order of glyphs in a segment which is
 visually messed up.
 
 Will Emacs 24.1 rob me of this use case?

You know, it is quite ironic that, having heard about a major Free
Software project which now fully supports bidirectional scripts
including Hebrew, the first thing people here ask is how to disable
that feature.  Not whether it works, not if it's any good, not how
well it supports this or that aspect of bidirectional editing -- but
how to turn it off.  A sobering experience, I must say.

I sincerely hope that from now on people here will speak much more
about bidi on emacs-de...@gnu.org, reporting bugs, asking for
features, contributing code, etc.  This cannot be one-man's war like
it was until now -- that is, if we want Hebrew support in Emacs to be
not just good, but exceptional.

Enjoy Emacs 24.

___
Linux-il mailing list
Linux-il@cs.huji.ac.il
http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il

Re: Emacs Hebrew

[Eli Zaretskii]

 From: dov.grobg...@gmail.com (Dov Grobgeld)
 Date: Sun, 10 Jun 2012 19:17:09 +0300
 
 I'm using org-mode extensively during the last year, and getting Hebrew
 support helps with writing recepies and taking care of the home financies.
 There are still bugs though, e.g. like the interaction between org-tables
 and bidi is really broken.

How about if you M-x report-emacs-bug RET and describe all those
broken features?  Emacs 24.1 was in pretest for almost 9 months, but I
have no bug reports in my archives about anything related to bidi and
Org tables.  So your experience is quite unique; please make it
public, so that these bugs could be fixed.

The Org maintainers don't really care about bidi, so if we here don't
take the initiative, these problems could remain unfixed for a long
time.

Thanks in advance.

___
Linux-il mailing list
Linux-il@cs.huji.ac.il
http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il

Re: Emacs Hebrew

[Dov Grobgeld]

I absolutely agree with Eli, that getting reasonable Bidi display when
editing source code in Emacs is feasible, for the very reason that emacs is
*syntax aware*. As long as the syntax is understood, it is possible to make
sure that the various syntax elements (keywords, strings, comments) are
handled in isolation from one another, and that there is no influence
flowing e.g. from one embedded Hebrew string to the surrounding syntax
characters.

Consider a text with the following syntax:

print(HELLO!); // TELL THE USER HELLO

where small characters are LTR and capitals RTL. Without syntax awareness
the charactes !); // would be sandwiched between two RTL characters and
be turned into RTL direction by the Bidi algorithm. But by handling strings
and comments in isolation that does will not occur.

Thus I expect Bidi for syntaxes to influence strings and comments only and
leave everything else intact. This would work for Nadav's example as well:

   s/A/B/

as the syntax engine should isolate the bidi reordering of A from that of
B, there is no problem. It won't be flawless though as e.g. in the
following example:

   string.tr(abcdef,
 ABCDEF)

where you would like to substitute an 'a' for an 'A' and a 'b' for a 'B'.
By applying reordering on ABCDEF you would loose the visual alignment.

Regards,
Dov

On Fri, Jun 15, 2012 at 12:32 AM, Eli Zaretskii e...@gnu.org wrote:

  From: Omer Zak w...@zak.co.il
  Date: Thu, 14 Jun 2012 23:38:16 +0300
 
  First of all, it is GREAT that Emacs 24 has BiDi support.

 Thanks.

  Simply, have Emacs continue to support programming languages, just
  without default BiDi support when editing such text.
 
  Since it is anyway good practice to localize strings in separate *.po
  files, I suggest that all of we invest our nitpicking skills in
  specifying how to get Emacs to provide excellent support for BiDi
  rendering of *.po files and similar ones (such as Android's
  strings.xml).

 The same infrastructure that is needed to support *.po files, is also
 needed to support HTML/XML markup and comments/strings in programming
 languages.  After all, the messages in a .po file are just strings,
 they use string syntax.  Whether to use that infrastructure once it
 exists is up to the maintainers of the respective modes.  My job will
 be done when the infrastructure exists and is reasonably usable and
 flexible to satisfy these needs.

  About comments, let's not bother.  I don't think it'll be a good
  practice to write comments in any language other than English.  If
  anyone wants to do so, caveat emptor.

 I just showed you an example from Nadav's Perl script in the Hspell
 distribution.  The comments are mostly in English, but they have
 embedded Hebrew words and phrases, which is understandable given the
 goal of that script.  I don't want to dismiss this use case so easily,
 and don't really see a reason to do that, since comments are
 recognized by Emacs in any language it supports.

  What if anyone needs to illustrate how a function handles Hebrew text
  and because of this he needs to write a portion of the function's
  comment in Hebrew?

 That's what Nadav's script does.

  Would it be reasonable to require the user to insert one of the
  directionality special characters as a signal for Emacs to turn on
  visual BiDi ordering mode in that comment (and turn it back off at
  the comment's end as defined in that programming language)?

 There are no directional controls that turn on reordering.  There are
 directional controls that override the bidi properties of the
 characters, so you can have Hebrew characters have L2R directionality,
 which will effectively disable their reordering.  But these controls'
 effect ends at the newline, by virtue of the UBA, so you would need to
 use a lot of them to manually enable and disable reordering in the
 whole buffer, part by part.

  You ignored other places which might use Hebrew text in programming
  languages:
  - Perl's regular expressions (also Python).

 These are already recognized by Emacs for fontification, so the same
 machinery can be used to reorder these strings (or not, as the Perl
 users want).

  - Identifiers in computer languages, which allow non-ASCII characters in
  identifiers.

 Likewise: Emacs already fontifies identifiers, so they are
 recognizable.

  - Strings may have various and differing formatting characters/phrases
  (C, LISP and FORTRAN have their differing formatting languages).

 And they all are already recognized by Emacs, aren't they?  Otherwise
 you get bad fontification and bug reports.

  - HTML/XML fragments - can be part of either strings or comments.

 The relevant modes will have to write code to recognize them.  Emacs
 has powerful features -- regular expressions and syntax tables -- to
 facilitate that.

  - How will we let the programmer override, for one place in one file,
  the automatic derivation as needed to deal with pathological cases?

 By using the 

Re: Emacs Hebrew

[Omer Zak]

On Fri, 2012-06-15 at 10:41 +0300, Dov Grobgeld wrote:
 as the syntax engine should isolate the bidi reordering of A from that
 of B, there is no problem. It won't be flawless though as e.g. in the
 following example:
 
string.tr(abcdef,
  ABCDEF)
  
 where you would like to substitute an 'a' for an 'A' and a 'b' for a
 'B'. By applying reordering on ABCDEF you would loose the visual
 alignment.

Nice example!

It could be complicated further by having strings with mixture of LTR
and RTL glyphs.  Such as:

string.tr(abcdefghijklmnopqrstuvwxyz
  ABC,EFG abc: HIJKL de MN)
// small characters and LTR and capitals are RTL.

In which BiDi reordering would leave the software developer very
confused if he wants to figure out into which glyphs do 'q' and 'v' get
translated.  Or which glyph was the original for ':'.

It illustrates the reason why I prefer to have no automatic BiDi
reordering of source code displayed by Emacs.
My preference, when editing software, is to show all text in logical
order, but provide the user with a popup window showing the visual
order.

--- Omer


-- 
MS-Windows is the Pal-Kal of the PC world.
My own blog is at http://www.zak.co.il/tddpirate/

My opinions, as expressed in this E-mail message, are mine alone.
They do not represent the official policy of any organization with which
I may be affiliated in any way.
WARNING TO SPAMMERS:  at http://www.zak.co.il/spamwarning.html


___
Linux-il mailing list
Linux-il@cs.huji.ac.il
http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il

Re: Emacs Hebrew

[Eli Zaretskii]

 From: Omer Zak w...@zak.co.il
 Cc: Eli Zaretskii e...@gnu.org, linux-il@cs.huji.ac.il
 Date: Fri, 15 Jun 2012 10:55:13 +0300
 
 Nice example!
 
 It could be complicated further by having strings with mixture of LTR
 and RTL glyphs.  Such as:
 
 string.tr(abcdefghijklmnopqrstuvwxyz
   ABC,EFG abc: HIJKL de MN)
 // small characters and LTR and capitals are RTL.
 
 In which BiDi reordering would leave the software developer very
 confused if he wants to figure out into which glyphs do 'q' and 'v' get
 translated.  Or which glyph was the original for ':'.

We could have specially-formatted comments, or some other way, to tell
the display engine not to reorder a certain portion of the source
code.  Once selective reordering is available, it would be easy enough
to use it in such ways to cover special cases.  We already have
similar features via file-local variables, 'coding' tags, etc.

The main point is that we should not punish 90% of use cases for the
sake of the other 10%.  Holding a feature because it only handles 90%
of use cases is a bad call, IMO.

 My preference, when editing software, is to show all text in logical
 order, but provide the user with a popup window showing the visual
 order.

This will make reading a source with a lot of Hebrew text extremely
cumbersome, to say the least.  Again, the usual cases should just
work.

___
Linux-il mailing list
Linux-il@cs.huji.ac.il
http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il

Re: Emacs Hebrew

[Omer Zak]

On Fri, 2012-06-15 at 11:08 +0300, Eli Zaretskii wrote:
  From: Omer Zak w...@zak.co.il
[... a pathological example was snipped ...]
  In which BiDi reordering would leave the software developer very
  confused if he wants to figure out into which glyphs do 'q' and 'v' get
  translated.  Or which glyph was the original for ':'.
 
 We could have specially-formatted comments, or some other way, to tell
 the display engine not to reorder a certain portion of the source
 code.  Once selective reordering is available, it would be easy enough
 to use it in such ways to cover special cases.  We already have
 similar features via file-local variables, 'coding' tags, etc.
 
 The main point is that we should not punish 90% of use cases for the
 sake of the other 10%.  Holding a feature because it only handles 90%
 of use cases is a bad call, IMO.

With this I agree, as long as it is possible to deal with the other 10%
with just one extra command or another equally simpleeasy solution.

The other BiDi-enabled text editors horribly fail in this.  They provide
no built-in way to disable reordering of mixed directionality text.
This horror is a good reason for me, Nadav etc. to be worried about
ability to turn off BiDi reordering at will.

--- Omer


-- 
MS-Windows is the Pal-Kal of the PC world.
My own blog is at http://www.zak.co.il/tddpirate/

My opinions, as expressed in this E-mail message, are mine alone.
They do not represent the official policy of any organization with which
I may be affiliated in any way.
WARNING TO SPAMMERS:  at http://www.zak.co.il/spamwarning.html


___
Linux-il mailing list
Linux-il@cs.huji.ac.il
http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il

Re: Emacs Hebrew

[Eli Zaretskii]

 Date: Thu, 14 Jun 2012 07:36:21 +0300
 From: Nadav Har'El n...@math.technion.ac.il
 Cc: linux-il@cs.huji.ac.il
 
 Like I said, try the example Perl instruction of changing aleph into
 bet:
 
   s/א/ב/
 
 See how the Bidi algorithm makes it appear as if we're changing the
 other way around - bet into aleph. I honestly don't see what kind of
 base direction choice or other high-level protocol can save this
 case. I have to admit I didn't try it on Emacs 24 (which I don't have
 yet), but did try it on other bidi-capable programs. Maybe there is a
 workaround in this case, and you can educate me?

This snippet will indeed be rendered confusingly by any bidi-aware
application (including Emacs 24.1).  But the solution is not to turn
off the reordering wholesale.  That's because you do want the
reordering here:

$word =~ s/ה$/h/o if $opts{שמור_מפיק};

and here:

  # TODO: resolve the following linguistic question: Is there a difference
  # in the pronunciation and spelling of בדוחפם (when they push, bdoxpam)
  # and לדחופם (to push them, lidxpam)? The first is an subjectization of
  # לדחוף, and the second is a objectization. I do *not* know if the above
  # differentiation is valid or correct, and failed to find references to
  # support my gut feeling. Thus, on the mean while, I produce a waw-less
  # form, as done by rav-millim.

So what is needed is _selective_ reordering: some portions of the
buffer need to be reordered, while others need to stay in their
original logical order.

For buffers that hold source code in some programming language, the
parts to reorder are comments and strings.  Precisely what is a
comment or a string is something each major mode should determine --
and they all do already.  For markup languages such as HTML and XML,
these are labels and perhaps other things.  It's possible that there
are other types of non-plain text that need similar treatment; I need
feedback for making sure all the possibilities are covered.

Emacs currently lacks some fundamental infrastructure to support such
selective reordering, but it is on my todo list, and I will publish
the basic design principles in the near future.  However, my ideas of
which infrastructure is needed and how to expose it to Lisp
applications are based on a very limited number of use cases, most of
them from my daily experience, where I almost never use Hebrew.  I
cannot be sure my experience covers enough turf to be a solid basis
for supporting non-plain text.

This is where this community should enter the picture.  When you find
a use case where the default reordering doesn't do a good enough job,
don't just disable reordering.  Instead, report those cases as bugs or
missing features, and if you can, send suggestions for how to solve
this, so that you and others will be happy.  If you just disable
reordering, we will never be able to get it better.

Thanks in advance.


___
Linux-il mailing list
Linux-il@cs.huji.ac.il
http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il

Re: Emacs Hebrew

[Omer Zak]

The discussion below reminds me of the Worse Is Better debate (see, for
example, http://www.codinghorror.com/blog/2004/08/worse-is-better.html).

Once we start making BiDi rendering mode dependent upon nitpicking
details of the particular text displayed in a buffer, it is a losing
game.  There are so many special cases, you are bound to lose some
pathological corner cases.

And According to Behdad Esfahbod, the very Unicode BiDi algorithm itself
fails to correctly handle all kinds of corner cases in rendering Arabic
text.

How, for example, should we handle a Perl code snippet having Nadav
Har'EL's example, which is embedded in Hebrew text (say, a chapter in a
Perl textbook written in Hebrew)?

The solution that I propose is:
- Turn off BiDi by default in all programming language major modes.
- Turn on BiDi by default in major modes which handle text.

In both cases, provide an easy way to display a marked text snippet in
the opposite BiDi rendering mode.
- When the default is to turn off the BiDi mode, the display of text
after BiDi rendering can be an uneditable pop-up window.
- When the default is to turn on BiDi mode, then when the user wants to
see the text rendered in logical ordering (without BiDi), he should be
able to edit it in this mode (and with an easy way to insert
directionality modifying/overriding special characters) - I expect it to
be used to clean up places where the BiDi rendering engine messed up the
text.

--- Omer


On Thu, 2012-06-14 at 19:58 +0300, Eli Zaretskii wrote:
  Date: Thu, 14 Jun 2012 07:36:21 +0300
  From: Nadav Har'El n...@math.technion.ac.il
  Cc: linux-il@cs.huji.ac.il
  
  Like I said, try the example Perl instruction of changing aleph into
  bet:
  
  s/א/ב/
  
  See how the Bidi algorithm makes it appear as if we're changing the
  other way around - bet into aleph. I honestly don't see what kind of
  base direction choice or other high-level protocol can save this
  case. I have to admit I didn't try it on Emacs 24 (which I don't have
  yet), but did try it on other bidi-capable programs. Maybe there is a
  workaround in this case, and you can educate me?
 
 This snippet will indeed be rendered confusingly by any bidi-aware
 application (including Emacs 24.1).  But the solution is not to turn
 off the reordering wholesale.  That's because you do want the
 reordering here:
 
 $word =~ s/ה$/h/o if $opts{שמור_מפיק};
 
 and here:
 
   # TODO: resolve the following linguistic question: Is there a difference
   # in the pronunciation and spelling of בדוחפם (when they push, bdoxpam)
   # and לדחופם (to push them, lidxpam)? The first is an subjectization of
   # לדחוף, and the second is a objectization. I do *not* know if the above
   # differentiation is valid or correct, and failed to find references to
   # support my gut feeling. Thus, on the mean while, I produce a waw-less
   # form, as done by rav-millim.
 
 So what is needed is _selective_ reordering: some portions of the
 buffer need to be reordered, while others need to stay in their
 original logical order.
 
 For buffers that hold source code in some programming language, the
 parts to reorder are comments and strings.  Precisely what is a
 comment or a string is something each major mode should determine --
 and they all do already.  For markup languages such as HTML and XML,
 these are labels and perhaps other things.  It's possible that there
 are other types of non-plain text that need similar treatment; I need
 feedback for making sure all the possibilities are covered.
 
 Emacs currently lacks some fundamental infrastructure to support such
 selective reordering, but it is on my todo list, and I will publish
 the basic design principles in the near future.  However, my ideas of
 which infrastructure is needed and how to expose it to Lisp
 applications are based on a very limited number of use cases, most of
 them from my daily experience, where I almost never use Hebrew.  I
 cannot be sure my experience covers enough turf to be a solid basis
 for supporting non-plain text.
 
 This is where this community should enter the picture.  When you find
 a use case where the default reordering doesn't do a good enough job,
 don't just disable reordering.  Instead, report those cases as bugs or
 missing features, and if you can, send suggestions for how to solve
 this, so that you and others will be happy.  If you just disable
 reordering, we will never be able to get it better.
 
 Thanks in advance.
-- 
MS-Windows is the Pal-Kal of the PC world.
My own blog is at http://www.zak.co.il/tddpirate/

My opinions, as expressed in this E-mail message, are mine alone.
They do not represent the official policy of any organization with which
I may be affiliated in any way.
WARNING TO SPAMMERS:  at http://www.zak.co.il/spamwarning.html


___
Linux-il mailing list
Linux-il@cs.huji.ac.il
http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il

Re: Emacs Hebrew

[Eli Zaretskii]

 From: Omer Zak w...@zak.co.il
 Date: Thu, 14 Jun 2012 22:27:22 +0300
 
 Once we start making BiDi rendering mode dependent upon nitpicking
 details of the particular text displayed in a buffer, it is a losing
 game.  There are so many special cases, you are bound to lose some
 pathological corner cases.

You are, in effect, saying that there's no hope for Emacs to support
programming languages, something it already does quite well.  Because
the same machinery that is used now to find comments and strings,
fontify them correctly, and support various specialized commands for
them -- the same machinery is what is needed to reorder those same
comments and strings.

Maybe I'm missing something, but then please give specific examples
why you think this is a losing game.

 And According to Behdad Esfahbod, the very Unicode BiDi algorithm itself
 fails to correctly handle all kinds of corner cases in rendering Arabic
 text.

Irrelevant.  The issue here is how to reorder comments and strings in
a program source code without getting jumbled display due to
punctuation characters surrounding those comments and strings, which
are not part of the comments/strings.  If the current UBA cannot
handle some script well enough, then this pertains to reordering the
body of the comment/string itself, and fixing that is a separate issue
that should be pursued with the Unicode Consortium, not with Emacs
developers.

 How, for example, should we handle a Perl code snippet having Nadav
 Har'EL's example, which is embedded in Hebrew text (say, a chapter in a
 Perl textbook written in Hebrew)?

By putting text properties on each portion of that text guiding
the reordering engine which portions to reorder and with what base
embedding direction.  Typically, a textbook with embedded code
snippets has those snippets marked by some markup, like @code..@end code
in Texinfo; these can be used to place the text properties as required.

In any case, even if the use case you describe is hard to handle, it
doesn't yet mean that we shouldn't handle simpler cases.  Emacs is a
programmer's editor, so rendering correctly a program source code is
something that it should do well, even if it has problems with code
embedded in plain text.  If MS Studio does it, it would be a shame if
Emacs didn't, don't you think?

 The solution that I propose is:
 - Turn off BiDi by default in all programming language major modes.

That doesn't provide solution for displaying comments and strings in a
legible form.

 In both cases, provide an easy way to display a marked text snippet in
 the opposite BiDi rendering mode.

Isn't that what I describe above?  And if so, what are we exactly
arguing about? is that about who marks the text to be reordered, where
I think Emacs should know that automagically, while you want to
place that burden on the user?

 - When the default is to turn off the BiDi mode, the display of text
 after BiDi rendering can be an uneditable pop-up window.
 - When the default is to turn on BiDi mode, then when the user wants to
 see the text rendered in logical ordering (without BiDi), he should be
 able to edit it in this mode (and with an easy way to insert
 directionality modifying/overriding special characters) - I expect it to
 be used to clean up places where the BiDi rendering engine messed up the
 text.

These are separate features, not directly related to display of
program source code.  They can be easily implemented, if the consensus
is that they are needed, since the infrastructure for that already
exists in Emacs.  By contrast, selectively reordering portions of a
buffer is not yet possible, so that is where my work will happen in
the near future.

___
Linux-il mailing list
Linux-il@cs.huji.ac.il
http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il

Re: Emacs Hebrew

[Omer Zak]

Hello Eli,

Since I didn't say it before, I'll say it now.
First of all, it is GREAT that Emacs 24 has BiDi support.
None of my comments are meant to detract from this major achievement.

I am sorry for my part in leading you to feel that we detract in any way
from your achievement.

On Thu, 2012-06-14 at 23:04 +0300, Eli Zaretskii wrote:
  From: Omer Zak w...@zak.co.il
  Date: Thu, 14 Jun 2012 22:27:22 +0300
  
  Once we start making BiDi rendering mode dependent upon nitpicking
  details of the particular text displayed in a buffer, it is a losing
  game.  There are so many special cases, you are bound to lose some
  pathological corner cases.
 
 You are, in effect, saying that there's no hope for Emacs to support
 programming languages, something it already does quite well.

Simply, have Emacs continue to support programming languages, just
without default BiDi support when editing such text.

Since it is anyway good practice to localize strings in separate *.po
files, I suggest that all of we invest our nitpicking skills in
specifying how to get Emacs to provide excellent support for BiDi
rendering of *.po files and similar ones (such as Android's
strings.xml).

   Because
 the same machinery that is used now to find comments and strings,
 fontify them correctly, and support various specialized commands for
 them -- the same machinery is what is needed to reorder those same
 comments and strings.

About strings, see above.
About comments, let's not bother.  I don't think it'll be a good
practice to write comments in any language other than English.  If
anyone wants to do so, caveat emptor.

However, there is a question:
What if anyone needs to illustrate how a function handles Hebrew text
and because of this he needs to write a portion of the function's
comment in Hebrew?
Would it be reasonable to require the user to insert one of the
directionality special characters as a signal for Emacs to turn on
visual BiDi ordering mode in that comment (and turn it back off at
the comment's end as defined in that programming language)?

You ignored other places which might use Hebrew text in programming
languages:
- Perl's regular expressions (also Python).
- Identifiers in computer languages, which allow non-ASCII characters in
identifiers.

 Maybe I'm missing something, but then please give specific examples
 why you think this is a losing game.

- Strings may have various and differing formatting characters/phrases
(C, LISP and FORTRAN have their differing formatting languages).
- HTML/XML fragments - can be part of either strings or comments.

  How, for example, should we handle a Perl code snippet having Nadav
  Har'EL's example, which is embedded in Hebrew text (say, a chapter in a
  Perl textbook written in Hebrew)?
 
 By putting text properties on each portion of that text guiding
 the reordering engine which portions to reorder and with what base
 embedding direction.  Typically, a textbook with embedded code
 snippets has those snippets marked by some markup, like @code..@end code
 in Texinfo; these can be used to place the text properties as required.

I agree with this approach.
With time, algorithms for automatic derivation of text directionality
handling for each programming language will become better and better.
Questions/points:
- How will we let the programmer override, for one place in one file,
the automatic derivation as needed to deal with pathological cases?
- How reasonable will it be to ask the programmer to insert extra
characters into the text just to get it correctly rendered?

 In any case, even if the use case you describe is hard to handle, it
 doesn't yet mean that we shouldn't handle simpler cases.  Emacs is a
 programmer's editor, so rendering correctly a program source code is
 something that it should do well, even if it has problems with code
 embedded in plain text.  If MS Studio does it, it would be a shame if
 Emacs didn't, don't you think?

Strings are not that simple cases (see above).
Also, in several cases, the correct way to render a piece of text
depends upon various external considrations, of which there is no
reasonable way to make Emacs aware.
Hence, my preference is toward making it easy for the user to see the
text in both logical and visual ordering (not necessarily at the same
time).

  The solution that I propose is:
  - Turn off BiDi by default in all programming language major modes.
 
 That doesn't provide solution for displaying comments and strings in a
 legible form.

See my comment about *.po files usage above.

  In both cases, provide an easy way to display a marked text snippet in
  the opposite BiDi rendering mode.
 
 Isn't that what I describe above?  And if so, what are we exactly
 arguing about? is that about who marks the text to be reordered, where
 I think Emacs should know that automagically, while you want to
 place that burden on the user?

My claim is that the computer will get it wrong anyway, frequently
enough that we need to provide 

Re: Emacs Hebrew

[Eli Zaretskii]

 From: Omer Zak w...@zak.co.il
 Date: Thu, 14 Jun 2012 23:38:16 +0300
 
 First of all, it is GREAT that Emacs 24 has BiDi support.

Thanks.

 Simply, have Emacs continue to support programming languages, just
 without default BiDi support when editing such text.
 
 Since it is anyway good practice to localize strings in separate *.po
 files, I suggest that all of we invest our nitpicking skills in
 specifying how to get Emacs to provide excellent support for BiDi
 rendering of *.po files and similar ones (such as Android's
 strings.xml).

The same infrastructure that is needed to support *.po files, is also
needed to support HTML/XML markup and comments/strings in programming
languages.  After all, the messages in a .po file are just strings,
they use string syntax.  Whether to use that infrastructure once it
exists is up to the maintainers of the respective modes.  My job will
be done when the infrastructure exists and is reasonably usable and
flexible to satisfy these needs.

 About comments, let's not bother.  I don't think it'll be a good
 practice to write comments in any language other than English.  If
 anyone wants to do so, caveat emptor.

I just showed you an example from Nadav's Perl script in the Hspell
distribution.  The comments are mostly in English, but they have
embedded Hebrew words and phrases, which is understandable given the
goal of that script.  I don't want to dismiss this use case so easily,
and don't really see a reason to do that, since comments are
recognized by Emacs in any language it supports.

 What if anyone needs to illustrate how a function handles Hebrew text
 and because of this he needs to write a portion of the function's
 comment in Hebrew?

That's what Nadav's script does.

 Would it be reasonable to require the user to insert one of the
 directionality special characters as a signal for Emacs to turn on
 visual BiDi ordering mode in that comment (and turn it back off at
 the comment's end as defined in that programming language)?

There are no directional controls that turn on reordering.  There are
directional controls that override the bidi properties of the
characters, so you can have Hebrew characters have L2R directionality,
which will effectively disable their reordering.  But these controls'
effect ends at the newline, by virtue of the UBA, so you would need to
use a lot of them to manually enable and disable reordering in the
whole buffer, part by part.

 You ignored other places which might use Hebrew text in programming
 languages:
 - Perl's regular expressions (also Python).

These are already recognized by Emacs for fontification, so the same
machinery can be used to reorder these strings (or not, as the Perl
users want).

 - Identifiers in computer languages, which allow non-ASCII characters in
 identifiers.

Likewise: Emacs already fontifies identifiers, so they are
recognizable.

 - Strings may have various and differing formatting characters/phrases
 (C, LISP and FORTRAN have their differing formatting languages).

And they all are already recognized by Emacs, aren't they?  Otherwise
you get bad fontification and bug reports.

 - HTML/XML fragments - can be part of either strings or comments.

The relevant modes will have to write code to recognize them.  Emacs
has powerful features -- regular expressions and syntax tables -- to
facilitate that.

 - How will we let the programmer override, for one place in one file,
 the automatic derivation as needed to deal with pathological cases?

By using the directional controls, or by inserting newlines
judiciously.  This should cover almost everything.

 - How reasonable will it be to ask the programmer to insert extra
 characters into the text just to get it correctly rendered?

I don't know, I guess it depends on the programmer.

 Unless priorities have changed without my knowing about this, Emacs is
 an editor.  It is not meant to be a WYSIWYG type editor.  It is not
 meant to be a text viewer (such as a Web browser).  It must make it easy
 to edit the text, not necessarily render it in some final form.

Well, I guess priorities have changed, then: Emacs now has a lot of
specialized modes whose goal is to display plain-text files in a lot
of complex ways that border on WYSIWYG.  Just to mention a few
examples, take the Org mode, or the way cross-references are displayed
in Info manuals, or the fancy tabulated list displays provided by
tabulated-list.el (used by buffer-menu.el).

 And when programming, there are enough cases in which it is easier to
 edit BiDi text displayed in logical order (not reordered) rather than
 visual order.

Easiness is not relevant here: the reordering engine is already coded
and fully operational.  If there are no strong R2L characters in a
buffer, the text displays as it was in previous versions of Emacs,
even though it passes through the reordering engine.

___
Linux-il mailing list
Linux-il@cs.huji.ac.il

Re: Emacs Hebrew

[Eli Zaretskii]

 From: w...@zak.co.il (Omer Zak)
 Date: Tue, 12 Jun 2012 22:57:06 +0300
 
 On Tue, 2012-06-12 at 19:05 +0300, Eli Zaretskii wrote:
  You know, it is quite ironic that, having heard about a major Free
  Software project which now fully supports bidirectional scripts
  including Hebrew, the first thing people here ask is how to disable
  that feature.  Not whether it works, not if it's any good, not how
  well it supports this or that aspect of bidirectional editing -- but
  how to turn it off.  A sobering experience, I must say.
 
 It is a question of control.

If you use Emacs, you know that giving the user control is one of
Emacs's main design principles.

I respond to some of your points below, but my point was that the last
thing I expected from this community, during almost 3 years it took to
develop bidirectional display engine, was that the first reaction
would be tell me how to turn it off.

 The problem is that Gedit implemented BiDi support without the option to
 turn it off.
 
 So when one wants to view BiDi text in visual order (the usual case),
 one opens the file in Gedit.  And when one wants to see it in logical
 order (e.g. to figure out how the visual order turned out to be so
 messed up), one opened it in Emacs.

Suppose the visual order is never messed up in Emacs -- do you still
need this switch?

 Another use case is by blind computer users, who prefer to use Braille.
 All BiDi Braille text must be laid out in logical order.  So even if
 your editor supports Hebrew Braille fonts, they are of no use if the
 text is - too helpfully - printed out in visual order.

I don't know enough about this, but isn't Hebrew Braille just an
encoding of Hebrew letters using Braille characters?  If so, then
Braille characters all have string left-to-right directionality, and
therefore will not be reordered for display by Emacs.

But even if the above does not solve the issue, I would expect people
to request that Emacs does TRT by default with Hebrew Braille, rather
than turning bidi off.

 Few years ago, when Abiword people were debugging BiDi support, I asked
 for an option to turn it off for the benefit of blind Hebrew computer
 users.  This request was turned down.

Well, I hope you agree now that Emacs is not Abiword...

___
Linux-il mailing list
Linux-il@cs.huji.ac.il
http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il

Re: Emacs Hebrew

[Omer Zak]

On Wed, 2012-06-13 at 19:15 +0300, Eli Zaretskii wrote:
  From: w...@zak.co.il (Omer Zak)
  It is a question of control.
 
 If you use Emacs, you know that giving the user control is one of
 Emacs's main design principles.

Great!

  So when one wants to view BiDi text in visual order (the usual case),
  one opens the file in Gedit.  And when one wants to see it in logical
  order (e.g. to figure out how the visual order turned out to be so
  messed up), one opened it in Emacs.
 
 Suppose the visual order is never messed up in Emacs -- do you still
 need this switch?

YES.  The Unicode BiDi algorithm is not perfect and it sometimes messes
up the visual order of BiDi strings.  The BiDi algorithm implementation
in Emacs must follow the Unicode standard.  Hence, it is inevitable that
there'll be some cases, in which the visual order is messed up in Emacs.
In fact, not to be messed up in those pathological cases would actually
be a bug in Emacs' BiDi algorithm implementation.

  Another use case is by blind computer users, who prefer to use Braille.
  All BiDi Braille text must be laid out in logical order.  So even if
  your editor supports Hebrew Braille fonts, they are of no use if the
  text is - too helpfully - printed out in visual order.
 
 I don't know enough about this, but isn't Hebrew Braille just an
 encoding of Hebrew letters using Braille characters?  If so, then
 Braille characters all have string left-to-right directionality, and
 therefore will not be reordered for display by Emacs.

No.  The same standard encoding is used.  Braille is just another font.
To do otherwise would render regular Hebrew text inaccessible to the
Hebrew speaking blind computer users.
This font, however, needs to be rendered by bypassing the BiDi
algorithm.

 But even if the above does not solve the issue, I would expect people
 to request that Emacs does TRT by default with Hebrew Braille, rather
 than turning bidi off.

What does TRT stand for?

Turning BiDi off would be a satisfactory solution in this use case.  I
envision that sometime in the future, an Emacs startup script be
devised, such that swithcing to Hebrew Braille font would by default
switch BiDi off.

 
  Few years ago, when Abiword people were debugging BiDi support, I asked
  for an option to turn it off for the benefit of blind Hebrew computer
  users.  This request was turned down.
 
 Well, I hope you agree now that Emacs is not Abiword...

:-)

--- Omer


-- 
My Commodore 64 is suffering from slowness and insufficiency of memory;
and its display device is grievously short of pixels.  Can anyone help?
My own blog is at http://www.zak.co.il/tddpirate/

My opinions, as expressed in this E-mail message, are mine alone.
They do not represent the official policy of any organization with which
I may be affiliated in any way.
WARNING TO SPAMMERS:  at http://www.zak.co.il/spamwarning.html


___
Linux-il mailing list
Linux-il@cs.huji.ac.il
http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il

Re: Emacs Hebrew

[Omer Zak]

On Wed, 2012-06-13 at 19:22 +0300, Eli Zaretskii wrote:
 And since the bidirectional display for Emacs was developed in almost
 complete isolation from this community -- not a single input or
 response to several design discussions I posted -- why would anyone
 assume that an essentially one-man project will not end up being a
 complete disaster from usability point of view?

At the time I considered helping the effort.  However, Emacs is too big
piece of software to dive into without investing serious time.

  Bidi is great for writing texts, but since until now writing Hebrew
  text in Emacs wasn't a great idea, people didn't do it. What they
  did do with Emacs is writing code, editing config files, and similar
  things. With those, Bidi is sometimes a distraction, not a desired
  feature - so people want to be able to turn it off.
 
 How do you know it's a distraction, if you didn't yet try it?

I did try to edit code with embedded Hebrew strings using BiDi compliant
editors such as Gedit, and it was indeed a disaster.  Not because of any
usability shortcoming of the editor, but because of the very nature of
the BiDi algorithm.  Emacs was good for editing such files, and I am
glad that Emacs remains this way (by having the option to turn off BiDi
algorithm - and I suggest to turn BiDi off by default in all major
modes, which are used for programming language supporting Hebrew glyphs
in constant strings, such as Python, Java, etc.).

--- Omer
P.S.: I hope to install Emacs 24 as soon as it is backported to Debian
Squeeze or Debian Wheezy.


-- 
My Commodore 64 is suffering from slowness and insufficiency of memory;
and its display device is grievously short of pixels.  Can anyone help?
My own blog is at http://www.zak.co.il/tddpirate/

My opinions, as expressed in this E-mail message, are mine alone.
They do not represent the official policy of any organization with which
I may be affiliated in any way.
WARNING TO SPAMMERS:  at http://www.zak.co.il/spamwarning.html


___
Linux-il mailing list
Linux-il@cs.huji.ac.il
http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il

Re: Emacs Hebrew

[Nadav Har'El]

On Wed, Jun 13, 2012, Eli Zaretskii wrote about Re: Emacs  Hebrew:
 You mean, disappointing?  Yes, it is.  To hear such questions from
 Nadav Har'El, of all the people.

Sorry to disappoint you...

One of my random signatures go better to be thought a fool, than to
open your mouth and remove all doubt. Too bad I opened my mouth :-)

 And since the bidirectional display for Emacs was developed in almost
 complete isolation from this community -- not a single input or
 response to several design discussions I posted -- why would anyone
 assume that an essentially one-man project will not end up being a
 complete disaster from usability point of view?

Because I know you, and don't believe you are capable of producing a
complete disaster? :-)

  Bidi is great for writing texts, but since until now writing Hebrew
  text in Emacs wasn't a great idea, people didn't do it. What they
  did do with Emacs is writing code, editing config files, and similar
  things. With those, Bidi is sometimes a distraction, not a desired
  feature - so people want to be able to turn it off.
 
 How do you know it's a distraction, if you didn't yet try it?

Let me explain what I meant, and why I wrote my question.

As you may know, Hspell has a perl script which takes Hebrew base words,
and inflects them (e.g., produces plural, possesives, etc., of nouns).
The Perl script is, of course, mostly ASCII, but with various short Hebrew
strings sprinkled in it.
It is, in my opinion, less confusing to view these strings without BIDI
applied, which is why I edit it with regular non-bidi-capable vim.

As an example of the confusion that bidi might cause me when editing the
Hspell source code, conside what it might to do a regular expression:
Imagine that I wrote s/א/ב/ with the intension of switching aleph into bet,
but (I just checked...) the bidi algorithm applied to it will display the
order *reversed* making it confusingly appear that the code is changing bet
into aleph...

I think you must agree that the unicode bidi algorithm was never designed
for such use cases - and I'm not blaming these issues on your
implementation, of course - and I don't even need to try your
implementation to know it will have this - or similar - issues when
editing source code. And editing mostly-ASCII source code is 99% of what I
personally use Emacs for.

Again, nothing I said or asked applies to writing *text*, such as email.
For text, bidi is definitely required, and really welcome. I sorely miss a
bidi-capable editor for my email writing - unfortunately I have a 20 year
old habit of writing emails in vi, so it'll take me a bit of time to decide
to switch to emacs also for emails. But now I know I definitely should!

-- 
Nadav Har'El| Wednesday, Jun 13 2012, 
n...@math.technion.ac.il |-
Phone +972-523-790466, ICQ 13349191 |A Nobel Peace Prize? I would KILL for one
http://nadav.harel.org.il   |of those.

___
Linux-il mailing list
Linux-il@cs.huji.ac.il
http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il

Re: Emacs Hebrew

[Eli Zaretskii]

 From: Omer Zak w...@zak.co.il
 Date: Wed, 13 Jun 2012 19:33:29 +0300
 
   So when one wants to view BiDi text in visual order (the usual case),
   one opens the file in Gedit.  And when one wants to see it in logical
   order (e.g. to figure out how the visual order turned out to be so
   messed up), one opened it in Emacs.
  
  Suppose the visual order is never messed up in Emacs -- do you still
  need this switch?
 
 YES.  The Unicode BiDi algorithm is not perfect and it sometimes messes
 up the visual order of BiDi strings.  The BiDi algorithm implementation
 in Emacs must follow the Unicode standard.  Hence, it is inevitable that
 there'll be some cases, in which the visual order is messed up in Emacs.
 In fact, not to be messed up in those pathological cases would actually
 be a bug in Emacs' BiDi algorithm implementation.

It remains to be seen whether these cases should not have a better
solution than to turn off reordering.  You didn't give any practical
examples, so it's hard to reason about that.

In any case, just to see the logical sequence, it is enough to move
the cursor, since cursor moves in logical order in Emacs.

   Another use case is by blind computer users, who prefer to use Braille.
   All BiDi Braille text must be laid out in logical order.  So even if
   your editor supports Hebrew Braille fonts, they are of no use if the
   text is - too helpfully - printed out in visual order.
  
  I don't know enough about this, but isn't Hebrew Braille just an
  encoding of Hebrew letters using Braille characters?  If so, then
  Braille characters all have string left-to-right directionality, and
  therefore will not be reordered for display by Emacs.
 
 No.  The same standard encoding is used.  Braille is just another font.

Too bad.  Using fonts to change semantics is a bad idea.

In any case, this situation should have an automatic solution in
Emacs.  If someone reports enough details about it, a specialized
feature can be developed.

 To do otherwise would render regular Hebrew text inaccessible to the
 Hebrew speaking blind computer users.
 This font, however, needs to be rendered by bypassing the BiDi
 algorithm.

Bypassing reordering does not mean turn it off.  Emacs should do that
automatically.

  But even if the above does not solve the issue, I would expect people
  to request that Emacs does TRT by default with Hebrew Braille, rather
  than turning bidi off.
 
 What does TRT stand for?

The Right Thing.

 Turning BiDi off would be a satisfactory solution in this use case.

It's a kludge.  That variable exists for debugging and as a stop-gap
for situations that are currently not supported yet.  It should not be
built upon as _the_ solution.  Reporting the situations that need it
actively will go a long way towards making sure Emacs's support for
bidi is better.


___
Linux-il mailing list
Linux-il@cs.huji.ac.il
http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il

Re: Emacs Hebrew

[Eli Zaretskii]

 From: Omer Zak w...@zak.co.il
 Date: Wed, 13 Jun 2012 19:41:30 +0300
 
 On Wed, 2012-06-13 at 19:22 +0300, Eli Zaretskii wrote:
  And since the bidirectional display for Emacs was developed in almost
  complete isolation from this community -- not a single input or
  response to several design discussions I posted -- why would anyone
  assume that an essentially one-man project will not end up being a
  complete disaster from usability point of view?
 
 At the time I considered helping the effort.  However, Emacs is too big
 piece of software to dive into without investing serious time.

User experience and feature requests are an important feedback that I
sorely lacked.

 Emacs was good for editing such files, and I am
 glad that Emacs remains this way (by having the option to turn off BiDi
 algorithm - and I suggest to turn BiDi off by default in all major
 modes, which are used for programming language supporting Hebrew glyphs
 in constant strings, such as Python, Java, etc.).

If you write programs without any Hebrew characters, bidi reordering
should not reorder anything, so it's pointless to disable it.  If you
do use Hebrew character (e.g., in comments and strings), then you
_will_ want the reordering to get those readable.

Emacs needs one more feature to support bidi comments and strings 100%
correctly, but what's there now is IMO good enough already, certainly
when bidi characters are not used.  Why don't you try that?

___
Linux-il mailing list
Linux-il@cs.huji.ac.il
http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il

Re: Emacs Hebrew

[Eli Zaretskii]

 Date: Wed, 13 Jun 2012 23:33:27 +0300
 From: Nadav Har'El n...@math.technion.ac.il
 Cc: linux-il@cs.huji.ac.il
 
 As an example of the confusion that bidi might cause me when editing the
 Hspell source code, conside what it might to do a regular expression:
 Imagine that I wrote s/א/ב/ with the intension of switching aleph into bet,
 but (I just checked...) the bidi algorithm applied to it will display the
 order *reversed* making it confusingly appear that the code is changing bet
 into aleph...

The right thing is have an Emacs feature to help you in these
situations.  Not to turn bidi reordering off, because there might be
other parts of the same buffer (e.g., comments) where you _do_ want
reordering.

 I think you must agree that the unicode bidi algorithm was never designed
 for such use cases - and I'm not blaming these issues on your
 implementation, of course - and I don't even need to try your
 implementation to know it will have this - or similar - issues when
 editing source code. And editing mostly-ASCII source code is 99% of what I
 personally use Emacs for.

We are talking about Emacs, not about the UBA.  Emacs already uses the
high-level protocols fire escape provided by the UBA to redefine
what a paragraph is (the basic UBA says it's a physical line, which
would make the bidi display useless).  It could use such protocols
more, and it could use its advanced display features to produce better
display in these cases.  The key to having this is to use Emacs 24 and
report bugs and missing features.  By contrast, turning the reordering
off will never yield any improvement in this area, and is only a
stopgap solution never designed to be a real feature.

 Again, nothing I said or asked applies to writing *text*, such as email.
 For text, bidi is definitely required, and really welcome. I sorely miss a
 bidi-capable editor for my email writing - unfortunately I have a 20 year
 old habit of writing emails in vi, so it'll take me a bit of time to decide
 to switch to emacs also for emails. But now I know I definitely should!

I use Emacs 24 for my emails for quite some time now, and it works
well.


___
Linux-il mailing list
Linux-il@cs.huji.ac.il
http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il

Re: Emacs Hebrew

[Nadav Har'El]

On Thu, Jun 14, 2012, Eli Zaretskii wrote about Re: Emacs  Hebrew:
  I think you must agree that the unicode bidi algorithm was never designed
...
 We are talking about Emacs, not about the UBA.  Emacs already uses the
 high-level protocols fire escape provided by the UBA to redefine
 what a paragraph is (the basic UBA says it's a physical line, which
 would make the bidi display useless).  It could use such protocols
 more, and it could use its advanced display features to produce better
 display in these cases.  The key to having this is to use Emacs 24 and
 report bugs and missing features.  By contrast, turning the reordering
 off will never yield any improvement in this area, and is only a
 stopgap solution never designed to be a real feature.

Like I said, try the example Perl instruction of changing aleph into
bet:

s/א/ב/

See how the Bidi algorithm makes it appear as if we're changing the
other way around - bet into aleph. I honestly don't see what kind of
base direction choice or other high-level protocol can save this
case. I have to admit I didn't try it on Emacs 24 (which I don't have
yet), but did try it on other bidi-capable programs. Maybe there is a
workaround in this case, and you can educate me?

Thanks,

Nadav.

-- 
Nadav Har'El|  Thursday, Jun 14 2012, 
n...@math.technion.ac.il |-
Phone +972-523-790466, ICQ 13349191 |Guests, like fish, begin to smell after
http://nadav.harel.org.il   |three days. -- Benjamin Franklin

___
Linux-il mailing list
Linux-il@cs.huji.ac.il
http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il

Re: Emacs Hebrew

[Omer Zak]

On Tue, 2012-06-12 at 19:05 +0300, Eli Zaretskii wrote:
 You know, it is quite ironic that, having heard about a major Free
 Software project which now fully supports bidirectional scripts
 including Hebrew, the first thing people here ask is how to disable
 that feature.  Not whether it works, not if it's any good, not how
 well it supports this or that aspect of bidirectional editing -- but
 how to turn it off.  A sobering experience, I must say.

It is a question of control.

The problem is that Gedit implemented BiDi support without the option to
turn it off.

So when one wants to view BiDi text in visual order (the usual case),
one opens the file in Gedit.  And when one wants to see it in logical
order (e.g. to figure out how the visual order turned out to be so
messed up), one opened it in Emacs.

Another use case is by blind computer users, who prefer to use Braille.
All BiDi Braille text must be laid out in logical order.  So even if
your editor supports Hebrew Braille fonts, they are of no use if the
text is - too helpfully - printed out in visual order.

Few years ago, when Abiword people were debugging BiDi support, I asked
for an option to turn it off for the benefit of blind Hebrew computer
users.  This request was turned down.

--- Omer


-- 
Perrin's Principle of Inclusion:
The strength of any system is directly proportional to the
power of the tools it provides for the general public.
My own blog is at http://www.zak.co.il/tddpirate/

My opinions, as expressed in this E-mail message, are mine alone.
They do not represent the official policy of any organization with which
I may be affiliated in any way.
WARNING TO SPAMMERS:  at http://www.zak.co.il/spamwarning.html


___
Linux-il mailing list
Linux-il@cs.huji.ac.il
http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il

Re: Emacs Hebrew

[Nadav Har'El]

On Tue, Jun 12, 2012, Omer Zak wrote about Re: Emacs  Hebrew:
 On Tue, 2012-06-12 at 19:05 +0300, Eli Zaretskii wrote:
  You know, it is quite ironic that, having heard about a major Free
  Software project which now fully supports bidirectional scripts
  including Hebrew, the first thing people here ask is how to disable
  that feature.  Not whether it works, not if it's any good, not how
  well it supports this or that aspect of bidirectional editing -- but
  how to turn it off.  A sobering experience, I must say.

Life's strange, isn't it ;-)

But seriously, when someone announces a new feature, why would you
expect the first question to be is it any good or whether it works?
Obviously, if it weren't any good, or didn't actually work - it wouldn't
have been announced... At least, that's how it (usually) works in free
software (as opposed to the commercial software world) - people don't announce
things because of the marketing buzz this generates, but because they
are proud of the new feature, which they often created to scratch their
own itch.

Finally, I don't think the question of how to turn it off should
surprise you in a list of developers. Bidi is great for writing texts,
but since until now writing Hebrew text in Emacs wasn't a great idea,
people didn't do it. What they did do with Emacs is writing code,
editing config files, and similar things. With those, Bidi is sometimes
a distraction, not a desired feature - so people want to be able to turn
it off.

-- 
Nadav Har'El| Wednesday, Jun 13 2012, 
n...@math.technion.ac.il |-
Phone +972-523-790466, ICQ 13349191 |There are 2 ways to do it - my way and
http://nadav.harel.org.il   |the right way

___
Linux-il mailing list
Linux-il@cs.huji.ac.il
http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il

Re: Emacs Hebrew

[Nadav Har'El]

On Sun, Jun 10, 2012, Tzafrir Cohen wrote about Re: Emacs  Hebrew:
 The Bidi has landed!
 
 Quoting https://www.gnu.org/software/emacs/NEWS.24.1 :

Do you know if there's an option to NOT do Bidi, and just show
everything left-to-right as previously?

-- 
Nadav Har'El|Sunday, Jun 10 2012, 
n...@math.technion.ac.il |-
Phone +972-523-790466, ICQ 13349191 |Cats are smarter than dogs. You can't get
http://nadav.harel.org.il   |eight cats to pull a sled through snow.

___
Linux-il mailing list
Linux-il@cs.huji.ac.il
http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il

Re: Emacs Hebrew

[Omer Zak]

On Sun, 2012-06-10 at 18:56 +0300, Nadav Har'El wrote:
 On Sun, Jun 10, 2012, Tzafrir Cohen wrote about Re: Emacs  Hebrew:
  The Bidi has landed!
  
  Quoting https://www.gnu.org/software/emacs/NEWS.24.1 :
 
 Do you know if there's an option to NOT do Bidi, and just show
 everything left-to-right as previously?

+1

It is a very important option.  Sometimes I open a mixed language file
in Emacs just to see the logical order of glyphs in a segment which is
visually messed up.

Will Emacs 24.1 rob me of this use case?

--- Omer


-- 
Philip Machanick: caution: if you write code like this, immediately
after you are fired the person assigned to maintaining your code after
you leave will resign
My own blog is at http://www.zak.co.il/tddpirate/

My opinions, as expressed in this E-mail message, are mine alone.
They do not represent the official policy of any organization with which
I may be affiliated in any way.
WARNING TO SPAMMERS:  at http://www.zak.co.il/spamwarning.html


___
Linux-il mailing list
Linux-il@cs.huji.ac.il
http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il

Re: Emacs Hebrew

[Dov Grobgeld]

Sure, just do.

(setq bidi-display-reordering nil)

By default, it is on.

I'm using org-mode extensively during the last year, and getting Hebrew
support helps with writing recepies and taking care of the home financies.
There are still bugs though, e.g. like the interaction between org-tables
and bidi is really broken.

Regards,
Dov

On Sun, Jun 10, 2012 at 6:56 PM, Nadav Har'El n...@math.technion.ac.ilwrote:

 On Sun, Jun 10, 2012, Tzafrir Cohen wrote about Re: Emacs  Hebrew:
  The Bidi has landed!
 
  Quoting https://www.gnu.org/software/emacs/NEWS.24.1 :

 Do you know if there's an option to NOT do Bidi, and just show
 everything left-to-right as previously?

 --
 Nadav Har'El|Sunday, Jun 10
 2012,
 n...@math.technion.ac.il
 |-
 Phone +972-523-790466, ICQ 13349191 |Cats are smarter than dogs. You
 can't get
 http://nadav.harel.org.il   |eight cats to pull a sled through
 snow.

 ___
 Linux-il mailing list
 Linux-il@cs.huji.ac.il
 http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il

___
Linux-il mailing list
Linux-il@cs.huji.ac.il
http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il

Re: Emacs Hebrew

[Omer Zak]

Answering myself:
I found in the document, immediately after the BiDi support
announcement:

 New buffer-local variable `bidi-display-reordering'.
To disable display reordering in a buffer, change this to nil.

I suppose it answers my and Nadav's question.

--- Omer


On Sun, 2012-06-10 at 19:11 +0300, Omer Zak wrote:
 On Sun, 2012-06-10 at 18:56 +0300, Nadav Har'El wrote:
  On Sun, Jun 10, 2012, Tzafrir Cohen wrote about Re: Emacs  Hebrew:
   The Bidi has landed!
   
   Quoting https://www.gnu.org/software/emacs/NEWS.24.1 :
  
  Do you know if there's an option to NOT do Bidi, and just show
  everything left-to-right as previously?
 
 +1
 
 It is a very important option.  Sometimes I open a mixed language file
 in Emacs just to see the logical order of glyphs in a segment which is
 visually messed up.
 
 Will Emacs 24.1 rob me of this use case?

-- 
My Commodore 64 is suffering from slowness and insufficiency of memory;
and its display device is grievously short of pixels.  Can anyone help?
My own blog is at http://www.zak.co.il/tddpirate/

My opinions, as expressed in this E-mail message, are mine alone.
They do not represent the official policy of any organization with which
I may be affiliated in any way.
WARNING TO SPAMMERS:  at http://www.zak.co.il/spamwarning.html


___
Linux-il mailing list
Linux-il@cs.huji.ac.il
http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il

Re: Emacs Hebrew

[Dotan Cohen]

On Sun, Jun 10, 2012 at 6:44 PM, Tzafrir Cohen tzaf...@cohens.org.il wrote:
 Long ago, On Fri, Feb 20, 2009 at 07:38:34AM +0200, Yuval Hager wrote:
 Hi,

 As I my Emacs usage increased recently following the discovery of the amazing
 org-mode (highly recommended), I began wondering about Bidi in Emacs.

 The Bidi has landed!

 Quoting https://www.gnu.org/software/emacs/NEWS.24.1 :

 *** Emacs now supports display and editing of bidirectional text.
 Right-to-left (RTL) scripts, such as Arabic, Farsi, and Hebrew, are
 displayed in the correct visual order as expected by users of those
 scripts.  The display reordering is a full bidirectionality class
 implementation of the Unicode Bidirectional Algorithm (UBA).  Buffers
 with no RTL text should look exactly the same as before.

 ...

 *** New Hebrew translation of the Emacs Tutorial.
 Type `C-u C-h t' to choose it in case your language setup doesn't
 automatically select it.

 Congrats!


Very nice. I use VIM but I will look at Emacs now. VIM is no good for
typing in Hebrew

-- 
Dotan Cohen

http://gibberish.co.il
http://what-is-what.com

___
Linux-il mailing list
Linux-il@cs.huji.ac.il
http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il

Re: Emacs Hebrew

[Amit Aronovitch]

2009/2/20 Dov Grobgeld dov.grobg...@gmail.com

 There was a complex issue of emacs bidi that I tried to follow back in
 2002. The mailing list still up at:

 http://lists.gnu.org/archive/html/emacs-bidi/

 What I remember from the discussion was that fribidi and the Unicode Bidi
 algorithm were ruled out, because they were considered not to be
 sufficient for the emacs bidi needs. The discussion eventually reached
 such a technical complexity that I lost interest. In the end, appearently
 nothing came out of the effort and the bidi changes never made it into the
 trunk.

 It is probably a good idea to restart and try to copy the Gtk or the Qt
 behavior. Note that BiDi is a lot more than just the application of the
 algorithm. You also need to take into account things such as:

- Cursor movement.
- Internal splitting of the visual and the logical positions.
- Hit detection for mouse down.
- The fact that a continous logical selection may be one, two, or three
visual selections.
- How to deal with zero width characters. (Is there a view-control-code
mode in emacs?).

 I'd be willing to do the work, if someone else would fund it. I'm leaving
 my current employment in another few months, and I'm currently looking for
 large and small jobs.


I doubt that we can find serious funding for this issue - emacs is not very
fashionable these days and I don't know of any local organization that uses
it regularly as a major platform (although you can find Emacsers scattered
practically everywhere).
However, maybe we can try setting up a bounty, through Hamakor, or maybe
even through FSF.

Having said that, I am certainly interested in this issue. I use emacs for
daily work, but switch to gedit whenever there's Hebrew involved - not very
convenient (but managable since work is in English)...
If someone else starts working on this, I'll follow closely and maybe give a
hand (if I find time).

Amit
___
Linux-il mailing list
Linux-il@cs.huji.ac.il
http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il

Re: Emacs Hebrew

[Dov Grobgeld]

I agree that I don't believe chances are big of finding funding. And even if
there was funding I think there are more urgent BiDi issues like fixing bugs
in OpenOffice, Gtk, or KDE.

So unless I sit at home rolling my thumbs for too long, I'm afraid that I
won't have time looking into it.

Regards,
Dov

2009/2/21 Amit Aronovitch aronovi...@gmail.com

 2009/2/20 Dov Grobgeld dov.grobg...@gmail.com

 There was a complex issue of emacs bidi that I tried to follow back in
 2002. The mailing list still up at:

 http://lists.gnu.org/archive/html/emacs-bidi/

 What I remember from the discussion was that fribidi and the Unicode Bidi
 algorithm were ruled out, because they were considered not to be
 sufficient for the emacs bidi needs. The discussion eventually reached
 such a technical complexity that I lost interest. In the end, appearently
 nothing came out of the effort and the bidi changes never made it into the
 trunk.

 It is probably a good idea to restart and try to copy the Gtk or the Qt
 behavior. Note that BiDi is a lot more than just the application of the
 algorithm. You also need to take into account things such as:

- Cursor movement.
- Internal splitting of the visual and the logical positions.
- Hit detection for mouse down.
- The fact that a continous logical selection may be one, two, or
three visual selections.
- How to deal with zero width characters. (Is there a
view-control-code mode in emacs?).

 I'd be willing to do the work, if someone else would fund it. I'm leaving
 my current employment in another few months, and I'm currently looking for
 large and small jobs.


 I doubt that we can find serious funding for this issue - emacs is not very
 fashionable these days and I don't know of any local organization that uses
 it regularly as a major platform (although you can find Emacsers scattered
 practically everywhere).
 However, maybe we can try setting up a bounty, through Hamakor, or maybe
 even through FSF.

 Having said that, I am certainly interested in this issue. I use emacs for
 daily work, but switch to gedit whenever there's Hebrew involved - not very
 convenient (but managable since work is in English)...
 If someone else starts working on this, I'll follow closely and maybe give
 a hand (if I find time).

 Amit


___
Linux-il mailing list
Linux-il@cs.huji.ac.il
http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il

Re: Emacs Hebrew

[Yuval Hager]

On Sunday 22 February 2009, Dov Grobgeld wrote:
 I agree that I don't believe chances are big of finding funding. And even
 if there was funding I think there are more urgent BiDi issues like fixing
 bugs in OpenOffice, Gtk, or KDE.

 So unless I sit at home rolling my thumbs for too long, I'm afraid that I
 won't have time looking into it.


I guess you are right. Anyway, if you ever get to work on this sometime, call 
me up, maybe I can join the effort.

--y


signature.asc
Description: This is a digitally signed message part.
___
Linux-il mailing list
Linux-il@cs.huji.ac.il
http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il

Re: Emacs Hebrew

[Yuval Hager]

On Friday 20 February 2009, Omer Zak wrote:
 I thought about this about 1 1/2 years ago but did not get around to
 implementing it.

 XEmacs has support for loading external modules, however I use emacs.


Me too.

 In Emacs version 22, it is possible to use IPC to communicate with
 another process.  Then one needs to write a module which invokes FriBidi
 functions and have an emacs mode, which communicates with the module
 when the buffer's contents need to be rendered.


I saw this: http://www.emacswiki.org/cgi-bin/emacs/poor-mans-bidi.el

But I'd rather have a rich mans bidi.

What about Mule? Can it be used with the standard Emacs?

--y


signature.asc
Description: This is a digitally signed message part.
___
Linux-il mailing list
Linux-il@cs.huji.ac.il
http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il

Re: Emacs Hebrew

[Dov Grobgeld]

There was a complex issue of emacs bidi that I tried to follow back in 2002.
The mailing list still up at:

http://lists.gnu.org/archive/html/emacs-bidi/

What I remember from the discussion was that fribidi and the Unicode Bidi
algorithm were ruled out, because they were considered not to be
sufficient for the emacs bidi needs. The discussion eventually reached
such a technical complexity that I lost interest. In the end, appearently
nothing came out of the effort and the bidi changes never made it into the
trunk.

It is probably a good idea to restart and try to copy the Gtk or the Qt
behavior. Note that BiDi is a lot more than just the application of the
algorithm. You also need to take into account things such as:

   - Cursor movement.
   - Internal splitting of the visual and the logical positions.
   - Hit detection for mouse down.
   - The fact that a continous logical selection may be one, two, or three
   visual selections.
   - How to deal with zero width characters. (Is there a view-control-code
   mode in emacs?).

I'd be willing to do the work, if someone else would fund it. I'm leaving my
current employment in another few months, and I'm currently looking for
large and small jobs.

Regards,
Dov

2009/2/20 Yuval Hager yu...@avramzon.net

 On Friday 20 February 2009, Omer Zak wrote:
  I thought about this about 1 1/2 years ago but did not get around to
  implementing it.
 
  XEmacs has support for loading external modules, however I use emacs.
 

 Me too.

  In Emacs version 22, it is possible to use IPC to communicate with
  another process.  Then one needs to write a module which invokes FriBidi
  functions and have an emacs mode, which communicates with the module
  when the buffer's contents need to be rendered.
 

 I saw this: http://www.emacswiki.org/cgi-bin/emacs/poor-mans-bidi.el

 But I'd rather have a rich mans bidi.

 What about Mule? Can it be used with the standard Emacs?

 --y

 ___
 Linux-il mailing list
 Linux-il@cs.huji.ac.il
 http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il


___
Linux-il mailing list
Linux-il@cs.huji.ac.il
http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il

Re: Emacs Hebrew

[Omer Zak]

I thought about this about 1 1/2 years ago but did not get around to
implementing it.

XEmacs has support for loading external modules, however I use emacs.

In Emacs version 22, it is possible to use IPC to communicate with
another process.  Then one needs to write a module which invokes FriBidi
functions and have an emacs mode, which communicates with the module
when the buffer's contents need to be rendered.

Are there more people interested in BiDi in Emacs, and can join the
effort of designing and implementing it?
 --- Omer


On Fri, 2009-02-20 at 07:38 +0200, Yuval Hager wrote:
 Hi,
 
 As I my Emacs usage increased recently following the discovery of the amazing 
 org-mode (highly recommended), I began wondering about Bidi in Emacs.
 Searching for info on the web only got me confused - it seems there have been 
 a number of attempts, but it is not clear to me if I can get bidi working 
 (for example in gnus) and how simple/hacky is that.
 
 Can anybody update on the latest emacs bidi status, or simply provide a 
 pointer I missed?

-- 
Delay is the deadliest form of denial.C. Northcote Parkinson
My own blog is at http://www.zak.co.il/tddpirate/

My opinions, as expressed in this E-mail message, are mine alone.
They do not represent the official policy of any organization with which
I may be affiliated in any way.
WARNING TO SPAMMERS:  at http://www.zak.co.il/spamwarning.html


___
Linux-il mailing list
Linux-il@cs.huji.ac.il
http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il