subject:"Re\: LyX, XeTeX, bidi and Hebrew"

Re: LyX, XeTeX, bidi and Hebrew

2012-07-22 Thread Jürgen Spitzmüller

Ronen Abravanel wrote:
 OK. I fixed it.
 
 (All the functions mentioned below are from Paragraph.cpp)
 
 It seems like there is a proper function to do it, getUChar, and I just had
 to call it from the proper place.
 
 I called it from latexSpecialChar, and it works, but in order to do so, I
 hat to pass BufferParams to latexSpecialChar, which is ugly.
 
 Diff file and examples can be found here:
 http://www.technion.ac.il/~ronen/temp/bidi_Tex/

Ronen, I had a look at this and, without knowing much of Bidi, I came up with 
the attached slightly changes (see http://www.lyx.org/trac/ticket/8251).

I think that the \beginL and \L should be supressed when we use polyglossia 
(i.e., _not_ suppressed when XeTeX is used with babel), notwithstanding the 
language. This is already done at other parts of the source and just forgotten 
here. However, this needs more testing from someone who is more familiar with 
the script.

The other changes are optimizations.

Could you test if this works with XeTeX, with and without polyglossia and with 
and without TeX fonts.

BTW, it seems that Hebrew does not yet work with LuaTeX, does it?

Jürgendiff --git a/src/Paragraph.cpp b/src/Paragraph.cpp
index d93594a..979dcfb 100644
--- a/src/Paragraph.cpp
+++ b/src/Paragraph.cpp
@@ -330,6 +330,7 @@ public:
 	///
 	void latexSpecialChar(
    otexstream  os,
+   BufferParams const  bparams,
    OutputParams const  runparams,
    Font const  running_font,
    Change const  running_change,
@@ -1076,6 +1077,7 @@ void Paragraph::Private::latexInset(BufferParams const  bparams,
 	odocstream::pos_type const len = os.os().tellp();
 
 	if (inset-forceLTR()
+	 !runparams.use_polyglossia
 	 running_font.isRightToLeft()
 	// ERT is an exception, it should be output with no
 	// decorations at all
@@ -1155,6 +1157,7 @@ void Paragraph::Private::latexInset(BufferParams const  bparams,
 
 
 void Paragraph::Private::latexSpecialChar(otexstream  os,
+	  BufferParams const  bparams,
 	  OutputParams const  runparams,
 	  Font const  running_font,
 	  Change const  running_change,
@@ -1163,7 +1166,8 @@ void Paragraph::Private::latexSpecialChar(otexstream  os,
 	  pos_type end_pos,
 	  unsigned int  column)
 {
-	char_type const c = text_[i];
+	char_type const c = (runparams.use_polyglossia) ?
+		owner_-getUChar(bparams, i) : text_[i];
 
 	if (style.pass_thru || runparams.pass_thru) {
 		if (c != '\0') {
@@ -1896,7 +1900,7 @@ FontSize Paragraph::highestFontInRange
 char_type Paragraph::getUChar(BufferParams const  bparams, pos_type pos) const
 {
 	char_type c = d-text_[pos];
-	if (!lyxrc.rtl_support)
+	if (!lyxrc.rtl_support || !getFontSettings(bparams, pos).isRightToLeft())
 		return c;
 
 	char_type uc = c;
@@ -1926,9 +1930,8 @@ char_type Paragraph::getUChar(BufferParams const  bparams, pos_type pos) const
 		uc = '';
 		break;
 	}
-	if (uc != c  getFontSettings(bparams, pos).isRightToLeft())
-		return uc;
-	return c;
+
+	return uc;
 }
 
 
@@ -2566,7 +2569,7 @@ void Paragraph::latex(BufferParams const  bparams,
 		} else {
 			if (i = start_pos  (end_pos == -1 || i  end_pos)) {
 try {
-	d-latexSpecialChar(os, rp, running_font, runningChange,
+	d-latexSpecialChar(os, bparams, rp, running_font, runningChange,
 			style, i, end_pos, column);
 } catch (EncodingException  e) {
 if (runparams.dryrun) {

Re: LyX, XeTeX, bidi and Hebrew

2012-07-22 Thread Jürgen Spitzmüller

Ronen Abravanel wrote:
> OK. I fixed it.
> 
> (All the functions mentioned below are from Paragraph.cpp)
> 
> It seems like there is a proper function to do it, getUChar, and I just had
> to call it from the proper place.
> 
> I called it from latexSpecialChar, and it works, but in order to do so, I
> hat to pass BufferParams to latexSpecialChar, which is ugly.
> 
> Diff file and examples can be found here:
> http://www.technion.ac.il/~ronen/temp/bidi_Tex/

Ronen, I had a look at this and, without knowing much of Bidi, I came up with 
the attached slightly changes (see http://www.lyx.org/trac/ticket/8251).

I think that the \beginL and \L should be supressed when we use polyglossia 
(i.e., _not_ suppressed when XeTeX is used with babel), notwithstanding the 
language. This is already done at other parts of the source and just forgotten 
here. However, this needs more testing from someone who is more familiar with 
the script.

The other changes are optimizations.

Could you test if this works with XeTeX, with and without polyglossia and with 
and without TeX fonts.

BTW, it seems that Hebrew does not yet work with LuaTeX, does it?

Jürgendiff --git a/src/Paragraph.cpp b/src/Paragraph.cpp
index d93594a..979dcfb 100644
--- a/src/Paragraph.cpp
+++ b/src/Paragraph.cpp
@@ -330,6 +330,7 @@ public:
 	///
 	void latexSpecialChar(
    otexstream & os,
+   BufferParams const & bparams,
    OutputParams const & runparams,
    Font const & running_font,
    Change const & running_change,
@@ -1076,6 +1077,7 @@ void Paragraph::Private::latexInset(BufferParams const & bparams,
 	odocstream::pos_type const len = os.os().tellp();
 
 	if (inset->forceLTR()
+	&& !runparams.use_polyglossia
 	&& running_font.isRightToLeft()
 	// ERT is an exception, it should be output with no
 	// decorations at all
@@ -1155,6 +1157,7 @@ void Paragraph::Private::latexInset(BufferParams const & bparams,
 
 
 void Paragraph::Private::latexSpecialChar(otexstream & os,
+	  BufferParams const & bparams,
 	  OutputParams const & runparams,
 	  Font const & running_font,
 	  Change const & running_change,
@@ -1163,7 +1166,8 @@ void Paragraph::Private::latexSpecialChar(otexstream & os,
 	  pos_type end_pos,
 	  unsigned int & column)
 {
-	char_type const c = text_[i];
+	char_type const c = (runparams.use_polyglossia) ?
+		owner_->getUChar(bparams, i) : text_[i];
 
 	if (style.pass_thru || runparams.pass_thru) {
 		if (c != '\0') {
@@ -1896,7 +1900,7 @@ FontSize Paragraph::highestFontInRange
 char_type Paragraph::getUChar(BufferParams const & bparams, pos_type pos) const
 {
 	char_type c = d->text_[pos];
-	if (!lyxrc.rtl_support)
+	if (!lyxrc.rtl_support || !getFontSettings(bparams, pos).isRightToLeft())
 		return c;
 
 	char_type uc = c;
@@ -1926,9 +1930,8 @@ char_type Paragraph::getUChar(BufferParams const & bparams, pos_type pos) const
 		uc = '<';
 		break;
 	}
-	if (uc != c && getFontSettings(bparams, pos).isRightToLeft())
-		return uc;
-	return c;
+
+	return uc;
 }
 
 
@@ -2566,7 +2569,7 @@ void Paragraph::latex(BufferParams const & bparams,
 		} else {
 			if (i >= start_pos && (end_pos == -1 || i < end_pos)) {
 try {
-	d->latexSpecialChar(os, rp, running_font, runningChange,
+	d->latexSpecialChar(os, bparams, rp, running_font, runningChange,
 			style, i, end_pos, column);
 } catch (EncodingException & e) {
 if (runparams.dryrun) {

Re: LyX, XeTeX, bidi and Hebrew

2012-07-21 Thread Guenter Milde

On 2012-07-19, Ronen Abravanel wrote:

 XeTexX get in in the modern (unicode) sense of right.
 LyX, by default, input it wrong, but LyX output is good for pdfLaTeX.

 LyX fixes it's output in the plain-text output, and my patch apply the same
 correction for Hebrew-XeTeX output...

I still consider it better to fix the output to pdflatex, so that the input
can be modern.

Günter

Re: LyX, XeTeX, bidi and Hebrew

2012-07-21 Thread Guenter Milde

On 2012-07-19, Ronen Abravanel wrote:

> XeTexX get in in the modern (unicode) sense of right.
> LyX, by default, input it "wrong", but LyX output is good for pdfLaTeX.

> LyX fixes it's output in the plain-text output, and my patch apply the same
> correction for Hebrew-XeTeX output...

I still consider it better to fix the output to pdflatex, so that the input
can be "modern".

Günter

Re: LyX, XeTeX, bidi and Hebrew

2012-07-19 Thread Guenter Milde

On 2012-07-18, Ronen Abravanel wrote:
 On Wed, Jul 18, 2012 at 11:09 AM, Guenter Milde mi...@users.sf.net wrote:
 On 2012-07-16, Ronen Abravanel wrote:

...

 * What do you get if the example is written as

 שלום (שלום)

   in LyX?

 Just Hello (Hello) in hebrew
 In LyX it's
 שלום (שלום)


 * How would you write the same in OpenOffice or some other word
 processor?

 If I *write* the same thing in a different word processor (OO, ms word or
 other bidi-supported WP) I get the brace proper. If I paste the tex-code
 into a word processor, it's inverted as in the PDF.

This is why I wonder whether XeTeX gets it right (i.e. compatible with other
applications) and pdfLaTeX does it wrong.

Then, instead of fixing XeTeX input, we would need a fix for the export
to traditional TeX and a way to convert existing LyX documents.

Günter

Re: LyX, XeTeX, bidi and Hebrew

2012-07-19 Thread Ronen Abravanel

On Thu, Jul 19, 2012 at 11:17 AM, Guenter Milde mi...@users.sf.net wrote:

 On 2012-07-18, Ronen Abravanel wrote:
  On Wed, Jul 18, 2012 at 11:09 AM, Guenter Milde mi...@users.sf.net
 wrote:
  On 2012-07-16, Ronen Abravanel wrote:

 ...

  * What do you get if the example is written as

  שלום (שלום)

in LyX?

  Just Hello (Hello) in hebrew
  In LyX it's
  שלום (שלום)


  * How would you write the same in OpenOffice or some other word
  processor?

  If I *write* the same thing in a different word processor (OO, ms word or
  other bidi-supported WP) I get the brace proper. If I paste the tex-code
  into a word processor, it's inverted as in the PDF.

 This is why I wonder whether XeTeX gets it right (i.e. compatible with
 other
 applications) and pdfLaTeX does it wrong.

 Then, instead of fixing XeTeX input, we would need a fix for the export
 to traditional TeX and a way to convert existing LyX documents.

 Günter


XeTexX get in in the modern (unicode) sense of right.
LyX, by default, input it wrong, but LyX output is good for pdfLaTeX.

LyX fixes it's output in the plain-text output, and my patch apply the same
correction for Hebrew-XeTeX output...

Ronen

Re: LyX, XeTeX, bidi and Hebrew

2012-07-19 Thread Guenter Milde

On 2012-07-18, Ronen Abravanel wrote:
> On Wed, Jul 18, 2012 at 11:09 AM, Guenter Milde  wrote:
>> On 2012-07-16, Ronen Abravanel wrote:

...

>> * What do you get if the example is written as

>> שלום (שלום)

>>   in LyX?

> Just "Hello (Hello)" in hebrew
> In LyX it's
> שלום (שלום)


>> * How would you write the same in OpenOffice or some other "word
>> processor"?

> If I *write* the same thing in a different word processor (OO, ms word or
> other bidi-supported WP) I get the brace proper. If I paste the tex-code
> into a word processor, it's inverted as in the PDF.

This is why I wonder whether XeTeX gets it right (i.e. compatible with other
applications) and pdfLaTeX does it wrong.

Then, instead of "fixing" XeTeX input, we would need a fix for the export
to traditional TeX and a way to convert existing LyX documents.

Günter

Re: LyX, XeTeX, bidi and Hebrew

2012-07-19 Thread Ronen Abravanel

On Thu, Jul 19, 2012 at 11:17 AM, Guenter Milde  wrote:

> On 2012-07-18, Ronen Abravanel wrote:
> > On Wed, Jul 18, 2012 at 11:09 AM, Guenter Milde 
> wrote:
> >> On 2012-07-16, Ronen Abravanel wrote:
>
> ...
>
> >> * What do you get if the example is written as
>
> >> שלום (שלום)
>
> >>   in LyX?
>
> > Just "Hello (Hello)" in hebrew
> > In LyX it's
> > שלום (שלום)
>
>
> >> * How would you write the same in OpenOffice or some other "word
> >> processor"?
>
> > If I *write* the same thing in a different word processor (OO, ms word or
> > other bidi-supported WP) I get the brace proper. If I paste the tex-code
> > into a word processor, it's inverted as in the PDF.
>
> This is why I wonder whether XeTeX gets it right (i.e. compatible with
> other
> applications) and pdfLaTeX does it wrong.
>
> Then, instead of "fixing" XeTeX input, we would need a fix for the export
> to traditional TeX and a way to convert existing LyX documents.
>
> Günter
>
>
XeTexX get in in the modern (unicode) sense of right.
LyX, by default, input it "wrong", but LyX output is good for pdfLaTeX.

LyX fixes it's output in the plain-text output, and my patch apply the same
correction for Hebrew-XeTeX output...

Ronen

Re: LyX, XeTeX, bidi and Hebrew

2012-07-18 Thread Guenter Milde

On 2012-07-16, Ronen Abravanel wrote:

 [-- Type: text/plain, Encoding: quoted-printable --]

 These files produced directly by the GIT version of LyX.
 As you can see, the braces are the inverted in both the PDF and the TeX
 files.

However, they are inverted in the source file too!

* What do you get if the example is written as

שלום (שלום)

  in LyX?

* How would you write the same in OpenOffice or some other word processor?

After all, this is rather a XeTeX vs. pdfLaTeX problem: what does an
internet recherche for Hebrew polyglossia XeTeX reveal? Did you also try
LuaTeX?

Günter

Re: LyX, XeTeX, bidi and Hebrew

2012-07-18 Thread Ronen Abravanel

On Wed, Jul 18, 2012 at 11:09 AM, Guenter Milde mi...@users.sf.net wrote:

 On 2012-07-16, Ronen Abravanel wrote:

  [-- Type: text/plain, Encoding: quoted-printable --]

  These files produced directly by the GIT version of LyX.
  As you can see, the braces are the inverted in both the PDF and the TeX
  files.

 However, they are inverted in the source file too!

 * What do you get if the example is written as

 שלום (שלום)

   in LyX?

Just Hello (Hello) in hebrew
In LyX it's
שלום (שלום)


 * How would you write the same in OpenOffice or some other word
 processor?

If I *write* the same thing in a different word processor (OO, ms word or
other bidi-supported WP) I get the brace proper. If I paste the tex-code
into a word processor, it's inverted as in the PDF.


 After all, this is rather a XeTeX vs. pdfLaTeX problem: what does an
 internet recherche for Hebrew polyglossia XeTeX reveal? Did you also try
 LuaTeX?


Never seen working LuaTeX in Hebrew.


 Günter



Anyway, I fixed it (patch in my previews mail).
The bug is in the old latex\latex export: the hebrew support in LaTeX
was broken so it was fixed in LyX. now, XeTeX need to get proper text so
LyX should produce proper unicode file.

Ronen.

Re: LyX, XeTeX, bidi and Hebrew

2012-07-18 Thread Guenter Milde

On 2012-07-16, Ronen Abravanel wrote:

> [-- Type: text/plain, Encoding: quoted-printable --]

> These files produced directly by the GIT version of LyX.
> As you can see, the braces are the inverted in both the PDF and the TeX
> files.

However, they are inverted in the source file too!

* What do you get if the example is written as

שלום (שלום)

  in LyX?

* How would you write the same in OpenOffice or some other "word processor"?

After all, this is rather a XeTeX vs. pdfLaTeX problem: what does an
internet recherche for "Hebrew polyglossia XeTeX" reveal? Did you also try
LuaTeX?

Günter

Re: LyX, XeTeX, bidi and Hebrew

2012-07-18 Thread Ronen Abravanel

On Wed, Jul 18, 2012 at 11:09 AM, Guenter Milde  wrote:

> On 2012-07-16, Ronen Abravanel wrote:
>
> > [-- Type: text/plain, Encoding: quoted-printable --]
>
> > These files produced directly by the GIT version of LyX.
> > As you can see, the braces are the inverted in both the PDF and the TeX
> > files.
>
> However, they are inverted in the source file too!
>
> * What do you get if the example is written as
>
> שלום (שלום)
>
>   in LyX?
>
Just "Hello (Hello)" in hebrew
In LyX it's
שלום (שלום)


> * How would you write the same in OpenOffice or some other "word
> processor"?
>
If I *write* the same thing in a different word processor (OO, ms word or
other bidi-supported WP) I get the brace proper. If I paste the tex-code
into a word processor, it's inverted as in the PDF.


> After all, this is rather a XeTeX vs. pdfLaTeX problem: what does an
> internet recherche for "Hebrew polyglossia XeTeX" reveal? Did you also try
> LuaTeX?
>

Never seen working LuaTeX in Hebrew.

>
> Günter
>
>

Anyway, I fixed it (patch in my previews mail).
The "bug" is in the "old" latex\latex export: the hebrew support in LaTeX
was broken so it was fixed in LyX. now, XeTeX need to get proper text so
LyX should produce proper unicode file.

Ronen.

Re: LyX, XeTeX, bidi and Hebrew

2012-07-16 Thread Ronen Abravanel

These files produced directly by the GIT version of LyX.
As you can see, the braces are the inverted in both the PDF and the TeX
files.

Ronen.

On Fri, Jul 13, 2012 at 9:38 PM, Ronen Abravanel ron...@gmail.com wrote:

1. you are right. one can do that, and it would be a nice workaround. But
I prefer lyx will produce clean TeX file.
2. Attached some examples:
the text in the *xetex and *pdf files are identical. in the pdf, you can
see that the braces are rendered backword, and also if you will open the
.tex file in unicode-enables editor.

note: I cheated a little. In my work-computer I have lyx 2.1 development
version, and the xetex file compiled straight forward. In my lyx 2.03 in my
laptop, I had to move one line in the preamble.

Ronen

On Fri, Jul 13, 2012 at 11:00 AM, Guenter Milde mi...@users.sf.netwrote:

On 2012-07-12, Ronen Abravanel wrote:

as the Hebrew support of LaTeX is not developed for the last several
years, I decided to try and export Hebrew document to PDF using XeTeX,

Are there problems that did not get solved for years, or is there no
development because it just works?

where the grate Bidi packge by Vafa Khalighi is under active
development, and merged in polyglossia.

LyX 2.0 support XeTeX support, but there are some slight problem:

1. when exporting to regular latex, one should wrap english text
(like
numbers, math, etc..) inside Hebrew parts by \L{.,, }.

Do you do this with raw LaTeX (ERT) or is there an inset for the task
(should be easy to write one if its not already there).

not true in XeTeX. Attached an ugly patch that removes such \L, but:
a. it's pretty ugly. I'm sure there is better way to do it. I will be
happy
for suggestions about ways to improve it.

For documents that shall work in both, pdflatex and xelatex, it might be
best to have a dummy definition like

\ifxetex
\providecommand*{\L}[1]{#1}
\fi

either in the documents LaTeX preamble or in the module defining the text
inset).

...

2. LyX making some problems with the braces { } ( ) etc in Right-to-left
parts:
When one writes Right-to-left text that should look like
(1) text ( inside braces ) more text
it outputs as
(2) text ) inside braces ( more text
or, to be more accurate,
עברית ( עוד טקסט this part is INSIDE עוד טקסט ) בחוץ this part is
outside
. בחוץ
Will be output as
עברית ) עוד טקסט this part is INSIDE עוד טקסט ( בחוץ this part is
outside
. בחוץ
It all looks OK when rendering using pdflatex, but using XeTeX, it
appears
wrong.
I couldn't find the part in LyX that handle that part. I will be happy
for
directions regarding where to look.

I don't know a solution for this part. It looks like the two engines do
not agree on whether ( and ) are opening vs. closing or right vs.
left parantheses.

You may try to create a minimal *.tex exampe file that works with XeTeX
and import this to LyX or try to recreate the correct syntax in LyX
(comparing the ViewSource output with the correct XeTeX source).

Günter

brase_test_lyx_git.lyx
Description: Binary data

brase_test_lyx_git.pdf
Description: Adobe PDF document

brase_test_lyx_git.tex
Description: TeX document

Re: LyX, XeTeX, bidi and Hebrew

2012-07-16 Thread Ronen Abravanel

OK. I fixed it.

(All the functions mentioned below are from Paragraph.cpp)

It seems like there is a proper function to do it, getUChar, and I just had
to call it from the proper place.

I called it from latexSpecialChar, and it works, but in order to do so, I
hat to pass BufferParams to latexSpecialChar, which is ugly.

Diff file and examples can be found here:
http://www.technion.ac.il/~ronen/temp/bidi_Tex/

The patch is very specific and very ugly. How should I improve it in order
to be able to merge it into LyX's upstream?

Thanks,
Ronen.

On Mon, Jul 16, 2012 at 5:22 PM, Ronen Abravanel ron...@gmail.com wrote: