Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".
Guenter Milde wrote: > I assume, heb1.tex is "splash_good.tex" and heb2.tex is "splash_bad.tex", > right? I guess so. My newsreader did not display any names for the attachments at Scotts message. > >> So, the real diferences of this change are: > >> 1) LyX writes cp1255 encoding instead of utf8 > > Yes. This is intended -- if document source is set to use 8-bit TeX fonts, > the input encoding should be the same for pdftex and luatex. > (Before, it was forced to be utf8-plain with luatex and xetex, which led > to many errors (or wrong output).) Thanks for the explanation. I did not think of the \inputencoding setting of the LyX file. > >> 2) LyX adds a \cyrtext command > > If I read the diff correctly, the \cyrtext was added before (heb1) but is > now only added if required. Right, I mixed this up with the first version of the diff that I did the other way round. Georg
Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".
On Sun, Feb 14, 2016 at 08:49:26PM +, Guenter Milde wrote: > However, I can confirm that the changes are intended. Thanks for checking. Scott signature.asc Description: PGP signature
Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".
On 2016-02-14, Georg Baum wrote: > Scott Kostyshak wrote: >> On Sat, Feb 13, 2016 at 08:54:57PM +, Guenter Milde wrote: >>> On 2016-02-13, Scott Kostyshak wrote: >>> > After this commit, when I run the command >>> > lyx -e luatex lib/doc/he/Intro.lyx >>> > it either fails to produce Intro.tex or (on current master) it produces >>> > a Intro.tex with gibberish where there should be Hebrew (at least this >>> > is what Vim shows me). > It is not gibberish. vim fails to detect the encoding cp1255. If you open > the file with kate it correctly detects the encoding and displays hebrew > text. Thanks for finding this out. >> Do you think it is OK that we now produce splash_bad.tex and before we >> produced splash_good.tex? If you tell me this is fine and that there >> is no encoding issue, then I trust your judgment since I know nothing >> about encoding. > Attached is a diff that was produced after converting the utf8 encoded file > to cp1255: > recode utf8..cp1255 < heb1.tex > heb1-cp1255.tex > diff -ub heb2.tex heb1-cp1255.tex > x.diff I assume, heb1.tex is "splash_good.tex" and heb2.tex is "splash_bad.tex", right? > So, the real diferences of this change are: > 1) LyX writes cp1255 encoding instead of utf8 Yes. This is intended -- if document source is set to use 8-bit TeX fonts, the input encoding should be the same for pdftex and luatex. (Before, it was forced to be utf8-plain with luatex and xetex, which led to many errors (or wrong output).) > 2) LyX adds a \cyrtext command If I read the diff correctly, the \cyrtext was added before (heb1) but is now only added if required. > I don't know if 1) is wanted, but the way it is done is correct: The file is > written in cp1255 encoding, and luainputenc is loaded with the same > encoding. > 2) looks unnneeded to me. This is fixed sice some months now. > Obviously, luatex+hebrew is not completely broken, since the utf8 version > worked. Did the utf8 version actually compile and give the correct result? > What is the advantage of using cp1255 instead of utf8 now? Less surprise for the user: if the document is set to use "language default", is uses "language default" (which is cp1255 for Hebrew). If the document is set to use utf8, it uses utf8. The advantage might not be visible in this example, but, e.g. with external files (bibtex, ...) the encoding of the generated latex source can be important. > --- heb2.tex 2016-02-14 10:35:23.083348413 +0100 > +++ heb1-cp1255.tex 2016-02-14 10:51:29.859358650 +0100 > @@ -2,7 +2,7 @@ > %% Do not edit unless you really know what you are doing. > \documentclass[english,hebrew]{article} > \usepackage[T1]{fontenc} > -\usepackage[cp1255]{luainputenc} > +\usepackage[utf8]{luainputenc} > \usepackage{url} > \usepackage{amssymb} > @@ -10,6 +10,10 @@ > %% LyX specific LaTeX commands. > \providecommand{\LyX}{L\kern-.1667em\lower.25em\hbox{Y}\kern-.125emX\@} > +\DeclareRobustCommand{\cyrtext}{% > + \fontencoding{T2A}\selectfont\def\encodingdefault{T2A}} > +\DeclareRobustCommand{\textcyr}[1]{\leavevmode{\cyrtext #1}} > + > %% Textclass specific LaTeX commands. > \usepackage{theorem} This does not solve the case of not-working LaTeX export. However, I can confirm that the changes are intended. (The disadvantages of the current input encoding default are a different topic that are handled in a different bug report ...) Thanks, Günter
Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".
On Sun, Feb 14, 2016 at 11:01:06AM +0100, Georg Baum wrote: > Scott Kostyshak wrote: > > > On Sat, Feb 13, 2016 at 08:54:57PM +, Guenter Milde wrote: > >> On 2016-02-13, Scott Kostyshak wrote: > >> > >> > After this commit, when I run the command > >> > >> > lyx -e luatex lib/doc/he/Intro.lyx > >> > >> > it either fails to produce Intro.tex or (on current master) it produces > >> > a Intro.tex with gibberish where there should be Hebrew (at least this > >> > is what Vim shows me). > > It is not gibberish. vim fails to detect the encoding cp1255. If you open > the file with kate it correctly detects the encoding and displays hebrew > text. OK I see. > >> No, as Luatex+Hebrew failing is an upstream bug. > > > > But I am just talking about exporting to .tex and opening in a text > > editor. LuaTeX does not even have a chance if our export to .tex > > corrupts the Hebrew. To be clear, attached are splash_good.tex and > > splash_bad.tex. Do you think it is OK that we now produce splash_bad.tex > > and before we produced splash_good.tex? If you tell me this is fine and > > that there is no encoding issue, then I trust your judgment since I > > know nothing about encoding. > > Attached is a diff that was produced after converting the utf8 encoded file > to cp1255: > > recode utf8..cp1255 < heb1.tex > heb1-cp1255.tex > diff -ub heb2.tex heb1-cp1255.tex > x.diff > > (recode adds windows line ends, therefore the -b argument for diff) Thanks for walking me through this. I did not know about recode. > So, the real diferences of this change are: > > 1) LyX writes cp1255 encoding instead of utf8 > > 2) LyX adds a \cyrtext command > > I don't know if 1) is wanted, but the way it is done is correct: The file is > written in cp1255 encoding, and luainputenc is loaded with the same > encoding. 2) looks unnneeded to me. > > Obviously, luatex+hebrew is not completely broken, since the utf8 version > worked. What is the advantage of using cp1255 instead of utf8 now? Thanks for looking into this. Scott signature.asc Description: PGP signature
Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".
Scott Kostyshak wrote: > On Sat, Feb 13, 2016 at 08:54:57PM +, Guenter Milde wrote: >> On 2016-02-13, Scott Kostyshak wrote: >> >> > After this commit, when I run the command >> >> > lyx -e luatex lib/doc/he/Intro.lyx >> >> > it either fails to produce Intro.tex or (on current master) it produces >> > a Intro.tex with gibberish where there should be Hebrew (at least this >> > is what Vim shows me). It is not gibberish. vim fails to detect the encoding cp1255. If you open the file with kate it correctly detects the encoding and displays hebrew text. >> No, as Luatex+Hebrew failing is an upstream bug. > > But I am just talking about exporting to .tex and opening in a text > editor. LuaTeX does not even have a chance if our export to .tex > corrupts the Hebrew. To be clear, attached are splash_good.tex and > splash_bad.tex. Do you think it is OK that we now produce splash_bad.tex > and before we produced splash_good.tex? If you tell me this is fine and > that there is no encoding issue, then I trust your judgment since I > know nothing about encoding. Attached is a diff that was produced after converting the utf8 encoded file to cp1255: recode utf8..cp1255 < heb1.tex > heb1-cp1255.tex diff -ub heb2.tex heb1-cp1255.tex > x.diff (recode adds windows line ends, therefore the -b argument for diff) So, the real diferences of this change are: 1) LyX writes cp1255 encoding instead of utf8 2) LyX adds a \cyrtext command I don't know if 1) is wanted, but the way it is done is correct: The file is written in cp1255 encoding, and luainputenc is loaded with the same encoding. 2) looks unnneeded to me. Obviously, luatex+hebrew is not completely broken, since the utf8 version worked. What is the advantage of using cp1255 instead of utf8 now? Georg --- heb2.tex 2016-02-14 10:35:23.083348413 +0100 +++ heb1-cp1255.tex 2016-02-14 10:51:29.859358650 +0100 @@ -2,7 +2,7 @@ %% Do not edit unless you really know what you are doing. \documentclass[english,hebrew]{article} \usepackage[T1]{fontenc} -\usepackage[cp1255]{luainputenc} +\usepackage[utf8]{luainputenc} \usepackage{url} \usepackage{amssymb} @@ -10,6 +10,10 @@ %% LyX specific LaTeX commands. \providecommand{\LyX}{L\kern-.1667em\lower.25em\hbox{Y}\kern-.125emX\@} +\DeclareRobustCommand{\cyrtext}{% + \fontencoding{T2A}\selectfont\def\encodingdefault{T2A}} +\DeclareRobustCommand{\textcyr}[1]{\leavevmode{\cyrtext #1}} + %% Textclass specific LaTeX commands. \usepackage{theorem}
Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".
On Sat, Feb 13, 2016 at 08:54:57PM +, Guenter Milde wrote: > On 2016-02-13, Scott Kostyshak wrote: > > On Tue, Oct 20, 2015 at 07:16:44PM +0200, Günter Milde wrote: > >> commit 1523fc6023440f10ca0d82a681ded5c060d8fd33 > >> Author: Günter Milde > >> Date: Tue Oct 20 19:14:39 2015 +0200 > > >> Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems". > > >> Fixes output for 3 of the 4 test lyx-files. > > >> Includes "FIXME"s at places where further action is required to get > >> the XeTeX > >> export right but I don't know how. > > > After this commit, when I run the command > > > lyx -e luatex lib/doc/he/Intro.lyx > > > it either fails to produce Intro.tex or (on current master) it produces > > a Intro.tex with gibberish where there should be Hebrew (at least this > > is what Vim shows me). > > Are you sure this has to do with this (partial) fix? No. I did a git bisect that led me to this commit, but there were various types of "failing" which makes me less confident that the problem was linear and thus less likely the bisect works. By multiple types of failings, I mean that either the Intro.tex file would not even be created or it would but would not contain Hebrew. So I would not be too surprised if the git bisect worked. I can do a manual check (compile just before), but first I want to check if something is indeed wrong. > In suspiciousTests, I see: > > # Hebrew docs do not currently work with LuaTeX (will be fixed in > TeXLive-16) # TODO: set working system fonts in the lyx sources > export/.*/he/.*(dvi3|pdf5).* > > > Is this expected? > > So, until further notice of a fix, I expect problems with Hebrew and > LuaTeX with both, tex- and unicode-fonts. > > > I know you put FIXMEs and one of them talks about ASCII encoding, so > > I'm guessing it is indeed expected, but I want to make sure. > > ASCII is required for XeTeX, LuaTeX can work with encoded source using > luainputenc. > > > If it is expected, do we have a bug report? > > No, as Luatex+Hebrew failing is an upstream bug. But I am just talking about exporting to .tex and opening in a text editor. LuaTeX does not even have a chance if our export to .tex corrupts the Hebrew. To be clear, attached are splash_good.tex and splash_bad.tex. Do you think it is OK that we now produce splash_bad.tex and before we produced splash_good.tex? If you tell me this is fine and that there is no encoding issue, then I trust your judgment since I know nothing about encoding. Scott splash_good.tex Description: TeX document splash_bad.tex Description: TeX document signature.asc Description: PGP signature
Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".
On 2016-02-13, Scott Kostyshak wrote: > On Tue, Oct 20, 2015 at 07:16:44PM +0200, Günter Milde wrote: >> commit 1523fc6023440f10ca0d82a681ded5c060d8fd33 >> Author: Günter Milde >> Date: Tue Oct 20 19:14:39 2015 +0200 >> Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems". >> Fixes output for 3 of the 4 test lyx-files. >> Includes "FIXME"s at places where further action is required to get the >> XeTeX >> export right but I don't know how. > After this commit, when I run the command > lyx -e luatex lib/doc/he/Intro.lyx > it either fails to produce Intro.tex or (on current master) it produces > a Intro.tex with gibberish where there should be Hebrew (at least this > is what Vim shows me). Are you sure this has to do with this (partial) fix? In suspiciousTests, I see: # Hebrew docs do not currently work with LuaTeX (will be fixed in TeXLive-16) # TODO: set working system fonts in the lyx sources export/.*/he/.*(dvi3|pdf5).* > Is this expected? So, until further notice of a fix, I expect problems with Hebrew and LuaTeX with both, tex- and unicode-fonts. > I know you put FIXMEs and one of them talks about ASCII encoding, so > I'm guessing it is indeed expected, but I want to make sure. ASCII is required for XeTeX, LuaTeX can work with encoded source using luainputenc. > If it is expected, do we have a bug report? No, as Luatex+Hebrew failing is an upstream bug. Günter
Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".
On Tue, Oct 20, 2015 at 07:16:44PM +0200, Günter Milde wrote: > commit 1523fc6023440f10ca0d82a681ded5c060d8fd33 > Author: Günter Milde > Date: Tue Oct 20 19:14:39 2015 +0200 > > Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems". > > Fixes output for 3 of the 4 test lyx-files. > > Includes "FIXME"s at places where further action is required to get the > XeTeX > export right but I don't know how. After this commit, when I run the command lyx -e luatex lib/doc/he/Intro.lyx it either fails to produce Intro.tex or (on current master) it produces a Intro.tex with gibberish where there should be Hebrew (at least this is what Vim shows me). Is this expected? I know you put FIXMEs and one of them talks about ASCII encoding, so I'm guessing it is indeed expected, but I want to make sure. If it is expected, do we have a bug report? Scott signature.asc Description: PGP signature
Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".
Am Mittwoch, 4. November 2015 um 10:33:58, schrieb Guenter Milde > On 2015-11-03, Scott Kostyshak wrote: > > On Tue, Nov 03, 2015 at 06:49:01AM -0800, Kornel Benko wrote: > >> > > >> > * to fix the regressions with "XeTeX + TeX fonts", a simple change in > >> > PDFOptions.cpp is sufficient. > > >> That is, 29 tests are now better than before. > >> Checking the output, all 29 are of type pdf4_texF (XeTeX + TeX fonts). > > Good news. > > Committed: f739c98fd7aa9cc6c7607/lyxgit Thanks. > ... > > >> > * the updated "comprehensive but incomplete" patch based on your work is > >> > now > >> > at http://www.lyx.org/trac/ticket/9839, so the work is not lost. > >> > > >> > With the "show output anyway" option in 2.2, the status quo is actually > >> > tolerable. I'll try the patch for #9830, then there is also a > >> > workaround for > >> > ASCII export (i.e. the user can write G\"unter instead of Günter). > > > Do we have a bug report for the workaround for ASCII export? > > I added a comment http://www.lyx.org/trac/ticket/9839#comment:1 > > ... > > > Günter, thanks for your work on these tricky issues. After alpha is > > released, let me know if there is anything pending that you would like > > for me to work on. Since I know nothing about this topic, you would need > > to break it down into simple steps like you did for me before so I can > > focus on the C++. +1 > Thanks for your offer. We will see what we can do one alpha is out... > > I'll also have a look at the other FIXMES and may ask for help there. > > > Günter Kornel signature.asc Description: This is a digitally signed message part.
Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".
On 2015-11-03, Scott Kostyshak wrote: > On Tue, Nov 03, 2015 at 06:49:01AM -0800, Kornel Benko wrote: >> > >> > * to fix the regressions with "XeTeX + TeX fonts", a simple change in >> > PDFOptions.cpp is sufficient. >> That is, 29 tests are now better than before. >> Checking the output, all 29 are of type pdf4_texF (XeTeX + TeX fonts). Good news. Committed: f739c98fd7aa9cc6c7607/lyxgit ... >> > * the updated "comprehensive but incomplete" patch based on your work is >> > now >> > at http://www.lyx.org/trac/ticket/9839, so the work is not lost. >> > >> > With the "show output anyway" option in 2.2, the status quo is actually >> > tolerable. I'll try the patch for #9830, then there is also a workaround >> > for >> > ASCII export (i.e. the user can write G\"unter instead of Günter). > Do we have a bug report for the workaround for ASCII export? I added a comment http://www.lyx.org/trac/ticket/9839#comment:1 ... > Günter, thanks for your work on these tricky issues. After alpha is > released, let me know if there is anything pending that you would like > for me to work on. Since I know nothing about this topic, you would need > to break it down into simple steps like you did for me before so I can > focus on the C++. Thanks for your offer. We will see what we can do one alpha is out... I'll also have a look at the other FIXMES and may ask for help there. Günter
Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".
On Tue, Nov 03, 2015 at 06:49:01AM -0800, Kornel Benko wrote: > Am Dienstag, 3. November 2015 um 12:40:22, schrieb Guenter Milde > > > On 2015-11-02, Scott Kostyshak wrote: > > > On Mon, Nov 02, 2015 at 10:40:27PM +, Guenter Milde wrote: > > > > Dear Scott, > > > > >> > thanks for the patch. > > > > >> I modified it to use UTF8 with XeTeX without the > > >> \inputencoding commands. > > > > ... > > > > > I would say go ahead and commit if you are satisfied with it. From what I > > > understand, the patch is now a complete patch in the sense that it is a > > > full > > > step forward. > > > > Actually, I am not satisfied, because the "normal" use case (exporting to a > > LaTeX file with inputencoding ASCII) is now worse than before. I this sense, > > it is a partial patch. OK makes sense. > > For the use case "XeTeX with TeX fonts", it is overly complicated. > > > > So I split it in two: > > > > * to fix the regressions with "XeTeX + TeX fonts", a simple change in > > PDFOptions.cpp is sufficient. See below. > > > > The nice thing is, that this is simple, localized and only changes > > where change is required - actually a completion for the patch-set > > fixing #9740. > > > > >> A short test here showed that this > > >> helps with Umlauts or Cyrillic charactes in the PDF Info. > > > > * the updated "comprehensive but incomplete" patch based on your work is now > > at http://www.lyx.org/trac/ticket/9839, so the work is not lost. > > > > With the "show output anyway" option in 2.2, the status quo is actually > > tolerable. I'll try the patch for #9830, then there is also a workaound > > for > > ASCII export (i.e. the user can write G\"unter instead of Günter). Do we have a bug report for the workaround for ASCII export? > > > Let me know if you want me to run the export tests before/after. > > > > Please check, if this patch cures some regressions introduced by > > fixing #9740. > > Check with 'ctest -L suspended' (my local version) > Now the following tests (previously not passing) pass: > 390 - INVERTED_SEE-README.ctest_export/doc/Math_pdf4_texF (Failed) > 702 - INVERTED_SEE-README.ctest_export/doc/de/Math_pdf4_texF (Failed) > 870 - INVERTED_SEE-README.ctest_export/doc/es/Math_pdf4_texF (Failed) > 1052 - INVERTED_SEE-README.ctest_export/doc/fr/Math_pdf4_texF (Failed) > 1787 - INVERTED_SEE-README.ctest_export/examples/PDF-comment_pdf4_texF > (Failed) > > They can now be removed from revertedTests. > > checking now with 'ctest -L export -E '(lyx16|xhtml)$' > -> 94% tests passed, 179 tests failed out of 3041 > > That is, 29 tests are now better than before. > Checking the output, all 29 are of type pdf4_texF (XeTeX + TeX fonts). > > I'd say, please commit. Thanks for checking Kornel. Günter, thanks for your work on these tricky issues. After alpha is released, let me know if there is anything pending that you would like for me to work on. Since I know nothing about this topic, you would need to break it down into simple steps like you did for me before so I can focus on the C++. Scott
Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".
Am Dienstag, 3. November 2015 um 12:40:22, schrieb Guenter Milde > On 2015-11-02, Scott Kostyshak wrote: > > On Mon, Nov 02, 2015 at 10:40:27PM +, Guenter Milde wrote: > > Dear Scott, > > >> > thanks for the patch. > > >> I modified it to use UTF8 with XeTeX without the > >> \inputencoding commands. > > ... > > > I would say go ahead and commit if you are satisfied with it. From what I > > understand, the patch is now a complete patch in the sense that it is a full > > step forward. > > Actually, I am not satisfied, because the "normal" use case (exporting to a > LaTeX file with inputencoding ASCII) is now worse than before. I this sense, > it is a partial patch. > > For the use case "XeTeX with TeX fonts", it is overly complicated. > > So I split it in two: > > * to fix the regressions with "XeTeX + TeX fonts", a simple change in > PDFOptions.cpp is sufficient. See below. > > The nice thing is, that this is simple, localized and only changes > where change is required - actually a completion for the patch-set > fixing #9740. > > >> A short test here showed that this > >> helps with Umlauts or Cyrillic charactes in the PDF Info. > > * the updated "comprehensive but incomplete" patch based on your work is now > at http://www.lyx.org/trac/ticket/9839, so the work is not lost. > > With the "show output anyway" option in 2.2, the status quo is actually > tolerable. I'll try the patch for #9830, then there is also a workaound for > ASCII export (i.e. the user can write G\"unter instead of Günter). > > > > Let me know if you want me to run the export tests before/after. > > Please check, if this patch cures some regressions introduced by > fixing #9740. Check with 'ctest -L suspended' (my local version) Now the following tests (previously not passing) pass: 390 - INVERTED_SEE-README.ctest_export/doc/Math_pdf4_texF (Failed) 702 - INVERTED_SEE-README.ctest_export/doc/de/Math_pdf4_texF (Failed) 870 - INVERTED_SEE-README.ctest_export/doc/es/Math_pdf4_texF (Failed) 1052 - INVERTED_SEE-README.ctest_export/doc/fr/Math_pdf4_texF (Failed) 1787 - INVERTED_SEE-README.ctest_export/examples/PDF-comment_pdf4_texF (Failed) They can now be removed from revertedTests. checking now with 'ctest -L export -E '(lyx16|xhtml)$' -> 94% tests passed, 179 tests failed out of 3041 That is, 29 tests are now better than before. Checking the output, all 29 are of type pdf4_texF (XeTeX + TeX fonts). I'd say, please commit. > Günter Kornel signature.asc Description: This is a digitally signed message part.
Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".
On 2015-11-02, Scott Kostyshak wrote: > On Mon, Nov 02, 2015 at 10:40:27PM +, Guenter Milde wrote: Dear Scott, >> > thanks for the patch. >> I modified it to use UTF8 with XeTeX without the >> \inputencoding commands. ... > I would say go ahead and commit if you are satisfied with it. From what I > understand, the patch is now a complete patch in the sense that it is a full > step forward. Actually, I am not satisfied, because the "normal" use case (exporting to a LaTeX file with inputencoding ASCII) is now worse than before. I this sense, it is a partial patch. For the use case "XeTeX with TeX fonts", it is overly complicated. So I split it in two: * to fix the regressions with "XeTeX + TeX fonts", a simple change in PDFOptions.cpp is sufficient. See below. The nice thing is, that this is simple, localized and only changes where change is required - actually a completion for the patch-set fixing #9740. >> A short test here showed that this >> helps with Umlauts or Cyrillic charactes in the PDF Info. * the updated "comprehensive but incomplete" patch based on your work is now at http://www.lyx.org/trac/ticket/9839, so the work is not lost. With the "show output anyway" option in 2.2, the status quo is actually tolerable. I'll try the patch for #9830, then there is also a workaound for ASCII export (i.e. the user can write G\"unter instead of Günter). > Let me know if you want me to run the export tests before/after. Please check, if this patch cures some regressions introduced by fixing #9740. Günter Exec: git 'diff' 'PDFOptions.cpp' 2>&1 Dir: /usr/local/src/lyx/src/ diff --git a/src/PDFOptions.cpp b/src/PDFOptions.cpp index 6646888..143d715 100644 --- a/src/PDFOptions.cpp +++ b/src/PDFOptions.cpp @@ -174,21 +174,15 @@ void PDFOptions::writeLaTeX(OutputParams & runparams, otexstream & os, opt = "\\hypersetup{" + rtrim(opt + hyperset, ",") + "}\n"; } - // hyperref expects LICR macros for non-ASCII chars. With Xe/LuaTeX utf-8 works, too. - // Usually, "(lua)inputenc" converts the input to LICR. + // hyperref expects LICR macros for non-ASCII chars. + // Usually, "(lua)inputenc" converts the input to LICR, with XeTeX utf-8 works, too. // As hyperref provides good coverage for \inputencoding{utf8}, we can try // this if the current input encoding does not support a character. - // FIXME: inputenc (part 1 of 2) - // Replace the "FullUnicode" check with - // check for loading of inputenc or luainputenc package - // (see BufferParams::writeEncodingPreamble and runparams.encoding->package()). - // Otherwise \inputencoding is not defined - // (e.g. if "latex-encoding" is set to "ascii"). - // Dont forget to keep the check below (part 2) in sync! - if (need_unicode && enc && enc->iconvName() != "UTF-8" - &&!runparams.isFullUnicode()) { - os << "\\inputencoding{utf8}\n" - << setEncoding("UTF-8"); + // FIXME: don't use \inputencoding if "inputenc" is not loaded (#9839). + if (need_unicode && enc && enc->iconvName() != "UTF-8") { + if (runparams.flavor != OutputParams::XETEX) + os << "\\inputencoding{utf8}\n"; + os << setEncoding("UTF-8"); } // If hyperref is loaded by the document class, we output // \hypersetup \AtBeginDocument if hypersetup is not (yet) @@ -204,11 +198,10 @@ void PDFOptions::writeLaTeX(OutputParams & runparams, otexstream & os, << "\\fi\n"; } else os << from_utf8(opt); - // FIXME: inputenc (part 2 of 2) - if (need_unicode && enc && enc->iconvName() != "UTF-8" - &&!runparams.isFullUnicode()) { - os << setEncoding(enc->iconvName()) - << "\\inputencoding{" << from_ascii(enc->latexName()) << "}\n"; + if (need_unicode && enc && enc->iconvName() != "UTF-8") { + os << setEncoding(enc->iconvName()); + if (runparams.flavor != OutputParams::XETEX) + os << "\\inputencoding{" << from_ascii(enc->latexName()) << "}\n"; } }
Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".
On Mon, Nov 02, 2015 at 10:40:27PM +, Guenter Milde wrote: > On 2015-11-01, Guenter Milde wrote: > > Dear Scott, > > > thanks for the patch. > > I modified it to use UTF8 with XeTeX without the > \inputencoding commands. A short test here showed that this > helps with Umlauts or Cyrillic charactes in the PDF Info. > > Give it a try. Nice. I am not able to test it because I don't understand the underlying issue, but I did look at the code and it looks good to me. I have a couple picky comments about using tabs instead of spaces below (by the way, there are ways of configuring Git to catch these types of formatting inconsistencies, as well as your editor. Ask on the list if curious). > + if (inputenc_loaded) replace spaces with tab? > + if (inputenc_loaded) replace spaces with tabs? I would say go ahead and commit if you are satisfied with it. From what I understand, the patch is now a complete patch in the sense that it is a full step forward. I know there are other things you want to do, but I think this patch is a self-contained step forward and not a partial patch. If that's correct, then I suggest you commit. Let me know if you want me to run the export tests before/after. Scott
Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".
On 2015-11-01, Guenter Milde wrote: > Dear Scott, > thanks for the patch. I modified it to use UTF8 with XeTeX without the \inputencoding commands. A short test here showed that this helps with Umlauts or Cyrillic charactes in the PDF Info. Give it a try. Günter Exec: git 'diff' 2>&1 Dir: /usr/local/src/lyx diff --git a/src/BufferParams.cpp b/src/BufferParams.cpp index 537178e..de39309 100644 --- a/src/BufferParams.cpp +++ b/src/BufferParams.cpp @@ -1577,7 +1577,7 @@ bool BufferParams::writeLaTeX(otexstream & os, LaTeXFeatures & features, } // handle inputenc etc. - writeEncodingPreamble(os, features); + bool const inputenc_loaded = writeEncodingPreamble(os, features); // includeonly if (!features.runparams().includeall && !included_children_.empty()) { @@ -1903,7 +1903,7 @@ bool BufferParams::writeLaTeX(otexstream & os, LaTeXFeatures & features, OutputParams tmp_params = features.runparams(); pdfoptions().writeLaTeX(tmp_params, os, - features.isProvided("hyperref")); + features.isProvided("hyperref"), inputenc_loaded); // set back for the rest lyxpreamble.clear(); // correctly break URLs with hyperref and dvi output @@ -2924,19 +2924,20 @@ docstring BufferParams::getGraphicsDriver(string const & package) const } -void BufferParams::writeEncodingPreamble(otexstream & os, +bool BufferParams::writeEncodingPreamble(otexstream & os, LaTeXFeatures & features) const { // "inputenc" package not required with non-TeX fonts. if (useNonTeXFonts) - return; + return false; // "inputenc" fails with XeTeX (even in 8-bit compatiblitly mode) and with TeX fonts, // (this is a bug in the "inputenc" package see #9740). if (features.runparams().flavor == OutputParams::XETEX) - return; + return false; // For LuaTeX with TeX fonts, we can load // the "luainputenc" package with the specified encoding(s) (see below). + bool use_inputenc = features.isProvided("inputenc"); if (inputenc == "auto") { string const doc_encoding = language->encoding()->latexName(); @@ -2973,6 +2974,7 @@ void BufferParams::writeEncodingPreamble(otexstream & os, os << "]{luainputenc}\n"; else os << "]{inputenc}\n"; + use_inputenc = true; } if (package == Encoding::CJK || features.mustProvide("CJK")) { if (language->encoding()->name() == "utf8-cjk" @@ -2998,6 +3000,7 @@ void BufferParams::writeEncodingPreamble(otexstream & os, os << "]{luainputenc}\n"; else os << "]{inputenc}\n"; + use_inputenc = true; break; case Encoding::CJK: if (encoding().name() == "utf8-cjk" @@ -3017,6 +3020,7 @@ void BufferParams::writeEncodingPreamble(otexstream & os, os << "\\usepackage{CJK}\n"; } } + return use_inputenc; } diff --git a/src/BufferParams.h b/src/BufferParams.h index b5ab65e..e3ff4f1 100644 --- a/src/BufferParams.h +++ b/src/BufferParams.h @@ -412,7 +412,8 @@ public: /// return supported drivers for specific packages docstring getGraphicsDriver(std::string const & package) const; /// handle inputenc etc. - void writeEncodingPreamble(otexstream & os, LaTeXFeatures & features) const; +/// return true if (lua)inputenc is written here or already provided + bool writeEncodingPreamble(otexstream & os, LaTeXFeatures & features) const; /// std::string const parseFontName(std::string const & name) const; /// set up the document fonts diff --git a/src/PDFOptions.cpp b/src/PDFOptions.cpp index 6646888..b4e91df 100644 --- a/src/PDFOptions.cpp +++ b/src/PDFOptions.cpp @@ -90,7 +90,7 @@ void PDFOptions::writeFile(ostream & os) const void PDFOptions::writeLaTeX(OutputParams & runparams, otexstream & os, - bool hyperref_already_provided) const + bool hyperref_already_provided, bool inputenc_loaded) const { // FIXME Unicode string opt; @@ -176,20 +176,21 @@ void PDFOptions::writeLaTeX(OutputParams & runparams, otexstream & os, // hyperref expects LICR macros for non-ASCII chars. With Xe/LuaTeX utf-8 works, too. // Usually, "(lua)inputenc" converts the input to LICR. - // As hyperref provides good coverage for \inputencoding{utf8}, we can try - // this if the current input encoding do
Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".
Dear Scott, thanks for the patch. On 2015-10-31, Scott Kostyshak wrote: > On Sat, Oct 31, 2015 at 05:57:46PM -0400, Scott Kostyshak wrote: >> On Sat, Oct 31, 2015 at 09:21:15PM +, Guenter Milde wrote: > Does the attached patch look good? It looks like a step in the right direction. However, this is just a prerequisite for solving the encoding problems, not the fix yet. Therefore, it is to be expected that this patch does not cure any of the export tests (they just fail for another reason). > We need to do something similar for the other FIXMEs you put in > 1523fc60, right? This depends on what you mean with "similar". AFAIK, the problem there is not about PDFstrings and hence the solution also different. > Scott > [-- Type: text/plain, Encoding: , Filename: > 0001-Improve-XeTeX-LuaTeX-with-TeX-fonts-9740.patch --] > From ef7d21fc35679683f537c186ce9200cba232a8b3 Mon Sep 17 00:00:00 2001 > From: Scott Kostyshak > Date: Sat, 31 Oct 2015 18:59:23 -0400 > Subject: [PATCH] Improve XeTeX/LuaTeX with TeX fonts, #9740 This is not about XeTex/LuaTeX with TeX fonts: The problem fixed is "ensure \inputencoding is defined before using it in the document preamble". Conditions where this applies are "inputencoding ascii", "inputencoding (default without inputenc)" and "cjk without utf8" and a non-encodable character in the PDF info (most of them are currently not tested). * As XeTeX with TeX fonts requires "inputencoding ascii", it is affected indirectly. * LuaTeX works fine (as it loads luainputenc which defines \inputencoding). > More fixes are needed to solve the FIXMEs in 1523fc60. More fixes are needed to solve the errors with non-encodable characters in the PDF info fields. > --- a/src/BufferParams.cpp > +++ b/src/BufferParams.cpp > @@ -1577,7 +1577,8 @@ bool BufferParams::writeLaTeX(otexstream & os, > LaTeXFeatures & features, > } > // handle inputenc etc. > - writeEncodingPreamble(os, features); > + bool const ie = writeEncodingPreamble(os, features); > + bool const inputenc_loaded = ie || features.isProvided("inputenc"); As features.isProvided("inputenc") is also tested in writeEncodingPreamble, we could move this test there and write here simply: bool const inputenc_loaded = writeEncodingPreamble(os, features); > -void BufferParams::writeEncodingPreamble(otexstream & os, > +bool BufferParams::writeEncodingPreamble(otexstream & os, >LaTeXFeatures & features) const > { > // "inputenc" package not required with non-TeX fonts. > if (useNonTeXFonts) > - return; > + return false; > // "inputenc" fails with XeTeX (even in 8-bit compatiblitly mode) and > with TeX fonts, > // (this is a bug in the "inputenc" package see #9740). > if (features.runparams().flavor == OutputParams::XETEX) > - return; > + return false; > // For LuaTeX with TeX fonts, we can load > // the "luainputenc" package with the specified encoding(s) (see below). > + bool ret = false; Why not bool inputenc_provided = features.isProvided("inputenc"); > if (inputenc == "auto") { > string const doc_encoding = > language->encoding()->latexName(); > @@ -2955,6 +2957,7 @@ void BufferParams::writeEncodingPreamble(otexstream & > os, > && !features.isRequired("japanese") > && !features.isProvided("inputenc")) { > os << "\\usepackage["; > + ret = true; > set::const_iterator it = encodings.begin(); > set::const_iterator const end = encodings.end(); > if (it != end) { Place the "ret = ..." line rather below the complete loading command, not inbetween. > @@ -2993,6 +2996,7 @@ void BufferParams::writeEncodingPreamble(otexstream & > os, > || features.isProvided("inputenc")) > break; > os << "\\usepackage[" << > from_ascii(encoding().latexName()); > + ret = true; > if (features.runparams().flavor == OutputParams::LUATEX > || features.runparams().flavor == > OutputParams::DVILUATEX) > os << "]{luainputenc}\n"; > @@ -3017,6 +3021,7 @@ void BufferParams::writeEncodingPreamble(otexstream & > os, > os << "\\usepackage{CJK}\n"; > } > } > + return ret; > } ... > void PDFOptions::writeLaTeX(OutputParams & runparams, otexstream & os, > - bool hyperref_already_provided) const > + bool hyperref_already_provided, bool > inputenc_loaded) const > { > // FIXME Unicode > string opt; > @@ -178,15 +178,12 @@ void PDFOptions::writeLaTeX(OutputParams & runparams, > otexstream & os, > // U
Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".
On Sun, Nov 01, 2015 at 01:13:39PM +0100, Kornel Benko wrote: > Am Sonntag, 1. November 2015 um 11:55:47, schrieb Kornel Benko > > > Am Sonntag, 1. November 2015 um 11:52:26, schrieb Kornel Benko > > > > > Am Samstag, 31. Oktober 2015 um 19:11:10, schrieb Scott Kostyshak > > > > > > > On Sat, Oct 31, 2015 at 05:57:46PM -0400, Scott Kostyshak wrote: > > > > > On Sat, Oct 31, 2015 at 09:21:15PM +, Guenter Milde wrote: > > > > > > > > > > I'd prefer you to do this. > > > > > > > > > > OK I'll work on it. > > > > > > > > Does the attached patch look good? > > > > > > > > We need to do something similar for the other FIXMEs you put in > > > > 1523fc60, right? > > > > > > > > Scott > > > > > > Applying this patch now _all_ export tests 'export/.*(pdf|dvi)' are > > > failing. > > > Probably because the patch is only partial I suppose. > > > > > > > I was too fast. The first 208 (out of 3761) did not pass as I wrote the > > mail. > > Now I see some other passing. > > I will report the outcome, have to wait a 'little' more. > > Thanks for running the tests, Kornel. > It turned out, that all but the first 208 testcases pass. > Comparing the failed with the list of previously failed, there is no > difference. I think that ctest always runs the failed tests first. This has surprised me before. > So even if your patch does not improve any test, it does not harm either. OK let's wait and see what Günter says then. I'm not even confident my patch is correct. If he confirms that it is, then I can go on and make the full patch. I'm not sure if the full patch is supposed to fix any tests (although it would be nice if it whittles down some of those 208). Scott
Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".
Am Sonntag, 1. November 2015 um 11:55:47, schrieb Kornel Benko > Am Sonntag, 1. November 2015 um 11:52:26, schrieb Kornel Benko > > > Am Samstag, 31. Oktober 2015 um 19:11:10, schrieb Scott Kostyshak > > > > > On Sat, Oct 31, 2015 at 05:57:46PM -0400, Scott Kostyshak wrote: > > > > On Sat, Oct 31, 2015 at 09:21:15PM +, Guenter Milde wrote: > > > > > > > > I'd prefer you to do this. > > > > > > > > OK I'll work on it. > > > > > > Does the attached patch look good? > > > > > > We need to do something similar for the other FIXMEs you put in > > > 1523fc60, right? > > > > > > Scott > > > > Applying this patch now _all_ export tests 'export/.*(pdf|dvi)' are failing. > > Probably because the patch is only partial I suppose. > > > > I was too fast. The first 208 (out of 3761) did not pass as I wrote the mail. > Now I see some other passing. > I will report the outcome, have to wait a 'little' more. > It turned out, that all but the first 208 testcases pass. Comparing the failed with the list of previously failed, there is no difference. So even if your patch does not improve any test, it does not harm either. Kornel signature.asc Description: This is a digitally signed message part.
Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".
Am Sonntag, 1. November 2015 um 11:52:26, schrieb Kornel Benko > Am Samstag, 31. Oktober 2015 um 19:11:10, schrieb Scott Kostyshak > > > On Sat, Oct 31, 2015 at 05:57:46PM -0400, Scott Kostyshak wrote: > > > On Sat, Oct 31, 2015 at 09:21:15PM +, Guenter Milde wrote: > > > > > > I'd prefer you to do this. > > > > > > OK I'll work on it. > > > > Does the attached patch look good? > > > > We need to do something similar for the other FIXMEs you put in > > 1523fc60, right? > > > > Scott > > Applying this patch now _all_ export tests 'export/.*(pdf|dvi)' are failing. > Probably because the patch is only partial I suppose. > I was too fast. The first 208 (out of 3761) did not pass as I wrote the mail. Now I see some other passing. I will report the outcome, have to wait a 'little' more. Kornel signature.asc Description: This is a digitally signed message part.
Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".
On Sat, Oct 31, 2015 at 05:57:46PM -0400, Scott Kostyshak wrote: > On Sat, Oct 31, 2015 at 09:21:15PM +, Guenter Milde wrote: > > I'd prefer you to do this. > > OK I'll work on it. Does the attached patch look good? We need to do something similar for the other FIXMEs you put in 1523fc60, right? Scott >From ef7d21fc35679683f537c186ce9200cba232a8b3 Mon Sep 17 00:00:00 2001 From: Scott Kostyshak Date: Sat, 31 Oct 2015 18:59:23 -0400 Subject: [PATCH] Improve XeTeX/LuaTeX with TeX fonts, #9740 More fixes are needed to solve the FIXMEs in 1523fc60. --- src/BufferParams.cpp | 15 ++- src/BufferParams.h | 3 ++- src/PDFOptions.cpp | 18 +++--- src/PDFOptions.h | 2 +- 4 files changed, 20 insertions(+), 18 deletions(-) diff --git a/src/BufferParams.cpp b/src/BufferParams.cpp index 537178e..0fd747b 100644 --- a/src/BufferParams.cpp +++ b/src/BufferParams.cpp @@ -1577,7 +1577,8 @@ bool BufferParams::writeLaTeX(otexstream & os, LaTeXFeatures & features, } // handle inputenc etc. - writeEncodingPreamble(os, features); + bool const ie = writeEncodingPreamble(os, features); + bool const inputenc_loaded = ie || features.isProvided("inputenc"); // includeonly if (!features.runparams().includeall && !included_children_.empty()) { @@ -1903,7 +1904,7 @@ bool BufferParams::writeLaTeX(otexstream & os, LaTeXFeatures & features, OutputParams tmp_params = features.runparams(); pdfoptions().writeLaTeX(tmp_params, os, - features.isProvided("hyperref")); + features.isProvided("hyperref"), inputenc_loaded); // set back for the rest lyxpreamble.clear(); // correctly break URLs with hyperref and dvi output @@ -2924,19 +2925,20 @@ docstring BufferParams::getGraphicsDriver(string const & package) const } -void BufferParams::writeEncodingPreamble(otexstream & os, +bool BufferParams::writeEncodingPreamble(otexstream & os, LaTeXFeatures & features) const { // "inputenc" package not required with non-TeX fonts. if (useNonTeXFonts) - return; + return false; // "inputenc" fails with XeTeX (even in 8-bit compatiblitly mode) and with TeX fonts, // (this is a bug in the "inputenc" package see #9740). if (features.runparams().flavor == OutputParams::XETEX) - return; + return false; // For LuaTeX with TeX fonts, we can load // the "luainputenc" package with the specified encoding(s) (see below). + bool ret = false; if (inputenc == "auto") { string const doc_encoding = language->encoding()->latexName(); @@ -2955,6 +2957,7 @@ void BufferParams::writeEncodingPreamble(otexstream & os, && !features.isRequired("japanese") && !features.isProvided("inputenc")) { os << "\\usepackage["; + ret = true; set::const_iterator it = encodings.begin(); set::const_iterator const end = encodings.end(); if (it != end) { @@ -2993,6 +2996,7 @@ void BufferParams::writeEncodingPreamble(otexstream & os, || features.isProvided("inputenc")) break; os << "\\usepackage[" << from_ascii(encoding().latexName()); + ret = true; if (features.runparams().flavor == OutputParams::LUATEX || features.runparams().flavor == OutputParams::DVILUATEX) os << "]{luainputenc}\n"; @@ -3017,6 +3021,7 @@ void BufferParams::writeEncodingPreamble(otexstream & os, os << "\\usepackage{CJK}\n"; } } + return ret; } diff --git a/src/BufferParams.h b/src/BufferParams.h index b5ab65e..328ca17 100644 --- a/src/BufferParams.h +++ b/src/BufferParams.h @@ -412,7 +412,8 @@ public: /// return supported drivers for specific packages docstring getGraphicsDriver(std::string const & package) const; /// handle inputenc etc. - void writeEncodingPreamble(otexstream & os, LaTeXFeatures & features) const; +/// return true if (lua)inputenc is written here + bool writeEncodingPreamble(otexstream & os, LaTeXFeatures & features) const; /// std::string const parseFontName(std::string const & name) const; /// set up the document fonts diff --git a/src/PDFOptions.cpp b/src/PDFOptions.cpp index 6646888..2ecc5c6 100644 --- a/src/PDFOptions.cpp +++ b/src/PDFOptions.cpp @@ -90,7 +90,7 @@ void PDFOptions::writeFile(ostream & os) const void PDFOptions::writeLaTeX(OutputParams & runpa
Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".
On Sat, Oct 31, 2015 at 09:21:15PM +, Guenter Milde wrote: > On 2015-10-30, Scott Kostyshak wrote: > > On Fri, Oct 30, 2015 at 10:45:38PM +, Guenter Milde wrote: > >> On 2015-10-30, Scott Kostyshak wrote: > >> > On Thu, Oct 29, 2015 at 09:12:04AM +, Guenter Milde wrote: > >> >> On 2015-10-29, Scott Kostyshak wrote: > > > OK so it seems we're on the same page. The next question is what > > information exactly do you need from writeEncodingPreamble? Do you just > > need a bool 'use_inputenc' that is true if either > > \usepackage[...]{inputenc} or \usepackage[...]{luainputenc} is output by > > writeEncodingPreamble? > > The important thing is to know, whether it is safe to use the > \inputencoding macro. This could be a boolean value that is true if: > > * \usepackage[...]{inputenc} or \usepackage[...]{luainputenc} is output by > writeEncodingPreamble > > * a package/class that loads inputenc (or luainputenc) is loaded > (this is tested in writeEncodingPreamble via > features.isProvided("inputenc")). Thanks. I have a clear idea of what is needed now. > > Do you want to do the C++ work or would you prefer for me to? I think it > > should be simple but I'm happy to do the work since I've finally wrapped > > my head around the issue. > > I'd prefer you to do this. OK I'll work on it. > Deciding, what to do in case inputenc is not availabel remains an open > issue: with XeTeX, we can just use UTF8, otherwise we have to decide > between conversion with lib/unicodesymbols (or something better suited > for "pdfstrings") or giving a warning or error. I see. This is in essence the (currently non-existing) "else" after the "if" conditions that we've been focusing on in PDFOptions.cpp. Right? Scott
Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".
On 2015-10-30, Scott Kostyshak wrote: > On Fri, Oct 30, 2015 at 10:45:38PM +, Guenter Milde wrote: >> On 2015-10-30, Scott Kostyshak wrote: >> > On Thu, Oct 29, 2015 at 09:12:04AM +, Guenter Milde wrote: >> >> On 2015-10-29, Scott Kostyshak wrote: > OK so it seems we're on the same page. The next question is what > information exactly do you need from writeEncodingPreamble? Do you just > need a bool 'use_inputenc' that is true if either > \usepackage[...]{inputenc} or \usepackage[...]{luainputenc} is output by > writeEncodingPreamble? The important thing is to know, whether it is safe to use the \inputencoding macro. This could be a boolean value that is true if: * \usepackage[...]{inputenc} or \usepackage[...]{luainputenc} is output by writeEncodingPreamble * a package/class that loads inputenc (or luainputenc) is loaded (this is tested in writeEncodingPreamble via features.isProvided("inputenc")). > Do you want to do the C++ work or would you prefer for me to? I think it > should be simple but I'm happy to do the work since I've finally wrapped > my head around the issue. I'd prefer you to do this. Deciding, what to do in case inputenc is not availabel remains an open issue: with XeTeX, we can just use UTF8, otherwise we have to decide between conversion with lib/unicodesymbols (or something better suited for "pdfstrings") or giving a warning or error. Thanks, Günter
Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".
On Fri, Oct 30, 2015 at 10:45:38PM +, Guenter Milde wrote: > On 2015-10-30, Scott Kostyshak wrote: > > On Thu, Oct 29, 2015 at 09:12:04AM +, Guenter Milde wrote: > >> On 2015-10-29, Scott Kostyshak wrote: > > ... > > > words, the only place that knows whether the inputenc package is > > loaded is BufferParams::writeEncodingPreamble() > > This is my understanding, too. > > > Actually, after thinking about this a bit longer my new favorite > > approach is simply for BufferParams::writeEncodingPreamble to return the > > necessary information (e.g. it could return a boolean using_inputenc) > > that is then just passed to pdfoptions().writeLaTeX(). That seems the > > cleanest and safest approach. I thought that it might not be common for > > a write method to return such information (I figured if anything it > > would return success/fail), but actually we already do this: > > BufferParams::writeLaTeX() returns a boolean use_babel. > > Good to know, that there is a precedence. OK so it seems we're on the same page. The next question is what information exactly do you need from writeEncodingPreamble? Do you just need a bool 'use_inputenc' that is true if either \usepackage[...]{inputenc} or \usepackage[...]{luainputenc} is output by writeEncodingPreamble? Do you want to do the C++ work or would you prefer for me to? I think it should be simple but I'm happy to do the work since I've finally wrapped my head around the issue. Scott
Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".
On 2015-10-30, Scott Kostyshak wrote: > On Thu, Oct 29, 2015 at 09:12:04AM +, Guenter Milde wrote: >> On 2015-10-29, Scott Kostyshak wrote: ... > words, the only place that knows whether the inputenc package is > loaded is BufferParams::writeEncodingPreamble() This is my understanding, too. > Actually, after thinking about this a bit longer my new favorite > approach is simply for BufferParams::writeEncodingPreamble to return the > necessary information (e.g. it could return a boolean using_inputenc) > that is then just passed to pdfoptions().writeLaTeX(). That seems the > cleanest and safest approach. I thought that it might not be common for > a write method to return such information (I figured if anything it > would return success/fail), but actually we already do this: > BufferParams::writeLaTeX() returns a boolean use_babel. Good to know, that there is a precedence. Thanks, Günter
Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".
On Thu, Oct 29, 2015 at 09:12:04AM +, Guenter Milde wrote: > On 2015-10-29, Scott Kostyshak wrote: > As my FIXME was wrong, so is your patch. I see. > > In regards to "Dont forget to keep this check in sync with the check > > above!" this seems to be a good situation for a helper function in order > > to share code. > > Maybe define a new setting/helper_function "inputenc package available" > in BufferParams.cpp? I'm starting to understand more the situation and what you want. Here is what I think I understand: - the BufferParams inputenc member is not what you want because that is just the setting in the document. It is a complicated process (as shown by BufferParams::writeEncodingPreamble()) to figure out what is actually done. - you cannot use features.isRequired("inputenc") because even if that is false, it seems that the inputenc package could still be loaded, as is done in BufferParams::writeEncodingPreamble(). In other words, the only place that knows whether the inputenc package is loaded is BufferParams::writeEncodingPreamble() and it does not store the information anywhere, it just writes to the preamble. (EDIT for reader: skip to the paragraph starting with "Actually" unless you want to see the clumsy thought process leading up to that) So the solution might be to factor out the code that determines whether to load the inputenc package from BufferParams::writeEncodingPreamble() in a place that would be acccessible both by BufferParams::writeEncodingPreamble() and PDFOptions::writeLaTeX(). writeEncodingPreamble() and pdfoptions().writeLaTeX() are both called in BufferParams::writeLaTeX(), so one possibility is that instead of writeEncodingPreamble() doing the necessary steps to determine whether to load the inputenc package, that could be done in BufferParams::writeLaTeX(), either through a helper function (maybe named something like determineEncodingPreamble() ?) or just directly (but I think a helper function would be preferred). For example, we could create a boolean using_inputenc that is local to BufferParams::writeLaTeX() and then pass that boolean to writeEncodingPreamble() and pdfoptions().writeLaTeX(). Another possibility is to store the information in the BufferParams class. If we do this, I imagine we should not write new information from bufferParams::writeLaTeX(). At least, I would not expect it to write any new members judging from its name. So we would have to factor out the code further. Actually, after thinking about this a bit longer my new favorite approach is simply for BufferParams::writeEncodingPreamble to return the necessary information (e.g. it could return a boolean using_inputenc) that is then just passed to pdfoptions().writeLaTeX(). That seems the cleanest and safest approach. I thought that it might not be common for a write method to return such information (I figured if anything it would return success/fail), but actually we already do this: BufferParams::writeLaTeX() returns a boolean use_babel. Scott
Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".
On 2015-10-29, Scott Kostyshak wrote: > On Tue, Oct 20, 2015 at 07:16:44PM +0200, Günter Milde wrote: >> commit 1523fc6023440f10ca0d82a681ded5c060d8fd33 >> Author: Günter Milde >> Date: Tue Oct 20 19:14:39 2015 +0200 >> Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems". >> Fixes output for 3 of the 4 test lyx-files. >> Includes "FIXME"s at places where further action is required to get the >> XeTeX >> export right but I don't know how. > I'm not sure if you know or expect this (e.g. because of the FIXMEs), but this > commit broke some of the tests for PDF-comment.lyx. It is not actually > expected > that this file should be compiled successfully with XeTeX or LuaTeX but it > used > to work (perhaps by luck?). At least, I am not surprised. I know there are cases that fail now but worked before, especially XeTeX with TeX-fonts and non-ASCII characters in the PDFinfo: * before, the inputencoding was utf8, which is + fine for the PDFinfo but - wrong with non-ASCII characters in the text (either compilation error or wrong characters in the output). * with the partial patch (fix 2), the inputencoding is set to "ascii", which is + correct for the text - fails with non-ASCII characters in the PDFinfo > Do you think it is worth taking a look at why the tests started failing > or should we forget about it in this case because the file is supposed > to be compiled with pdfTeX? My outlined FIXME would correct this (using ascii for the text and utf8 for the PDFinfo. Until some kind soul with proper understanding is going to complete it, I propose to * open a bug report for non-ASCII characters in the PDFinfo with ASCII encoding (also fails with 8-bit TeX) (I could to this.) * invert the tests with a pointer to this ticket. > It seems that XeTeX with system fonts was broken by this commit but > then it was fixed later on. The same thing happened for a couple of the > other tests. So I guess we don't have to look at those. The following, > however, were broken by this commit and the results from master > indicate that they're still broken: > export/examples/PDF-comment_pdf5_texF (Failed) > export/examples/de/PDF-comment_pdf4_texF (Failed) > export/examples/de/PDF-comment_pdf5_texF (Failed) > export/examples/fr/PDF-comment_pdf5_texF (Failed) > Recall that pdf5 refers to compiling with LuaTeX and pdf4 refers to > compiling with XeTeX. All of the four above tests are regarding TeX > fonts. With XeTeX, the problem is the non-ASCII ö (o-umlaut) in the Author field. With LuaTeX, the error is unrelated (at least here): LaTeX info: "xparse/define-command" . . Defining command \selectchemgreekmapping with sig. 'm' on line 485. . ! Argument of \__str_case_end:nw has an extra }. \par l.573 \chemgreek_drop_symbols: > I have checked in the commit before this one that the PDFs of > the above four compilations look OK to me at first look (i.e. not > garbled from what I see). Before which commit? Günter
Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".
On 2015-10-29, Scott Kostyshak wrote: > On Wed, Oct 28, 2015 at 10:28:00PM +, Guenter Milde wrote: >> On 2015-10-28, Scott Kostyshak wrote: >> > On Wed, Oct 28, 2015 at 09:31:50AM +, Guenter Milde wrote: >> >> On 2015-10-26, Guenter Milde wrote: >> >> > On 2015-10-26, Scott Kostyshak wrote: >> >> >> Has anything changed in the last couple of days? >> >> I'l commit an updated FIXME. >> > This does indeed sound tricky. Can you figure out the necessary items to >> > condition on from BufferParams or is not everything stored in there? >> It became a bit simpler after going through the alternatives: >> In pseudo-code: > I looked at the pseudo-code but since I don't know this code well it's > not clear if you still need my help. If you have access to the buffer > params (which my patch gave), is that enough? Can you test with the > patch I posted before? Or is something else missing? As my FIXME was wrong, so is your patch. In principle, I may be able to use your patch and combine it with my comments, but alas, I don't know this code either, nor do I know C++, so it would take too much time and the result would not be clean and I'd rather leave this to a more eligible person. > In regards to "Dont forget to keep this check in sync with the check > above!" this seems to be a good situation for a helper function in order > to share code. Maybe define a new setting/helper_function "inputenc package available" in BufferParams.cpp? Thanks, Günter
Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".
On Tue, Oct 20, 2015 at 07:16:44PM +0200, Günter Milde wrote: > commit 1523fc6023440f10ca0d82a681ded5c060d8fd33 > Author: Günter Milde > Date: Tue Oct 20 19:14:39 2015 +0200 > > Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems". > > Fixes output for 3 of the 4 test lyx-files. > > Includes "FIXME"s at places where further action is required to get the > XeTeX > export right but I don't know how. I'm not sure if you know or expect this (e.g. because of the FIXMEs), but this commit broke some of the tests for PDF-comment.lyx. It is not actually expected that this file should be compiled successfully with XeTeX or LuaTeX but it used to work (perhaps by luck?). Do you think it is worth taking a look at why the tests started failing or should we forget about it in this case because the file is supposed to be compiled with pdfTeX? In case you are interested: The command ctest -R "comment" gives me the following failed tests just *before* this commit (which is commit 9d7c3a48): INVERTED_SEE-README.ctest_export/examples/PDF-comment_dvi (Failed) INVERTED_SEE-README.ctest_export/examples/PDF-comment_pdf (Failed) INVERTED_SEE-README.ctest_export/examples/PDF-comment_pdf3 (Failed) INVERTED_SEE-README.ctest_export/examples/PDF-comment_pdf4_systemF (Failed) export/examples/de/PDF-comment_dvi3_texF (Failed) export/examples/de/PDF-comment_dvi3_systemF (Failed) export/examples/fr/PDF-comment_dvi3_texF (Failed) export/examples/fr/PDF-comment_dvi3_systemF (Failed) export/examples/fr/PDF-comment_pdf4_texF (Failed) With this commit (1523fc60), the same command gives the following: INVERTED_SEE-README.ctest_export/examples/PDF-comment_dvi (Failed) INVERTED_SEE-README.ctest_export/examples/PDF-comment_pdf (Failed) INVERTED_SEE-README.ctest_export/examples/PDF-comment_pdf3 (Failed) export/examples/PDF-comment_pdf5_texF (Failed) export/examples/de/PDF-comment_dvi3_texF (Failed) export/examples/de/PDF-comment_dvi3_systemF (Failed) export/examples/de/PDF-comment_pdf4_texF (Failed) export/examples/de/PDF-comment_pdf4_systemF (Failed) export/examples/de/PDF-comment_pdf5_texF (Failed) export/examples/fr/PDF-comment_dvi3_texF (Failed) export/examples/fr/PDF-comment_dvi3_systemF (Failed) export/examples/fr/PDF-comment_pdf4_texF (Failed) export/examples/fr/PDF-comment_pdf4_systemF (Failed) export/examples/fr/PDF-comment_pdf5_texF (Failed) Finally, compiling with current master (02db1833) gives: INVERTED_SEE-README.ctest_export/examples/PDF-comment_dvi (Failed) INVERTED_SEE-README.ctest_export/examples/PDF-comment_pdf (Failed) INVERTED_SEE-README.ctest_export/examples/PDF-comment_pdf3 (Failed) INVERTED_SEE-README.ctest_export/examples/PDF-comment_pdf4_systemF (Failed) export/examples/PDF-comment_pdf5_texF (Failed) export/examples/de/PDF-comment_dvi3_texF (Failed) export/examples/de/PDF-comment_dvi3_systemF (Failed) export/examples/de/PDF-comment_pdf4_texF (Failed) export/examples/de/PDF-comment_pdf5_texF (Failed) export/examples/fr/PDF-comment_dvi3_texF (Failed) export/examples/fr/PDF-comment_dvi3_systemF (Failed) export/examples/fr/PDF-comment_pdf4_texF (Failed) export/examples/fr/PDF-comment_pdf5_texF (Failed) It seems that XeTeX with system fonts was broken by this commit but then it was fixed later on. The same thing happened for a couple of the other tests. So I guess we don't have to look at those. The following, however, were broken by this commit and the results from master indicate that they're still broken: export/examples/PDF-comment_pdf5_texF (Failed) export/examples/de/PDF-comment_pdf4_texF (Failed) export/examples/de/PDF-comment_pdf5_texF (Failed) export/examples/fr/PDF-comment_pdf5_texF (Failed) Recall that pdf5 refers to compiling with LuaTeX and pdf4 refers to compiling with XeTeX. All of the four above tests are regarding TeX fonts. I have checked in the commit before this one that the PDFs of the above four compilations look OK to me at first look (i.e. not garbled from what I see). Scott
Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".
On Wed, Oct 28, 2015 at 10:28:00PM +, Guenter Milde wrote: > On 2015-10-28, Scott Kostyshak wrote: > > On Wed, Oct 28, 2015 at 09:31:50AM +, Guenter Milde wrote: > >> On 2015-10-26, Guenter Milde wrote: > >> > On 2015-10-26, Scott Kostyshak wrote: > >> >> Has anything changed in the last couple of days? > > >> I'l commit an updated FIXME. > > > This does indeed sound tricky. Can you figure out the necessary items to > > condition on from BufferParams or is not everything stored in there? > > It became a bit simpler after going through the alternatives: > > In pseudo-code: I looked at the pseudo-code but since I don't know this code well it's not clear if you still need my help. If you have access to the buffer params (which my patch gave), is that enough? Can you test with the patch I posted before? Or is something else missing? In regards to "Dont forget to keep this check in sync with the check above!" this seems to be a good situation for a helper function in order to share code. Scott
Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".
On 2015-10-28, Scott Kostyshak wrote: > On Wed, Oct 28, 2015 at 09:31:50AM +, Guenter Milde wrote: >> On 2015-10-26, Guenter Milde wrote: >> > On 2015-10-26, Scott Kostyshak wrote: >> >> Has anything changed in the last couple of days? >> I'l commit an updated FIXME. > This does indeed sound tricky. Can you figure out the necessary items to > condition on from BufferParams or is not everything stored in there? It became a bit simpler after going through the alternatives: In pseudo-code: if (PDFinfo has unencodable characters): if (inputenc or luainputenc is loaded): os << "\\inputencoding{utf8}\n" << setEncoding("UTF-8"); else if (Xe/LuaTeX) // support UTF-8 also without inputenc os << setEncoding("UTF-8"); else convert with lib/unicodesymbols or report an error Checking for availability of the \inputencoding command is nontrivial. I tried to get the relevant info from BufferParams::writeEncodingPreamble and put them in an expanded FIXME (but did not test under which name they are available in PDFOptions.cpp). Günter Here is a patch: Exec: git 'diff' 'PDFOptions.cpp' 2>&1 Dir: /usr/local/src/lyx/src/ diff --git a/src/PDFOptions.cpp b/src/PDFOptions.cpp index 6646888..729ee10 100644 --- a/src/PDFOptions.cpp +++ b/src/PDFOptions.cpp @@ -175,20 +175,25 @@ void PDFOptions::writeLaTeX(OutputParams & runparams, otexstream & os, } // hyperref expects LICR macros for non-ASCII chars. With Xe/LuaTeX utf-8 works, too. - // Usually, "(lua)inputenc" converts the input to LICR. + // Usually, (lua)inputenc converts the input to LICR. // As hyperref provides good coverage for \inputencoding{utf8}, we can try // this if the current input encoding does not support a character. - // FIXME: inputenc (part 1 of 2) - // Replace the "FullUnicode" check with - // check for loading of inputenc or luainputenc package - // (see BufferParams::writeEncodingPreamble and runparams.encoding->package()). - // Otherwise \inputencoding is not defined - // (e.g. if "latex-encoding" is set to "ascii"). - // Dont forget to keep the check below (part 2) in sync! - if (need_unicode && enc && enc->iconvName() != "UTF-8" - &&!runparams.isFullUnicode()) { - os << "\\inputencoding{utf8}\n" - << setEncoding("UTF-8"); + if (need_unicode && enc && enc->iconvName() != "UTF-8") { + // FIXME check for loading of inputenc or luainputenc package + // (See BufferParams::writeEncodingPreamble and runparams.encoding->package().) + // Dont forget to keep the check for the re-set below in sync! + // if (encoding().package() == Encoding::inputenc + // && inputencoding != "default" && inputencoding != "ascii" + // && !encodings.empty() && !features.isRequired("japanese") + //) // inputenc or luainputenc package provided + os << "\\inputencoding{utf8}\n" + << setEncoding("UTF-8"); + // else if (runparams.isFullUnicode()) // Xe/LuaTeX support UTF-8 also without inputenc + //os << setEncoding("UTF-8"); + // else + // FIXME: convert with lib/unicodesymbols or report an error! + //(cf. the check and report for unencodable character + //in the user preamble in BufferParams.cpp) } // If hyperref is loaded by the document class, we output // \hypersetup \AtBeginDocument if hypersetup is not (yet) @@ -204,11 +209,18 @@ void PDFOptions::writeLaTeX(OutputParams & runparams, otexstream & os, << "\\fi\n"; } else os << from_utf8(opt); - // FIXME: inputenc (part 2 of 2) - if (need_unicode && enc && enc->iconvName() != "UTF-8" - &&!runparams.isFullUnicode()) { + if (need_unicode && enc && enc->iconvName() != "UTF-8") { + // FIXME check for loading of inputenc or luainputenc package + // (See BufferParams::writeEncodingPreamble and runparams.encoding->package().) + // Dont forget to keep this check in sync with the check above! + // if (encoding().package() == Encoding::inputenc + // && inputencoding != "default" && inputencoding != "ascii" + // && !encodings.empty() && !features.isRequired("japanese") + //) // inputenc or luainputenc package provided os << setEncoding(enc->iconvName()) << "\\inputencoding{" << from_ascii(enc->latexName()) << "}\n"; + // else if (runparams.isFullUnicode()) // Xe/LuaTeX, reset encoding + //os << setEncoding(enc->iconvName()); } }
Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".
On Wed, Oct 28, 2015 at 09:31:50AM +, Guenter Milde wrote: > On 2015-10-26, Guenter Milde wrote: > > On 2015-10-26, Scott Kostyshak wrote: > >> Has anything changed in the last couple of days? > > Today, I found out that the problem is tricky: > > As the PDF-Info does not use TeX fonts, Xe/LuaTeX handling (in principle) > does not depend on the "use-nonTeXfonts" setting. However, as we use > "luainputenc" with LuaTeX and 8-bit encodings, we have to differentiate: > > LuaTeX: if "luainputenc" is used (i.e. with 8-bit input encodings) > this is called before the hyperref setup and hence the 8-bit > encoding is active -- "luainputenc" transformes the bytes into LICR -- > and we need the same \inputenc wrapper as with 8-bit TeX. > > XeTeX: does not use "inputenc", the input encoding is utf-8 > (but we need to export LICR ("ascii") for the document parts that > are typeset with TeX fonts). > > "hyperref" with XeTeX and "raw utf8" and works fine with any Unicode > character. > > However, we need the > << setEncoding("UTF-8"); > (without the > os << "\\inputencoding{utf8}\n" > ) > For XeTeX and non-TeX fonts. > > > I'l commit an updated FIXME. > > > It may well be, that the FIXME in PDFOptions is not enaugh - maybe we have > > to push the content through unicodesymbols conversion, too. > > This is true for "latexencoding"="ascii" with 8-bit TeX, see my other post. This does indeed sound tricky. Can you figure out the necessary items to condition on from BufferParams or is not everything stored in there? Scott
Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".
On 2015-10-26, Guenter Milde wrote: > On 2015-10-26, Scott Kostyshak wrote: >> On Fri, Oct 23, 2015 at 07:31:49PM +, Guenter Milde wrote: > With the latest 2 fixes (36b9645b5de6d463b4909 and 2aa65fdc8d2cde1ba863), > all 4 test files compile. >>> >> >> Includes "FIXME"s at places where further action is required >>> >> >> to get the XeTeX export right but I don't know how. >>> >> >> ... >>> >> >> + &&!runparams.isFullUnicode()) { // FIXME: check must be >>> >> >> done for useNonTeXFonts! >>> >> >> os << "\\inputencoding{utf8}\n" >>> >> >> << setEncoding("UTF-8"); >>> >> > So to make sure I understand what you want with this FIXME is you >>> >> > would like to do something like params().useNonTeXFonts as you do >>> >> > in Buffer.cpp but in PDFOptions.cpp it's not clear how to access >>> >> > that? >>> >> It may also be that I misunderstand what is done there, but >>> >> basically, yes: I believe that the test at these places must be for >>> >> "useNonTeXFonts" instead of "isFullUnicode", because XeTeX/LuaTeX >>> >> with TeX fonts behave more similar to 8-bit TeX regarding the in- >>> >> and output encodings. >>> > OK, what about the attached patch? Can you check that it does what you >>> > want as far as the two FIXMEs in PDFOptions.cpp? ... >> Has anything changed in the last couple of days? Today, I found out that the problem is tricky: As the PDF-Info does not use TeX fonts, Xe/LuaTeX handling (in principle) does not depend on the "use-nonTeXfonts" setting. However, as we use "luainputenc" with LuaTeX and 8-bit encodings, we have to differentiate: LuaTeX: if "luainputenc" is used (i.e. with 8-bit input encodings) this is called before the hyperref setup and hence the 8-bit encoding is active -- "luainputenc" transformes the bytes into LICR -- and we need the same \inputenc wrapper as with 8-bit TeX. XeTeX: does not use "inputenc", the input encoding is utf-8 (but we need to export LICR ("ascii") for the document parts that are typeset with TeX fonts). "hyperref" with XeTeX and "raw utf8" and works fine with any Unicode character. However, we need the << setEncoding("UTF-8"); (without the os << "\\inputencoding{utf8}\n" ) For XeTeX and non-TeX fonts. I'l commit an updated FIXME. > It may well be, that the FIXME in PDFOptions is not enaugh - maybe we have > to push the content through unicodesymbols conversion, too. This is true for "latexencoding"="ascii" with 8-bit TeX, see my other post. Günter
Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".
On 2015-10-26, Scott Kostyshak wrote: > On Fri, Oct 23, 2015 at 07:31:49PM +, Guenter Milde wrote: >> On 2015-10-23, Scott Kostyshak wrote: >> > On Tue, Oct 20, 2015 at 09:10:05PM +, Guenter Milde wrote: >> >> On 2015-10-20, Scott Kostyshak wrote: >> >> > On Tue, Oct 20, 2015 at 07:16:44PM +0200, Günter Milde wrote: >> >> >> commit 1523fc6023440f10ca0d82a681ded5c060d8fd33 >> >> >> Author: Günter Milde >> >> >> Date: Tue Oct 20 19:14:39 2015 +0200 >> >> >> Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems". >> >> >> Fixes output for 3 of the 4 test lyx-files. With the latest 2 fixes (36b9645b5de6d463b4909 and 2aa65fdc8d2cde1ba863), all 4 test files compile. >> >> >> Includes "FIXME"s at places where further action is required >> >> >> to get the XeTeX export right but I don't know how. >> >> >> ... >> >> >> + &&!runparams.isFullUnicode()) { // FIXME: check must be >> >> >> done for useNonTeXFonts! >> >> >>os << "\\inputencoding{utf8}\n" >> >> >> << setEncoding("UTF-8"); >> >> > So to make sure I understand what you want with this FIXME is you >> >> > would like to do something like params().useNonTeXFonts as you do >> >> > in Buffer.cpp but in PDFOptions.cpp it's not clear how to access >> >> > that? >> >> It may also be that I misunderstand what is done there, but >> >> basically, yes: I believe that the test at these places must be for >> >> "useNonTeXFonts" instead of "isFullUnicode", because XeTeX/LuaTeX >> >> with TeX fonts behave more similar to 8-bit TeX regarding the in- >> >> and output encodings. >> > OK, what about the attached patch? Can you check that it does what you >> > want as far as the two FIXMEs in PDFOptions.cpp? >> From what I see (and with my limited understanding of C++), I believe the >> fix to be OK. > Has anything changed in the last couple of days? Specifically, are the > commits 2aa65fdc and 36b9645b relevant to this discussion? Unless you modelled your patch on the wrong implementation (missing negation) before, no. > If nothing has changed, then I will proceed to work on extending this > patch to address the other FIXMEs that you added. As already said, I'd prefer to know what the code in question actually does and have an example to test the current and later behaviour. Probabely, ru/Intro.lyx could be minimalized to an example - it currently fails with non-TeX fonts and XeTeX due to (amongst other problems) an \inputencoding It may well be, that the FIXME in PDFOptions is not enaugh - maybe we have to push the content through unicodesymbols conversion, too. Günter
Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".
On Mon, Oct 26, 2015 at 10:12:32AM +0100, Kornel Benko wrote: > Am Montag, 26. Oktober 2015 um 01:53:24, schrieb Scott Kostyshak > > > On Sun, Oct 25, 2015 at 01:21:01PM +0100, Kornel Benko wrote: > > > > Scott, could you please test if this fixes some more export tests? > > > > Thanks for asking! Kornel already beat me to it but I will also run the > > tests and let you know the results tomorrow (sorry for the delay). > > > > > I get now with "ctest -j12 -R 'export.*pdf4_systemF'": > > > > > > 87% tests passed, 41 tests failed out of 316 > > > But 3 of the failed are 'INVERTED', which means, they are OK now. > > > > > > > > > Previously it was > > > 80% tests passed, 62 tests failed out of 316 > > > with 2 'INVERTED'. > > > > > > So yes, some export test are fixed. > > > > This is indeed good news. But did you check if there were any > > regressions? > > Sorry, I did not. I just ran the tests over night. There are no regressions and as Kornel mentioned many tests correctly pass now. Scott
Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".
Am Montag, 26. Oktober 2015 um 01:53:24, schrieb Scott Kostyshak > On Sun, Oct 25, 2015 at 01:21:01PM +0100, Kornel Benko wrote: > > Am Sonntag, 25. Oktober 2015 um 11:31:09, schrieb Guenter Milde > > > > > On 2015-10-25, Georg Baum wrote: > > > > Guenter Milde wrote: > > > > > > >> I'll commit the patch for the patch and hope this will solve some more > > > >> of > > > >> the automatic XeTeX tests. > > > > > > > Please do. > > > > > > It is in. > > > > > > Scott, could you please test if this fixes some more export tests? > > Thanks for asking! Kornel already beat me to it but I will also run the > tests and let you know the results tomorrow (sorry for the delay). > > > I get now with "ctest -j12 -R 'export.*pdf4_systemF'": > > > > 87% tests passed, 41 tests failed out of 316 > > But 3 of the failed are 'INVERTED', which means, they are OK now. > > > > > > Previously it was > > 80% tests passed, 62 tests failed out of 316 > > with 2 'INVERTED'. > > > > So yes, some export test are fixed. > > This is indeed good news. But did you check if there were any > regressions? Sorry, I did not. > Scott Kornel signature.asc Description: This is a digitally signed message part.
Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".
On Sun, Oct 25, 2015 at 01:21:01PM +0100, Kornel Benko wrote: > Am Sonntag, 25. Oktober 2015 um 11:31:09, schrieb Guenter Milde > > > On 2015-10-25, Georg Baum wrote: > > > Guenter Milde wrote: > > > > >> I'll commit the patch for the patch and hope this will solve some more of > > >> the automatic XeTeX tests. > > > > > Please do. > > > > It is in. > > > > Scott, could you please test if this fixes some more export tests? Thanks for asking! Kornel already beat me to it but I will also run the tests and let you know the results tomorrow (sorry for the delay). > I get now with "ctest -j12 -R 'export.*pdf4_systemF'": > > 87% tests passed, 41 tests failed out of 316 > But 3 of the failed are 'INVERTED', which means, they are OK now. > > > Previously it was > 80% tests passed, 62 tests failed out of 316 > with 2 'INVERTED'. > > So yes, some export test are fixed. This is indeed good news. But did you check if there were any regressions? Scott
Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".
On Fri, Oct 23, 2015 at 07:31:49PM +, Guenter Milde wrote: > On 2015-10-23, Scott Kostyshak wrote: > > On Tue, Oct 20, 2015 at 09:10:05PM +, Guenter Milde wrote: > >> On 2015-10-20, Scott Kostyshak wrote: > >> > On Tue, Oct 20, 2015 at 07:16:44PM +0200, Günter Milde wrote: > >> >> commit 1523fc6023440f10ca0d82a681ded5c060d8fd33 > >> >> Author: Günter Milde > >> >> Date: Tue Oct 20 19:14:39 2015 +0200 > > >> >> Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems". > > >> >> Fixes output for 3 of the 4 test lyx-files. > > >> >> Includes "FIXME"s at places where further action is required to get > >> >> the XeTeX > >> >> export right but I don't know how. > >> >> ... > > >> >> + &&!runparams.isFullUnicode()) { // FIXME: check must be > >> >> done for useNonTeXFonts! > >> >> os << "\\inputencoding{utf8}\n" > >> >><< setEncoding("UTF-8"); > > >> > So to make sure I understand what you want with this FIXME is you would > >> > like to do something like params().useNonTeXFonts as you do in > >> > Buffer.cpp but in PDFOptions.cpp it's not clear how to access that? > > >> It may also be that I misunderstand what is done there, but basically, > >> yes: > >> I believe that the test at these places must be for "useNonTeXFonts" > >> instead > >> of "isFullUnicode", because XeTeX/LuaTeX with TeX fonts behave more similar > >> to 8-bit TeX regarding the in- and output encodings. > > > OK, what about the attached patch? Can you check that it does what you > > want as far as the two FIXMEs in PDFOptions.cpp? > > From what I see (and with my limited understanding of C++), I believe the > fix to be OK. Has anything changed in the last couple of days? Specifically, are the commits 2aa65fdc and 36b9645b relevant to this discussion? If nothing has changed, then I will proceed to work on extending this patch to address the other FIXMEs that you added. Scott
Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".
On 2015-10-25, Georg Baum wrote: > Guenter Milde wrote: >> Actually, the patch does now what Georg expected the old patch to do: >> set encoding to ASCII for every use of XETEX, also with non-TeX fonts. >> This fails with characters without a LICR defined by "xunicode". >> So the current best known version would be to admit the limit: > IMO this points to a serious bug and needs to be investigated further: The > condition > if (!params().useNonTeXFonts && runparams.flavor == OutputParams::XETEX) > runparams.encoding = encodings.fromLyXName("ascii"); > describes exactly what should happen. So if this sets the encoding to ascii > even if non-TeX fonts are used, then params().useNonTeXFonts must be wrong, > and this would be a serious problem. My guess would be that the code is > correct, but that the other occurence of this code in writeLaTeXSource() > does the wrong thing. If the if-condition in this one is adjusted as well, > then it will probably work. The problem was part what you described and part use of a non-minimal test file: doc/ru/Intro.lyx failed to compile with XeTeX (both, TeX fonts and non-TeX fonts) for several reasons: * wrong preamble code prevents use of non-TeX fonts (overwritten by loading lmodern.sty) * the file does not compile with "inputenc = ascii", because of a spurious \textcyr font encoding change between an accent and the base letter (see #9637) with both, pdflatex and xelatex. The missing fix in writeLaTeXSource() suggested this were due to wrong input encoding in both cases, but finally I sorted this out. So, besides the FIXME (the re-setting to "ascii" should rather be in BufferEncoding.cpp) and the doubling of code in Buffer.cpp, the fixed fix is fine and with the second fix it works reasonable with both XeTeX and LuaTeX. Thank you for the helpfull hints. Günter
Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".
Richard Heck wrote: > On 10/25/2015 07:31 AM, Guenter Milde wrote: >> On 2015-10-25, Georg Baum wrote: >>> BTW, the braces around the equality comparison are not needed, so >>> to my eyes >>> if (!params().useNonTeXFonts && runparams.flavor == OutputParams::XETEX) >>> is more readable. >> I don't know the operator precedence in C++, therefore I added the >> braces. Now you do for this case (until you forget again;-) > Better to have them than not to have them, if one isn't sure, it always > seems to me. Sure. My motivation for mentioning this was to increase the knowledge about operator precedence. If you read and write statements like this more often, then it eases your work a lot if you know at least the most important precedence rules. At least this is my experience. Georg
Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".
Guenter Milde wrote: > Actually, the patch does now what Georg expected the old patch to do: > > set encoding to ASCII for every use of XETEX, also with non-TeX fonts. > > This fails with characters without a LICR defined by "xunicode". > > So the current best known version would be to admit the limit: IMO this points to a serious bug and needs to be investigated further: The condition if (!params().useNonTeXFonts && runparams.flavor == OutputParams::XETEX) runparams.encoding = encodings.fromLyXName("ascii"); describes exactly what should happen. So if this sets the encoding to ascii even if non-TeX fonts are used, then params().useNonTeXFonts must be wrong, and this would be a serious problem. My guess would be that the code is correct, but that the other occurence of this code in writeLaTeXSource() does the wrong thing. If the if-condition in this one is adjusted as well, then it will probably work. Georg
Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".
On 10/25/2015 07:31 AM, Guenter Milde wrote: On 2015-10-25, Georg Baum wrote: BTW, the braces around the equality comparison are not needed, so to my eyes if (!params().useNonTeXFonts && runparams.flavor == OutputParams::XETEX) is more readable. I don't know the operator precedence in C++, therefore I added the braces. Better to have them than not to have them, if one isn't sure, it always seems to me. Richard
Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".
On 2015-10-25, Guenter Milde wrote: > On 2015-10-25, Georg Baum wrote: >> Guenter Milde wrote: >>> I'll commit the patch for the patch and hope this will solve some more of >>> the automatic XeTeX tests. >> Please do. > It is in. > Scott, could you please test if this fixes some more export tests? Actually, the patch does now what Georg expected the old patch to do: set encoding to ASCII for every use of XETEX, also with non-TeX fonts. This fails with characters without a LICR defined by "xunicode". So the current best known version would be to admit the limit: Günter diff --git a/src/Buffer.cpp b/src/Buffer.cpp index 73d2007..0d621b1 100644 --- a/src/Buffer.cpp +++ b/src/Buffer.cpp @@ -1588,8 +1588,9 @@ bool Buffer::makeLaTeXFile(FileName const & fname, // XeTeX with TeX fonts is only safe with ASCII encoding, // See #9740 and FIXME in BufferParams::encoding() - if (!params().useNonTeXFonts && (runparams.flavor == OutputParams::XETEX)) - runparams.encoding = encodings.fromLyXName("ascii"); + // However, the code below also sets "ascii" for XeTeX with non-TeX fonts (why?) + // if (!params().useNonTeXFonts && (runparams.flavor == OutputParams::XETEX)) + // runparams.encoding = encodings.fromLyXName("ascii"); string const encoding = runparams.encoding->iconvName(); LYXERR(Debug::LATEX, "makeLaTeXFile encoding: " << encoding << ", fname=" << fname.realPath()); Günter
Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".
Am Sonntag, 25. Oktober 2015 um 11:31:09, schrieb Guenter Milde > On 2015-10-25, Georg Baum wrote: > > Guenter Milde wrote: > > >> I'll commit the patch for the patch and hope this will solve some more of > >> the automatic XeTeX tests. > > > Please do. > > It is in. > > Scott, could you please test if this fixes some more export tests? I get now with "ctest -j12 -R 'export.*pdf4_systemF'": 87% tests passed, 41 tests failed out of 316 But 3 of the failed are 'INVERTED', which means, they are OK now. Previously it was 80% tests passed, 62 tests failed out of 316 with 2 'INVERTED'. So yes, some export test are fixed. > > BTW, the braces around the equality comparison are not needed, so > > to my eyes > > > if (!params().useNonTeXFonts && runparams.flavor == OutputParams::XETEX) > > > is more readable. > > I don't know the operator precedence in C++, therefore I added the braces. > > > Günter Kornel signature.asc Description: This is a digitally signed message part.
Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".
On 2015-10-25, Georg Baum wrote: > Guenter Milde wrote: >> I'll commit the patch for the patch and hope this will solve some more of >> the automatic XeTeX tests. > Please do. It is in. Scott, could you please test if this fixes some more export tests? > BTW, the braces around the equality comparison are not needed, so > to my eyes > if (!params().useNonTeXFonts && runparams.flavor == OutputParams::XETEX) > is more readable. I don't know the operator precedence in C++, therefore I added the braces. Günter
Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".
Guenter Milde wrote: > Just to be sure we don't misunderstand: The question is about tests > compiling the documentation "as-is" (i.e. with *TeX fonts*) using > XeTeX. Thanks for the clarification, I think we misunderstood. For XeTeX + TeX fonts I agree. > For me, Development.lyx compiles fine with XeTeX an and my patch, if I > set "use non TeX fonts" to TRUE. You are right, I forgot that. > Please test again with a file that selects "non-TeX fonts". If in this > case my patch exports in an ecoding other than utf8-plain, this is a bug. > However, I did not experience this combination here. I do not have one. What I missed was the initialization of the encoding to utf8-plain in BufferParams::encoding(). Therefore I thought that the encoding for non-TeX-fonts was now not correctly set anymore, and since Development.lyx compiled again after my change I took this as a confirmation. In the end, I made two mistakes which cancelled each other. > Unfortunately, the case "XeTeX + fontspec" was not tested. The tests use > XeTeX with unmodified documents and unless bug #... is solved this means > they test the corner case XeTeX+TeX-fonts! Therefore I think that adding a "default" setting for the fonts (first part of #9744) is so important. > I'll commit the patch for the patch and hope this will solve some more of > the automatic XeTeX tests. Please do. BTW, the braces around the equality comparison are not needed, so to my eyes if (!params().useNonTeXFonts && runparams.flavor == OutputParams::XETEX) is more readable. Georg
Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".
On 2015-10-23, Georg Baum wrote: > [-- Type: text/plain, Encoding: 7bit --] > Guenter Milde wrote: >> On 2015-10-21, Scott Kostyshak wrote: >>> It compiles fine before this commit. >> Yes, but only "per accident": Development.lyx only uses some non-ASCII >> characters, and these are also at the same place in Unicode and T1. > It was not by accident. It did compile before correctly because the .tex > file was encoded in utf8, as XeTeX expects it, and this was explicitly > programmed. After your change, the file is encoded in latin1, which is > interpreted as utf8 by XeTeX, which does of course not work and causes the > "Invalid UTF-8" error above. Just to be sure we don't misunderstand: The question is about tests compiling the documentation "as-is" (i.e. with *TeX fonts*) using XeTeX. In this case, while XeTeX expects utf8, it does not convert non-ASCII characters to the "LaTeX internal character representation" (LICR), but forwards it directly to the font character selection. Therefore, * any character in the 128 ... 255 range (Latin extended) selects a glyph from the same range in the T1 TeX font encoding - some of them coincide, others differ. * any character in the range above 255 fails with TeX fonts. This is the reason for the many "missing –" (en-dash) errors. The proper fix would be to restrict the exported file to ASCII characters for the combination "XeTeX + TeX fonts". (Unfortunately, I did not manage to achieve this and therfore declared my patch as a "partial fix".) > The fix is simple: Re-add the deleted lines to set the encoding to utf8- > plain (see attachment). Then Development.lyx compiles again for me. For me, Development.lyx compiles fine with XeTeX an and my patch, if I set "use non TeX fonts" to TRUE. > The encoding must always be utf8 for full unicode backends if non-TeX > fonts are used. On this, we agree. Therefore my patch tests for "non-TeX fonts" - if non-TeX fonts are used, there is no need to test for the engine, as both allowed engines (XeTeX, LuaTeX) expect "utf8-plain" as input encoding. > I guess that you were mislead by the comment, which suggested that > setting the encoding was done for TeX fonts, but this was only a part > of the truth: It was done for non-TeX fonts as well. Please test again with a file that selects "non-TeX fonts". If in this case my patch exports in an ecoding other than utf8-plain, this is a bug. However, I did not experience this combination here. >> So, in this example the patch indeed broke the XeTeX export, while it may >> be that in other "broken" example files export with XeTeX run without >> error report but still produced wrong results! > Sure. A passing test after a change does not tell that the change did > not break anything. For such a statement we would need to keep > references of the exported files, and this is only possible for > selected, small examples, not for out complete documentation that does > change a lot. Only a test which fails after a change and did not fail > before tells that something became worse (it does not tell that the > previous state was ok, only that it was less broken). Yes. However, for "Development.lyx with TeX-fonts and XeTeX", I could establish, that the "broken" state does not indicate that the partial fix is wrong -- the now broken test is due to it beeing incomplete. ... > But it is quite likely that XeTeX export for non-TeX fonts (the main > use case for XeTeX) was correct before, since this does simply use utf8 > encoding throughout the whole file, which can represent every symbol > LyX does understand. Indeed. (Except for cases where missing characters were not reported before but lead to errors now, but this is a different topic.) Unfortunately, the case "XeTeX + fontspec" was not tested. The tests use XeTeX with unmodified documents and unless bug #... is solved this means they test the corner case XeTeX+TeX-fonts! But: your patch showed me where and why my patch failed: the correct version should be diff --git a/src/Buffer.cpp b/src/Buffer.cpp index 829616a..73d2007 100644 --- a/src/Buffer.cpp +++ b/src/Buffer.cpp @@ -1588,7 +1588,7 @@ bool Buffer::makeLaTeXFile(FileName const & fname, // XeTeX with TeX fonts is only safe with ASCII encoding, // See #9740 and FIXME in BufferParams::encoding() - if (params().useNonTeXFonts && (runparams.flavor == OutputParams::XETEX)) + if (!params().useNonTeXFonts && (runparams.flavor == OutputParams::XETEX)) runparams.encoding = encodings.fromLyXName("ascii"); string const encoding = runparams.encoding->iconvName(); i.e. I missed a negation! With this correction, also Development.lyx + TeX fonts + XeTeX works :-) I'll commit the patch for the patch and hope this will solve some more of the automatic XeTeX tests. Thanks, Günter
Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".
On Fri, Oct 23, 2015 at 11:05:59PM +0200, Georg Baum wrote: > The fix is simple: Re-add the deleted lines to set the encoding to utf8- > plain (see attachment). Then Development.lyx compiles again for me. The > encoding must always be utf8 for full unicode backends if non-TeX fonts are > used. I guess that you were mislead by the comment, which suggested that > setting the encoding was done for TeX fonts, but this was only a part of the > truth: It was done for non-TeX fonts as well. > > > Before my partial patch, the "latex encoding" was set to "utf-8" for > > export to XeTeX and LuaTeX, regardless of the used font encoding. This can > > lead to both, now reported errors for missing characters and wrong output > > for some non-ACII characters in the "latin extended" block. > > I think the part of the patch that changes the behaviour for XeTeX + TeX > fonts was fine and I am pretty sure that it fixes many more problems for > this combination than it creates. Let me know if it would be useful to compare the results of the export tests. Just let me know which two states to compare (e.g. master + Georg's patch vs. master). Or if you have confidence in the patch, feel free to put it in and I can just test master against master before the patch. > >> The commit also fixed a lot of ctests. > > > > Even here, we cannot be sure that XeTeX export is correct now. Agreed. Scott
Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".
Guenter Milde wrote: > On 2015-10-21, Scott Kostyshak wrote: > >> It compiles fine before this commit. > > Yes, but only "per accident": Development.lyx only uses some non-ASCII > characters, and these are also at the same place in Unicode and T1. It was not by accident. It did compile before correctly because the .tex file was encoded in utf8, as XeTeX expects it, and this was explicitly programmed. After your change, the file is encoded in latin1, which is interpreted as utf8 by XeTeX, which does of course not work and causes the "Invalid UTF-8" error above. The fix is simple: Re-add the deleted lines to set the encoding to utf8- plain (see attachment). Then Development.lyx compiles again for me. The encoding must always be utf8 for full unicode backends if non-TeX fonts are used. I guess that you were mislead by the comment, which suggested that setting the encoding was done for TeX fonts, but this was only a part of the truth: It was done for non-TeX fonts as well. > Before my partial patch, the "latex encoding" was set to "utf-8" for > export to XeTeX and LuaTeX, regardless of the used font encoding. This can > lead to both, now reported errors for missing characters and wrong output > for some non-ACII characters in the "latin extended" block. I think the part of the patch that changes the behaviour for XeTeX + TeX fonts was fine and I am pretty sure that it fixes many more problems for this combination than it creates. > So, in this example the patch indeed broke the XeTeX export, while it may > be that in other "broken" example files export with XeTeX run without > error report but still produced wrong results! Sure. A passing test after a change does not tell that the change did not break anything. For such a stement we would need to keep references of the exported files, and this is only possible for selected, small examples, not for out complete documentation that does change a lot. Only a test which fails after a change and did not fail before tells that something became worse (it does not tell that the previous state was ok, onlyx that it was less broken). >> The commit also fixed a lot of ctests. > > Even here, we cannot be sure that XeTeX export is correct now. But it is quite likely that XeTeX export for non-TeX fonts (the main use case for XeTeX) was correct before, since this does simply use utf8 encoding throughout the whole file, which can represent every symbol LyX does understand. Georg diff --git a/src/Buffer.cpp b/src/Buffer.cpp index 829616a..1f5f7c4 100644 --- a/src/Buffer.cpp +++ b/src/Buffer.cpp @@ -1590,6 +1590,8 @@ bool Buffer::makeLaTeXFile(FileName const & fname, // See #9740 and FIXME in BufferParams::encoding() if (params().useNonTeXFonts && (runparams.flavor == OutputParams::XETEX)) runparams.encoding = encodings.fromLyXName("ascii"); + else if (runparams.isFullUnicode()) + runparams.encoding = encodings.fromLyXName("utf8-plain"); string const encoding = runparams.encoding->iconvName(); LYXERR(Debug::LATEX, "makeLaTeXFile encoding: " << encoding << ", fname=" << fname.realPath());
Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".
On 2015-10-23, Scott Kostyshak wrote: > On Tue, Oct 20, 2015 at 09:10:05PM +, Guenter Milde wrote: >> On 2015-10-20, Scott Kostyshak wrote: >> > On Tue, Oct 20, 2015 at 07:16:44PM +0200, Günter Milde wrote: >> >> commit 1523fc6023440f10ca0d82a681ded5c060d8fd33 >> >> Author: Günter Milde >> >> Date: Tue Oct 20 19:14:39 2015 +0200 >> >> Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems". >> >> Fixes output for 3 of the 4 test lyx-files. >> >> Includes "FIXME"s at places where further action is required to get >> >> the XeTeX >> >> export right but I don't know how. >> >> ... >> >> + &&!runparams.isFullUnicode()) { // FIXME: check must be done for >> >> useNonTeXFonts! >> >> os << "\\inputencoding{utf8}\n" >> >> << setEncoding("UTF-8"); >> > So to make sure I understand what you want with this FIXME is you would >> > like to do something like params().useNonTeXFonts as you do in >> > Buffer.cpp but in PDFOptions.cpp it's not clear how to access that? >> It may also be that I misunderstand what is done there, but basically, yes: >> I believe that the test at these places must be for "useNonTeXFonts" instead >> of "isFullUnicode", because XeTeX/LuaTeX with TeX fonts behave more similar >> to 8-bit TeX regarding the in- and output encodings. > OK, what about the attached patch? Can you check that it does what you > want as far as the two FIXMEs in PDFOptions.cpp? >From what I see (and with my limited understanding of C++), I believe the fix to be OK. To be sure, I'd need a use case for this code. (Maybe the commit message you found with "blame" tells something helpfull about where it is needed?) The we could compare the exported file before and after patching and see which works better. > If so, hopefully someone else can confirm that it is reasonable to pass > the buffer parameters in this situation. Then after that I could extend > the fix to address the other FIXMEs that you have regarding > useNonTeXFonts. However, the main problem is ensuring the "latex-encoding" is "ASCII" for the combination of XeTeX and "TeX fonts". > Günter are you familiar with "git blame"? It is a useful tool for > purposes such as these where we might want to check with the person who > introduced a certain line of code. For example, if before your commit > (otherwise it will say you are the author of that line because you > edited it to put the comment) I > do > git blame PDFOptions.cpp > and then I search (with '/') for "isFullUnicode" it takes me to the > first instance and tells me that Jürgen introduced this line in the > commit baaceb10. So now we ping Jürgen (now CC'ed) and see what his > opinion is on this. Thanks Günter
Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".
On 2015-10-21, Scott Kostyshak wrote: > [-- Type: text/plain, Encoding: 8bit --] > On Tue, Oct 20, 2015 at 09:10:05PM +, Guenter Milde wrote: >> It may also be that I misunderstand what is done there, but basically, yes: >> I believe that the test at these places must be for "useNonTeXFonts" instead >> of "isFullUnicode", because XeTeX/LuaTeX with TeX fonts behave more similar >> to 8-bit TeX regarding the in- and output encodings. > This commit breaks a lot of ctests. While I was pretty sure that the partial patch will not fix all these strange use cases, I did not expect it to break working examples. > For example, the document > doc/Development.lyx no longer compiles with XeTeX (with TeX fonts). So let's have a look at the example. > I get the following error: > Missing character: There is no — in font ecrm1200! This reports a missing en-dash. Strange, because there is no en-dash in Development.lyx (otherwise it would fail before the patch with the above error. Here I get Missing character: There is no � in font ecrm1200! and this is because of: Invalid UTF-8 byte or sequence at line 300 replaced by U+FFFD. Missing character: There is no � in font ecrm1200! > It compiles fine before this commit. Yes, but only "per accident": Development.lyx only uses some non-ASCII characters, and these are also at the same place in Unicode and T1. Before my partial patch, the "latex encoding" was set to "utf-8" for export to XeTeX and LuaTeX, regardless of the used font encoding. This can lead to both, now reported errors for missing characters and wrong output for some non-ACII characters in the "latin extended" block. If you compile the example file "luatex-without-fontspec-latin9" using XeTeX with an "unpatched" LyX, you get both, the Missing character: There is no — in font ecrm1200! for the en-dash and wrong high bit characters: The input: £§¥©®°±²³´µ·½¿÷ø ÂÃÄÖÜ äöüÿß looks very different in the PDF (starting with "č" and other east-european Latin characters and ending with "ßSS" (while drag-and-drop here results in the almost correct "£§¥©®°±23 ́μ·1⁄2¿÷ø ÂÃÄÖÜ äöüÿß"). However, the German umlauts are kept. And actually the small o-umlaut "ö" is the only non-ASCII character in the whole file Development.lyx. So, in this example the patch indeed broke the XeTeX export, while it may be that in other "broken" example files export with XeTeX run without error report but still produced wrong results! > The commit also fixed a lot of ctests. Even here, we cannot be sure that XeTeX export is correct now. > Attached are the lists of tests that fail before this commit and the > list that fails after this commit. By comparing them you can see which > ones this commit changed. Let me know if you have any questions. I wonder, whether we could limit the number of tests for the corner case "XeTeX/LuaTeX with TeX fonts". In my view, we should just admit that this is currently broken and the proper fix would be "automatic" selection of non-TeX fonts for XeTeX/LuaTeX (if not explicitely set otherwise by the user). Otherwise, nothing will spare us a close examination of all these files and the export result "per hand"... Günter
Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".
On Tue, Oct 20, 2015 at 09:10:05PM +, Guenter Milde wrote: > On 2015-10-20, Scott Kostyshak wrote: > > On Tue, Oct 20, 2015 at 07:16:44PM +0200, Günter Milde wrote: > >> commit 1523fc6023440f10ca0d82a681ded5c060d8fd33 > >> Author: Günter Milde > >> Date: Tue Oct 20 19:14:39 2015 +0200 > > >> Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems". > > >> Fixes output for 3 of the 4 test lyx-files. > > >> Includes "FIXME"s at places where further action is required to get > >> the XeTeX > >> export right but I don't know how. > >> ... > > >> + &&!runparams.isFullUnicode()) { // FIXME: check must be done for > >> useNonTeXFonts! > >>os << "\\inputencoding{utf8}\n" > >> << setEncoding("UTF-8"); > > > So to make sure I understand what you want with this FIXME is you would > > like to do something like params().useNonTeXFonts as you do in > > Buffer.cpp but in PDFOptions.cpp it's not clear how to access that? > > It may also be that I misunderstand what is done there, but basically, yes: > I believe that the test at these places must be for "useNonTeXFonts" instead > of "isFullUnicode", because XeTeX/LuaTeX with TeX fonts behave more similar > to 8-bit TeX regarding the in- and output encodings. OK, what about the attached patch? Can you check that it does what you want as far as the two FIXMEs in PDFOptions.cpp? If so, hopefully someone else can confirm that it is reasonable to pass the buffer parameters in this situation. Then after that I could extend the fix to address the other FIXMEs that you have regarding useNonTeXFonts. Günter are you familiar with "git blame"? It is a useful tool for purposes such as these where we might want to check with the person who introduced a certain line of code. For example, if before your commit (otherwise it will say you are the author of that line because you edited it to put the comment) I do git blame PDFOptions.cpp and then I search (with '/') for "isFullUnicode" it takes me to the first instance and tells me that Jürgen introduced this line in the commit baaceb10. So now we ping Jürgen (now CC'ed) and see what his opinion is on this. Scott diff --git a/src/BufferParams.cpp b/src/BufferParams.cpp index 537178e..60f1a24 100644 --- a/src/BufferParams.cpp +++ b/src/BufferParams.cpp @@ -1901,8 +1901,9 @@ bool BufferParams::writeLaTeX(otexstream & os, LaTeXFeatures & features, // to access the stream itself in PDFOptions. os << lyxpreamble; - OutputParams tmp_params = features.runparams(); - pdfoptions().writeLaTeX(tmp_params, os, + OutputParams output_params = features.runparams(); + BufferParams buffer_params = *this; + pdfoptions().writeLaTeX(output_params, buffer_params, os, features.isProvided("hyperref")); // set back for the rest lyxpreamble.clear(); diff --git a/src/PDFOptions.cpp b/src/PDFOptions.cpp index afbd5e5..2a82f44 100644 --- a/src/PDFOptions.cpp +++ b/src/PDFOptions.cpp @@ -89,8 +89,8 @@ void PDFOptions::writeFile(ostream & os) const } -void PDFOptions::writeLaTeX(OutputParams & runparams, otexstream & os, - bool hyperref_already_provided) const +void PDFOptions::writeLaTeX(OutputParams & runparams, BufferParams & bufferparams, + otexstream & os, bool hyperref_already_provided) const { // FIXME Unicode string opt; @@ -176,7 +176,7 @@ void PDFOptions::writeLaTeX(OutputParams & runparams, otexstream & os, // hyperref expects utf8! if (need_unicode && enc && enc->iconvName() != "UTF-8" - &&!runparams.isFullUnicode()) { // FIXME: check must be done for useNonTeXFonts! + && !bufferparams.useNonTeXFonts) { os << "\\inputencoding{utf8}\n" << setEncoding("UTF-8"); } @@ -196,7 +196,7 @@ void PDFOptions::writeLaTeX(OutputParams & runparams, otexstream & os, os << from_utf8(opt); if (need_unicode && enc && enc->iconvName() != "UTF-8" - &&!runparams.isFullUnicode()) { // FIXME: check for useNonTeXFonts! + && !bufferparams.useNonTeXFonts) { os << setEncoding(enc->iconvName()) << "\\inputencoding{" << from_ascii(enc->latexName()) << "}\n"; } diff --git a/src/PDFOptions.h b/src/PDFOptions.h index d74caf4..f4d5250 100644 --- a/src/PDFOptions.h +++ b/src/PDFOptions.h @@ -13,6 +13,7 @@ #define PDFOPTIONS_H #include "OutputParams.h" +#include "BufferParams.h" #include "support/strfwd.h" @@ -36,7 +37,7 @@ public: /// output to lyx header void writeFile(std::ostream &) const; /// output to tex header - void writeLaTeX(OutputParams &, otexstream &, + void writeLaTeX(OutputParams &, BufferParams &, otexstream &, bool
Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".
On Tue, Oct 20, 2015 at 09:10:05PM +, Guenter Milde wrote: > It may also be that I misunderstand what is done there, but basically, yes: > I believe that the test at these places must be for "useNonTeXFonts" instead > of "isFullUnicode", because XeTeX/LuaTeX with TeX fonts behave more similar > to 8-bit TeX regarding the in- and output encodings. This commit breaks a lot of ctests. For example, the document doc/Development.lyx no longer compiles with XeTeX (with TeX fonts). I get the following error: Missing character: There is no — in font ecrm1200! It compiles fine before this commit. The commit also fixed a lot of ctests. Attached are the lists of tests that fail before this commit and the list that fails after this commit. By comparing them you can see which ones this commit changed. Let me know if you have any questions. Scott 209:export/doc/Additional_pdf4_texF 217:export/doc/Customization_dvi3_texF 222:export/doc/Customization_pdf4_texF 224:export/doc/Customization_pdf5_texF 225:export/doc/Customization_pdf5_systemF 274:export/doc/EmbeddedObjects_pdf4_texF 275:export/doc/EmbeddedObjects_pdf4_systemF 300:export/doc/Intro_pdf4_texF 313:export/doc/LFUNs_pdf4_texF 321:export/doc/LaTeXConfig_dvi3_texF 326:export/doc/LaTeXConfig_pdf4_texF 328:export/doc/LaTeXConfig_pdf5_texF 334:export/doc/Math_dvi3_texF 335:export/doc/Math_dvi3_systemF 339:export/doc/Math_pdf4_texF 341:export/doc/Math_pdf5_texF 342:export/doc/Math_pdf5_systemF 368:export/doc/Tutorial_pdf4_texF 376:export/doc/UserGuide_dvi3_texF 381:export/doc/UserGuide_pdf4_texF 383:export/doc/UserGuide_pdf5_texF 394:export/doc/attic/DocStyle_pdf4_texF 402:export/doc/attic/eu_Additional_dvi3_texF 407:export/doc/attic/eu_Additional_pdf4_texF 408:INVERTED_SEE-README.ctest_export/doc/attic/eu_Additional_pdf4_systemF 409:export/doc/attic/eu_Additional_pdf5_texF 415:export/doc/attic/eu_Customization_dvi3_texF 420:export/doc/attic/eu_Customization_pdf4_texF 422:export/doc/attic/eu_Customization_pdf5_texF 428:export/doc/attic/eu_UserGuide_dvi3_texF 433:export/doc/attic/eu_UserGuide_pdf4_texF 434:INVERTED_SEE-README.ctest_export/doc/attic/eu_UserGuide_pdf4_systemF 435:export/doc/attic/eu_UserGuide_pdf5_texF 441:export/doc/attic/it_Customization_dvi3_texF 446:export/doc/attic/it_Customization_pdf4_texF 448:export/doc/attic/it_Customization_pdf5_texF 449:export/doc/attic/it_Customization_pdf5_systemF 454:export/doc/attic/it_UserGuide_dvi3_texF 455:export/doc/attic/it_UserGuide_dvi3_systemF 459:export/doc/attic/it_UserGuide_pdf4_texF 460:export/doc/attic/it_UserGuide_pdf4_systemF 461:export/doc/attic/it_UserGuide_pdf5_texF 472:export/doc/attic/pl_Additional_pdf4_texF 473:INVERTED_SEE-README.ctest_export/doc/attic/pl_Additional_pdf4_systemF 480:export/doc/attic/sk_UserGuide_dvi3_texF 485:export/doc/attic/sk_UserGuide_pdf4_texF 487:export/doc/attic/sk_UserGuide_pdf5_texF 498:export/doc/ca/Intro_pdf4_texF 511:export/doc/cs/Tutorial_pdf4_texF 524:export/doc/da/Intro_pdf4_texF 537:export/doc/de/Additional_pdf4_texF 545:export/doc/de/Customization_dvi3_texF 550:export/doc/de/Customization_pdf4_texF 552:export/doc/de/Customization_pdf5_texF 553:export/doc/de/Customization_pdf5_systemF 589:export/doc/de/EmbeddedObjects_pdf4_texF 592:export/doc/de/EmbeddedObjects_pdf5_systemF 615:export/doc/de/Intro_pdf4_texF 623:export/doc/de/Math_dvi3_texF 624:export/doc/de/Math_dvi3_systemF 628:export/doc/de/Math_pdf4_texF 630:export/doc/de/Math_pdf5_texF 631:export/doc/de/Math_pdf5_systemF 654:export/doc/de/Tutorial_pdf4_texF 662:export/doc/de/UserGuide_dvi3_texF 667:export/doc/de/UserGuide_pdf4_texF 669:export/doc/de/UserGuide_pdf5_texF 670:export/doc/de/UserGuide_pdf5_systemF 680:export/doc/el/Intro_pdf4_texF 683:export/doc/el/Intro_pdf5_systemF 693:export/doc/es/Additional_pdf4_texF 706:export/doc/es/Customization_pdf4_texF 709:export/doc/es/Customization_pdf5_systemF 745:export/doc/es/EmbeddedObjects_pdf4_texF 748:export/doc/es/EmbeddedObjects_pdf5_systemF 771:export/doc/es/Intro_pdf4_texF 779:export/doc/es/Math_dvi3_texF 784:export/doc/es/Math_pdf4_texF 786:export/doc/es/Math_pdf5_texF 787:export/doc/es/Math_pdf5_systemF 810:export/doc/es/Tutorial_pdf4_texF 813:export/doc/es/Tutorial_pdf5_systemF 818:export/doc/es/UserGuide_dvi3_texF 823:export/doc/es/UserGuide_pdf4_texF 825:export/doc/es/UserGuide_pdf5_texF 826:export/doc/es/UserGuide_pdf5_systemF 836:export/doc/eu/Intro_pdf4_texF 849:export/doc/eu/Tutorial_pdf4_texF 857:export/doc/fr/Additional_dvi3_texF 862:export/doc/fr/Additional_pdf4_texF 864:export/doc/fr/Additional_pdf5_texF 865:export/doc/fr/Additional_pdf5_systemF 870:export/doc/fr/Customization_dvi3_texF 875:export/doc/fr/Customization_pdf4_texF 877:export/doc/fr/Customization_pdf5_texF 878:export/doc/fr/Customization_pdf5_systemF 914:export/doc/fr/EmbeddedObjects_pdf4_texF 940:export/doc/fr/Intro_pdf4_texF 948:export/doc/fr/Math_dvi3_texF 953:export/doc/fr/Math_pdf4_texF 955:export/doc/fr/Math_pdf5_texF 979:export/doc/fr/Tutorial_pdf4_texF 987:export/doc/f
Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".
On 2015-10-20, Scott Kostyshak wrote: > On Tue, Oct 20, 2015 at 07:16:44PM +0200, Günter Milde wrote: >> commit 1523fc6023440f10ca0d82a681ded5c060d8fd33 >> Author: Günter Milde >> Date: Tue Oct 20 19:14:39 2015 +0200 >> Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems". >> Fixes output for 3 of the 4 test lyx-files. >> Includes "FIXME"s at places where further action is required to get the >> XeTeX >> export right but I don't know how. >> ... >> +&&!runparams.isFullUnicode()) { // FIXME: check must be done for >> useNonTeXFonts! >> os << "\\inputencoding{utf8}\n" >> << setEncoding("UTF-8"); > So to make sure I understand what you want with this FIXME is you would > like to do something like params().useNonTeXFonts as you do in > Buffer.cpp but in PDFOptions.cpp it's not clear how to access that? It may also be that I misunderstand what is done there, but basically, yes: I believe that the test at these places must be for "useNonTeXFonts" instead of "isFullUnicode", because XeTeX/LuaTeX with TeX fonts behave more similar to 8-bit TeX regarding the in- and output encodings. Günter
Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".
On Tue, Oct 20, 2015 at 07:16:44PM +0200, Günter Milde wrote: > commit 1523fc6023440f10ca0d82a681ded5c060d8fd33 > Author: Günter Milde > Date: Tue Oct 20 19:14:39 2015 +0200 > > Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems". > > Fixes output for 3 of the 4 test lyx-files. > > Includes "FIXME"s at places where further action is required to get the > XeTeX > export right but I don't know how. > ... > + &&!runparams.isFullUnicode()) { // FIXME: check must be done for > useNonTeXFonts! > os << "\\inputencoding{utf8}\n" > << setEncoding("UTF-8"); So to make sure I understand what you want with this FIXME is you would like to do something like params().useNonTeXFonts as you do in Buffer.cpp but in PDFOptions.cpp it's not clear how to access that? Scott