Guenter Milde wrote:
> On 2015-10-21, Scott Kostyshak wrote:
>
>> It compiles fine before this commit.
>
> Yes, but only "per accident": Development.lyx only uses some non-ASCII
> characters, and these are also at the same place in Unicode and T1.
It was not by accident. It did compile before correctly because the .tex
file was encoded in utf8, as XeTeX expects it, and this was explicitly
programmed. After your change, the file is encoded in latin1, which is
interpreted as utf8 by XeTeX, which does of course not work and causes the
"Invalid UTF-8" error above.
The fix is simple: Re-add the deleted lines to set the encoding to utf8-
plain (see attachment). Then Development.lyx compiles again for me. The
encoding must always be utf8 for full unicode backends if non-TeX fonts are
used. I guess that you were mislead by the comment, which suggested that
setting the encoding was done for TeX fonts, but this was only a part of the
truth: It was done for non-TeX fonts as well.
> Before my partial patch, the "latex encoding" was set to "utf-8" for
> export to XeTeX and LuaTeX, regardless of the used font encoding. This can
> lead to both, now reported errors for missing characters and wrong output
> for some non-ACII characters in the "latin extended" block.
I think the part of the patch that changes the behaviour for XeTeX + TeX
fonts was fine and I am pretty sure that it fixes many more problems for
this combination than it creates.
> So, in this example the patch indeed broke the XeTeX export, while it may
> be that in other "broken" example files export with XeTeX run without
> error report but still produced wrong results!
Sure. A passing test after a change does not tell that the change did not
break anything. For such a stement we would need to keep references of the
exported files, and this is only possible for selected, small examples, not
for out complete documentation that does change a lot. Only a test which
fails after a change and did not fail before tells that something became
worse (it does not tell that the previous state was ok, onlyx that it was
less broken).
>> The commit also fixed a lot of ctests.
>
> Even here, we cannot be sure that XeTeX export is correct now.
But it is quite likely that XeTeX export for non-TeX fonts (the main use
case for XeTeX) was correct before, since this does simply use utf8 encoding
throughout the whole file, which can represent every symbol LyX does
understand.
Georg
diff --git a/src/Buffer.cpp b/src/Buffer.cpp
index 829616a..1f5f7c4 100644
--- a/src/Buffer.cpp
+++ b/src/Buffer.cpp
@@ -1590,6 +1590,8 @@ bool Buffer::makeLaTeXFile(FileName const & fname,
// See #9740 and FIXME in BufferParams::encoding()
if (params().useNonTeXFonts && (runparams.flavor == OutputParams::XETEX))
runparams.encoding = encodings.fromLyXName("ascii");
+ else if (runparams.isFullUnicode())
+ runparams.encoding = encodings.fromLyXName("utf8-plain");
string const encoding = runparams.encoding->iconvName();
LYXERR(Debug::LATEX, "makeLaTeXFile encoding: " << encoding << ", fname=" << fname.realPath());
