Re: [LyX/master] Sort the language nesting mess with polyglossia

[Kornel Benko]

Am Donnerstag, 2. November 2017 um 21:38:31, schrieb Jean-Pierre 

> >> > Here are the errors that I get;
> >> >   LaTeX Error: \begin{otherlanguage} on input line 173 ended by 
> >> > \end{letter}.
> >> >   LaTeX Error: \begin{letter} on input line 173 ended by \end{document}.
> >> >
> >> > To trigger these, just load lib/templates/lettre.lyx, set 'use non-TeX 
> >> fonts' in
> >> > Document>Settings>Fonts and compile.
> >> >
> >>
> >> Looks like the same as in commit 1d0794e for  es/Additional.lyx
> >>
> >
> > Ping ...
> >
> > The error for export/templates/lettre_pdf5_systemF persist.
> > Should this be ignored?
> >
> Yes, the origin of this issue is quite involved because it is due to the 
> extra code which has been added in the preamble. The letter class is not 
> the culprit by itself.
> 
> I'm investigating this.
> 
> -- 
> Jean-Pierre

Am Donnerstag, 2. November 2017 um 21:59:54, schrieb Jürgen Spitzmüller 

> > The error for export/templates/lettre_pdf5_systemF persist.
> > Should this be ignored?
> 
> No, but this has nothing to do with LyX's language nesting, but is
> rather a consequence of lettre's overly complicated preamble hacking
> (\lettre, \findemessage etc.). I think one could re-write this layout
> file in a more clean manner using LyX's current layout means.
> 
> Jürgen

Thanks Jean-Pierre and Jürgen.

Kornel

signature.asc
Description: This is a digitally signed message part.

Re: [LyX/master] Sort the language nesting mess with polyglossia

[Jürgen Spitzmüller]

Am Donnerstag, den 02.11.2017, 21:33 +0100 schrieb Kornel Benko:
> > > It seems that the Xetex error that I get with the lettre template
> is of that 
> > > kind. I was about to write to the class author about this, but it
> seems thus 
> > > that is is a LyX bug, right?
> > > 
> > > Here are the errors that I get;
> > >   LaTeX Error: \begin{otherlanguage} on input line 173 ended by
> \end{letter}.
> > >   LaTeX Error: \begin{letter} on input line 173 ended by
> \end{document}.
> > > 
> > > To trigger these, just load lib/templates/lettre.lyx, set 'use
> non-TeX fonts' in 
> > > Document>Settings>Fonts and compile.
> > > 
> > 
> > Looks like the same as in commit 1d0794e for  es/Additional.lyx
> > 
> 
> Ping ...
> 
> The error for export/templates/lettre_pdf5_systemF persist.
> Should this be ignored?

No, but this has nothing to do with LyX's language nesting, but is
rather a consequence of lettre's overly complicated preamble hacking
(\lettre, \findemessage etc.). I think one could re-write this layout
file in a more clean manner using LyX's current layout means.

Jürgen

signature.asc
Description: This is a digitally signed message part

Re: [LyX/master] Sort the language nesting mess with polyglossia

[Kornel Benko]

Am Sonntag, 8. Oktober 2017 um 23:05:29, schrieb Kornel Benko 
> Am Sonntag, 8. Oktober 2017 um 22:59:47, schrieb Jean-Pierre Chrétien 
> 
> > Le 08/10/2017 à 05:39, Scott Kostyshak a écrit :
> > > On Sat, Sep 24, 2016 at 01:25:42AM +, Enrico Forestieri wrote:
> > >> commit 3bc08a76c42cd350a3141f00f37082bc9fab8967
> > >> Author: Enrico Forestieri 
> > >> Date:   Sat Sep 24 03:15:02 2016 +0200
> > >>
> > >>  Sort the language nesting mess with polyglossia
> > >>  
> > >>  When using polyglossia, lyx was making a real mess when changing
> > >>  language inside nested insets. The \begin{language} and
> > >>  \end{language} commands were not well paired such that they could
> > >>  easily occur just before and after the start or end of an
> > >>  environment. Of course this was causing latex errors such that
> > >>  "\begin{otherlanguage} ended by \end{environment}".
> > >>  There may still be some cases I did not take into account.
> > 
> > It seems that the Xetex error that I get with the lettre template is of 
> > that 
> > kind. I was about to write to the class author about this, but it seems 
> > thus 
> > that is is a LyX bug, right?
> > 
> > Here are the errors that I get;
> >   LaTeX Error: \begin{otherlanguage} on input line 173 ended by 
> > \end{letter}.
> >   LaTeX Error: \begin{letter} on input line 173 ended by \end{document}.
> > 
> > To trigger these, just load lib/templates/lettre.lyx, set 'use non-TeX 
> > fonts' in 
> > Document>Settings>Fonts and compile.
> > 
> 
> Looks like the same as in commit 1d0794e for  es/Additional.lyx
> 

Ping ...

The error for export/templates/lettre_pdf5_systemF persist.
Should this be ignored?

Kornel

signature.asc
Description: This is a digitally signed message part.

Re: [LyX/master] Sort the language nesting mess with polyglossia

[Jean-Pierre Chrétien]

Le 08/10/2017 à 23:42, Jean-Pierre Chrétien a écrit :

Le 08/10/2017 à 23:26, Scott Kostyshak a écrit :




I don't know. From what I remember lettre started failing for me after a
TeX Live update. If it were a LyX bug, I wonder why it worked before. My
memory often fails me though (or perhaps I fail it?). I think I have a
VM with an older TeX Live installation. Would it helped if I tested
whether the lettre template, as of current 2.3.x, compiles fine with
the older TeX Live installation?


I don't know, a new version of the class has been introduced in August with a 
lot of changes.


I think I will go on what I began to do here before contacting the author of the 
class: downsize to a minimal example using the examples indicated in the class 
manual to check if polyglossia simply works with lettre.cls 


Done, after introducing the same changes as in the LyX template after checking 
"Use non-TeX fonts" in Document>Settings>Fonts:


-\usepackage[T1]{fontenc}
-\usepackage[latin9]{inputenc}
+\usepackage{fontspec}
+\setmainfont[Mapping=tex-text]{FreeSans}

-\usepackage{babel}
+\usepackage{polyglossia}
+\setdefaultlanguage{french}
 \begin{document}

in a MWE latex file using the lettre class, xelatex compilation outputs the pdf 
output all right.


(the LyX template 
includes a lot of preamble code to make the order of the elements in the LyX 
window irrelevant, and the error happens inside an added custom command).


I will review now the extra preamble code.

--
Jean-Pierre

Re: [LyX/master] Sort the language nesting mess with polyglossia

[Scott Kostyshak]

On Wed, Oct 11, 2017 at 10:05:21PM +, Enrico Forestieri wrote:
> On Wed, Oct 11, 2017 at 10:56:59AM +0200, Kornel Benko wrote:
> > 
> > +1 to 2.3.x
> > No regressions found so far.
> 
> Done.

Thanks, Enrico.

Scott


signature.asc
Description: PGP signature

Re: [LyX/master] Sort the language nesting mess with polyglossia

[Enrico Forestieri]

On Wed, Oct 11, 2017 at 10:56:59AM +0200, Kornel Benko wrote:
> 
> +1 to 2.3.x
> No regressions found so far.

Done.

-- 
Enrico

Re: [LyX/master] Sort the language nesting mess with polyglossia

[Kornel Benko]

Am Mittwoch, 11. Oktober 2017 um 09:25:39, schrieb Enrico Forestieri 

> On Wed, Oct 11, 2017 at 02:05:32AM -0400, Scott Kostyshak wrote:
> > On Tue, Oct 10, 2017 at 08:07:05PM +, Enrico Forestieri wrote:
> > > On Sun, Oct 08, 2017 at 02:41:03PM -0400, Scott Kostyshak wrote:
> > > > On Sun, Oct 08, 2017 at 12:48:36PM +, Enrico Forestieri wrote:
> > > > 
> > > > > Thanks Scott. I'll have a look when I find some time. This kind of 
> > > > > bugs
> > > > > can be very time consuming.
> > > > 
> > > > Thanks, there is no rush and I don't consider this a blocker for 2.3.0.
> > > 
> > > Please, try the attached patch. This seems to be an issue related to
> > > argument insets only. However, I think the patch should be thoroughly
> > > tested to ensure that it introduces no regression.
> > 
> > The ctest results suggest the patch works well: the fr/Additional tests
> > now all pass, and there are no changes to the other test results.
> 
> I committed it to master at ce4f39d4.
> 

+1 to 2.3.x
No regressions found so far.

Kornel

signature.asc
Description: This is a digitally signed message part.

Re: [LyX/master] Sort the language nesting mess with polyglossia

[Enrico Forestieri]

On Wed, Oct 11, 2017 at 02:05:32AM -0400, Scott Kostyshak wrote:
> On Tue, Oct 10, 2017 at 08:07:05PM +, Enrico Forestieri wrote:
> > On Sun, Oct 08, 2017 at 02:41:03PM -0400, Scott Kostyshak wrote:
> > > On Sun, Oct 08, 2017 at 12:48:36PM +, Enrico Forestieri wrote:
> > > 
> > > > Thanks Scott. I'll have a look when I find some time. This kind of bugs
> > > > can be very time consuming.
> > > 
> > > Thanks, there is no rush and I don't consider this a blocker for 2.3.0.
> > 
> > Please, try the attached patch. This seems to be an issue related to
> > argument insets only. However, I think the patch should be thoroughly
> > tested to ensure that it introduces no regression.
> 
> The ctest results suggest the patch works well: the fr/Additional tests
> now all pass, and there are no changes to the other test results.

I committed it to master at ce4f39d4.

-- 
Enrico

Re: [LyX/master] Sort the language nesting mess with polyglossia

[Scott Kostyshak]

On Tue, Oct 10, 2017 at 08:07:05PM +, Enrico Forestieri wrote:
> On Sun, Oct 08, 2017 at 02:41:03PM -0400, Scott Kostyshak wrote:
> > On Sun, Oct 08, 2017 at 12:48:36PM +, Enrico Forestieri wrote:
> > 
> > > Thanks Scott. I'll have a look when I find some time. This kind of bugs
> > > can be very time consuming.
> > 
> > Thanks, there is no rush and I don't consider this a blocker for 2.3.0.
> 
> Please, try the attached patch. This seems to be an issue related to
> argument insets only. However, I think the patch should be thoroughly
> tested to ensure that it introduces no regression.

The ctest results suggest the patch works well: the fr/Additional tests
now all pass, and there are no changes to the other test results.

Scott


signature.asc
Description: PGP signature

Re: [LyX/master] Sort the language nesting mess with polyglossia

[Enrico Forestieri]

On Sun, Oct 08, 2017 at 02:41:03PM -0400, Scott Kostyshak wrote:
> On Sun, Oct 08, 2017 at 12:48:36PM +, Enrico Forestieri wrote:
> 
> > Thanks Scott. I'll have a look when I find some time. This kind of bugs
> > can be very time consuming.
> 
> Thanks, there is no rush and I don't consider this a blocker for 2.3.0.

Please, try the attached patch. This seems to be an issue related to
argument insets only. However, I think the patch should be thoroughly
tested to ensure that it introduces no regression.

-- 
Enrico
diff --git a/src/output_latex.cpp b/src/output_latex.cpp
index e0163ad2fa..02667aae9f 100644
--- a/src/output_latex.cpp
+++ b/src/output_latex.cpp
@@ -882,7 +882,7 @@ void TeXOnePar(Buffer const & buf,
}
// With CJK, the CJK tag has to be closed first (see 
below)
if (runparams.encoding->package() != Encoding::CJK
-   && par_lang != openLanguageName(state)
+   && (par_lang != openLanguageName(state) || 
localswitch)
&& !par_lang.empty()) {
string bc = use_polyglossia ?
  
getPolyglossiaBegin(lang_begin_command, par_lang, 
par_language->polyglossiaOpts())

Re: [LyX/master] Sort the language nesting mess with polyglossia

[Jean-Pierre Chrétien]

Le 09/10/2017 à 09:54, Guenter Milde a écrit :


The same as with babel vs. polyglossia: a wrong nesting tolerated in a
previous version but problematic in the new one. Could be tested on the
LaTeX source level...


As far as lettre.lyx is concerned, thats's what I'm planning to do.

--
Jean-Pierre

Re: [LyX/master] Sort the language nesting mess with polyglossia

[Guenter Milde]

On 2017-10-08, Scott Kostyshak wrote:
> On Sun, Oct 08, 2017 at 08:59:47PM +, Jean-Pierre Chrétien wrote:
>> Le 08/10/2017 à 05:39, Scott Kostyshak a écrit :

>> > >  Sort the language nesting mess with polyglossia
>> > >  When using polyglossia, lyx was making a real mess when changing
>> > >  language inside nested insets. The \begin{language} and
>> > >  \end{language} commands were not well paired such that they could
>> > >  easily occur just before and after the start or end of an
>> > >  environment. Of course this was causing latex errors such that
>> > >  "\begin{otherlanguage} ended by \end{environment}".
>> > >  There may still be some cases I did not take into account.

If I remember right, the output for babel and polyglossia was the same but
babel is more tolerant towards mix up of nesting.

>> It seems that the Xetex error that I get with the lettre template is
>> of that kind. I was about to write to the class author about this, but
>> it seems thus that is is a LyX bug, right?

> I don't know. From what I remember lettre started failing for me after a
> TeX Live update. If it were a LyX bug, I wonder why it worked before.

The same as with babel vs. polyglossia: a wrong nesting tolerated in a
previous version but problematic in the new one. Could be tested on the
LaTeX source level...

Günter

Re: [LyX/master] Sort the language nesting mess with polyglossia

[Jean-Pierre Chrétien]

Le 08/10/2017 à 23:26, Scott Kostyshak a écrit :

On Sun, Oct 08, 2017 at 08:59:47PM +, Jean-Pierre Chrétien wrote:

Le 08/10/2017 à 05:39, Scott Kostyshak a écrit :

On Sat, Sep 24, 2016 at 01:25:42AM +, Enrico Forestieri wrote:

commit 3bc08a76c42cd350a3141f00f37082bc9fab8967
Author: Enrico Forestieri 
Date:   Sat Sep 24 03:15:02 2016 +0200

  Sort the language nesting mess with polyglossia
  When using polyglossia, lyx was making a real mess when changing
  language inside nested insets. The \begin{language} and
  \end{language} commands were not well paired such that they could
  easily occur just before and after the start or end of an
  environment. Of course this was causing latex errors such that
  "\begin{otherlanguage} ended by \end{environment}".
  There may still be some cases I did not take into account.


It seems that the Xetex error that I get with the lettre template is of that
kind. I was about to write to the class author about this, but it seems thus
that is is a LyX bug, right?


I don't know. From what I remember lettre started failing for me after a
TeX Live update. If it were a LyX bug, I wonder why it worked before. My
memory often fails me though (or perhaps I fail it?). I think I have a
VM with an older TeX Live installation. Would it helped if I tested
whether the lettre template, as of current 2.3.x, compiles fine with
the older TeX Live installation?


I don't know, a new version of the class has been introduced in August with a 
lot of changes.


I think I will go on what I began to do here before contacting the author of the 
class: downsize to a minimal example using the examples indicated in the class 
manual to check if polyglossia simply works with lettre.cls (the LyX template 
includes a lot of preamble code to make the order of the elements in the LyX 
window irrelevant, and the error happens inside an added custom command). Last 
time I worked on this I was fighting with font encoding when I copy latex code 
from the lettre pdf manual.


--
Jean-Pierre

Re: [LyX/master] Sort the language nesting mess with polyglossia

[Scott Kostyshak]

On Sun, Oct 08, 2017 at 08:59:47PM +, Jean-Pierre Chrétien wrote:
> Le 08/10/2017 à 05:39, Scott Kostyshak a écrit :
> > On Sat, Sep 24, 2016 at 01:25:42AM +, Enrico Forestieri wrote:
> > > commit 3bc08a76c42cd350a3141f00f37082bc9fab8967
> > > Author: Enrico Forestieri 
> > > Date:   Sat Sep 24 03:15:02 2016 +0200
> > > 
> > >  Sort the language nesting mess with polyglossia
> > >  When using polyglossia, lyx was making a real mess when changing
> > >  language inside nested insets. The \begin{language} and
> > >  \end{language} commands were not well paired such that they could
> > >  easily occur just before and after the start or end of an
> > >  environment. Of course this was causing latex errors such that
> > >  "\begin{otherlanguage} ended by \end{environment}".
> > >  There may still be some cases I did not take into account.
> 
> It seems that the Xetex error that I get with the lettre template is of that
> kind. I was about to write to the class author about this, but it seems thus
> that is is a LyX bug, right?

I don't know. From what I remember lettre started failing for me after a
TeX Live update. If it were a LyX bug, I wonder why it worked before. My
memory often fails me though (or perhaps I fail it?). I think I have a
VM with an older TeX Live installation. Would it helped if I tested
whether the lettre template, as of current 2.3.x, compiles fine with
the older TeX Live installation?

Thanks for taking a look,

Scott


signature.asc
Description: PGP signature

Re: [LyX/master] Sort the language nesting mess with polyglossia

[Kornel Benko]

Am Sonntag, 8. Oktober 2017 um 22:59:47, schrieb Jean-Pierre Chrétien 

> Le 08/10/2017 à 05:39, Scott Kostyshak a écrit :
> > On Sat, Sep 24, 2016 at 01:25:42AM +, Enrico Forestieri wrote:
> >> commit 3bc08a76c42cd350a3141f00f37082bc9fab8967
> >> Author: Enrico Forestieri 
> >> Date:   Sat Sep 24 03:15:02 2016 +0200
> >>
> >>  Sort the language nesting mess with polyglossia
> >>  
> >>  When using polyglossia, lyx was making a real mess when changing
> >>  language inside nested insets. The \begin{language} and
> >>  \end{language} commands were not well paired such that they could
> >>  easily occur just before and after the start or end of an
> >>  environment. Of course this was causing latex errors such that
> >>  "\begin{otherlanguage} ended by \end{environment}".
> >>  There may still be some cases I did not take into account.
> 
> It seems that the Xetex error that I get with the lettre template is of that 
> kind. I was about to write to the class author about this, but it seems thus 
> that is is a LyX bug, right?
> 
> Here are the errors that I get;
>   LaTeX Error: \begin{otherlanguage} on input line 173 ended by \end{letter}.
>   LaTeX Error: \begin{letter} on input line 173 ended by \end{document}.
> 
> To trigger these, just load lib/templates/lettre.lyx, set 'use non-TeX fonts' 
> in 
> Document>Settings>Fonts and compile.
> 

Looks like the same as in commit 1d0794e for  es/Additional.lyx

Kornel

signature.asc
Description: This is a digitally signed message part.

Re: [LyX/master] Sort the language nesting mess with polyglossia

[Jean-Pierre Chrétien]

Le 08/10/2017 à 05:39, Scott Kostyshak a écrit :

On Sat, Sep 24, 2016 at 01:25:42AM +, Enrico Forestieri wrote:

commit 3bc08a76c42cd350a3141f00f37082bc9fab8967
Author: Enrico Forestieri 
Date:   Sat Sep 24 03:15:02 2016 +0200

 Sort the language nesting mess with polyglossia
 
 When using polyglossia, lyx was making a real mess when changing

 language inside nested insets. The \begin{language} and
 \end{language} commands were not well paired such that they could
 easily occur just before and after the start or end of an
 environment. Of course this was causing latex errors such that
 "\begin{otherlanguage} ended by \end{environment}".
 There may still be some cases I did not take into account.


It seems that the Xetex error that I get with the lettre template is of that 
kind. I was about to write to the class author about this, but it seems thus 
that is is a LyX bug, right?


Here are the errors that I get;
 LaTeX Error: \begin{otherlanguage} on input line 173 ended by \end{letter}.
 LaTeX Error: \begin{letter} on input line 173 ended by \end{document}.

To trigger these, just load lib/templates/lettre.lyx, set 'use non-TeX fonts' in 
Document>Settings>Fonts and compile.


--
Jean-Pierre

Re: [LyX/master] Sort the language nesting mess with polyglossia

[Jean-Pierre Chrétien]

Le 08/10/2017 à 05:39, Scott Kostyshak a écrit :



Attached also is the XeTeX code that LyX writes for this file. The
problem is here:

\begin{english}%
\item[\(\star\)}]  \textfrench{Agissez sur chaque entrée...}
\end{english}%

 From what I can tell, the problem is the curly brace after "(\star\)".

Current 2.3.x produces a different LaTeX file (I attach this .tex file
also). There are two notable differences compared to the .tex file
created from this commit. For one, the package xunicode is no longer
loaded in current 2.3.x. And second, there is no longer a
"\end{english}", which causes an error even if the curly brace
referenced above is removed.

I created the attached .lyx MWE from the 2.3.x branch fr/Additional.lyx.
ctests started failing recently on that file (probably after we copied
some new English into it to be translated?). I don't know how that part
of fr/Additional.lyx was created. Perhaps the problem is in the the
creation of the file, not the parsing?


It's me who copied this piece of code from original English Additional.lyx file.
If I look at the current version of fr/Additional.lyx in master I see this:


\selectlanguage{english}%
\item[\foreignlanguage{english}{\(\star\)}]  \foreignlanguage{french}{Agissez 
sur chaque entrée individuellement

en écrivant le motif de la puce }in a ``Custom Item'' inset (available
at \textsf{Insert\lyxarrow Custom Item})\foreignlanguage{french}{


Could that unclosed
\selectlanguage{english}%
command which could be responsible of the extra }?
--
Jean-Pierre

Re: [LyX/master] Sort the language nesting mess with polyglossia

[Scott Kostyshak]

On Sun, Oct 08, 2017 at 12:48:36PM +, Enrico Forestieri wrote:

> Thanks Scott. I'll have a look when I find some time. This kind of bugs
> can be very time consuming.

Thanks, there is no rush and I don't consider this a blocker for 2.3.0.

Scott


signature.asc
Description: PGP signature

Re: [LyX/master] Sort the language nesting mess with polyglossia

[Enrico Forestieri]

On Sat, Oct 07, 2017 at 11:39:39PM -0400, Scott Kostyshak wrote:

> On Sat, Sep 24, 2016 at 01:25:42AM +, Enrico Forestieri wrote:
> > commit 3bc08a76c42cd350a3141f00f37082bc9fab8967
> > Author: Enrico Forestieri 
> > Date:   Sat Sep 24 03:15:02 2016 +0200
> > 
> > Sort the language nesting mess with polyglossia
> > 
> > When using polyglossia, lyx was making a real mess when changing
> > language inside nested insets. The \begin{language} and
> > \end{language} commands were not well paired such that they could
> > easily occur just before and after the start or end of an
> > environment. Of course this was causing latex errors such that
> > "\begin{otherlanguage} ended by \end{environment}".
> > There may still be some cases I did not take into account.
> 
> A git bisect suggests this caused a change in behavior. The attached
> .lyx file fails to compile to XeTeX or LuaTeX starting with this commit.
> 
> Attached also is the XeTeX code that LyX writes for this file. The
> problem is here:
> 
> \begin{english}%
> \item[\(\star\)}]  \textfrench{Agissez sur chaque entrée...}
> \end{english}%
> 
> From what I can tell, the problem is the curly brace after "(\star\)".
> 
> Current 2.3.x produces a different LaTeX file (I attach this .tex file
> also). There are two notable differences compared to the .tex file
> created from this commit. For one, the package xunicode is no longer
> loaded in current 2.3.x. And second, there is no longer a
> "\end{english}", which causes an error even if the curly brace
> referenced above is removed.
> 
> I created the attached .lyx MWE from the 2.3.x branch fr/Additional.lyx.
> ctests started failing recently on that file (probably after we copied
> some new English into it to be translated?). I don't know how that part
> of fr/Additional.lyx was created. Perhaps the problem is in the the
> creation of the file, not the parsing?
> 
> Enrico, thanks a lot for tackling these language nesting issues. The
> only thing I understand from this code is that it is very complex and
> fragile. Let me know if you want me to run the ctests on a potential
> patch.
> 
> I'm open to whatever you suggest for 2.3.0. If this code is so fragile
> that you suggest we leave it as it is, that's fine with me. 2.3.x is
> already in a much better state as far as language nesting, compared to
> 2.2.0, thanks to your work.

Thanks Scott. I'll have a look when I find some time. This kind of bugs
can be very time consuming.

-- 
Enrico

Re: [LyX/master] Sort the language nesting mess with polyglossia

[Scott Kostyshak]

On Sat, Sep 24, 2016 at 01:25:42AM +, Enrico Forestieri wrote:
> commit 3bc08a76c42cd350a3141f00f37082bc9fab8967
> Author: Enrico Forestieri 
> Date:   Sat Sep 24 03:15:02 2016 +0200
> 
> Sort the language nesting mess with polyglossia
> 
> When using polyglossia, lyx was making a real mess when changing
> language inside nested insets. The \begin{language} and
> \end{language} commands were not well paired such that they could
> easily occur just before and after the start or end of an
> environment. Of course this was causing latex errors such that
> "\begin{otherlanguage} ended by \end{environment}".
> There may still be some cases I did not take into account.

A git bisect suggests this caused a change in behavior. The attached
.lyx file fails to compile to XeTeX or LuaTeX starting with this commit.

Attached also is the XeTeX code that LyX writes for this file. The
problem is here:

\begin{english}%
\item[\(\star\)}]  \textfrench{Agissez sur chaque entrée...}
\end{english}%

From what I can tell, the problem is the curly brace after "(\star\)".

Current 2.3.x produces a different LaTeX file (I attach this .tex file
also). There are two notable differences compared to the .tex file
created from this commit. For one, the package xunicode is no longer
loaded in current 2.3.x. And second, there is no longer a
"\end{english}", which causes an error even if the curly brace
referenced above is removed.

I created the attached .lyx MWE from the 2.3.x branch fr/Additional.lyx.
ctests started failing recently on that file (probably after we copied
some new English into it to be translated?). I don't know how that part
of fr/Additional.lyx was created. Perhaps the problem is in the the
creation of the file, not the parsing?

Enrico, thanks a lot for tackling these language nesting issues. The
only thing I understand from this code is that it is very complex and
fragile. Let me know if you want me to run the ctests on a potential
patch.

I'm open to whatever you suggest for 2.3.0. If this code is so fragile
that you suggest we leave it as it is, that's fine with me. 2.3.x is
already in a much better state as far as language nesting, compared to
2.2.0, thanks to your work.

Scott


switching_mwe_bad.21.lyx
Description: application/lyx
%% LyX 2.3.0dev created this file.  For more info, see http://www.lyx.org/.
%% Do not edit unless you really know what you are doing.
\documentclass[english,french]{article}
\usepackage{fontspec}
\setmainfont[Mapping=tex-text]{FreeSans}
\setsansfont[Mapping=tex-text]{FreeSans}
\setmonofont{FreeSans}
\usepackage{polyglossia}
\setdefaultlanguage{french}
\setotherlanguage{english}
\begin{document}
\begin{itemize}
\item abc
\begin{english}[variant=american]%
\item[\(\star\)}]  \textfrench{Agissez sur chaque entrée...}
\end{itemize}

\end{document}
%% LyX 2.3.0dev created this file.  For more info, see http://www.lyx.org/.
%% Do not edit unless you really know what you are doing.
\documentclass[english,french]{article}
\usepackage{fontspec}
\setmainfont[Mapping=tex-text]{FreeSans}
\setsansfont[Mapping=tex-text]{FreeSans}
\setmonofont{FreeSans}
\usepackage{xunicode}
\usepackage{polyglossia}
\setdefaultlanguage{french}
\setotherlanguage{english}
\begin{document}
\begin{itemize}
\item abc
\begin{english}%
\item[\(\star\)}]  \textfrench{Agissez sur chaque entrée...}
\end{english}%
\end{itemize}

\end{document}


signature.asc
Description: PGP signature

Re: [LyX/master] Sort the language nesting mess with polyglossia

[Richard Heck]

On 05/29/2017 05:46 PM, Scott Kostyshak wrote:
> On Mon, May 29, 2017 at 11:15:01PM +0200, Enrico Forestieri wrote:
>
>> However, given the semantics of \vspace, that change should not have an
>> effect. Indeed, when used within a paragraph, the vertical space is added
>> after the line in which the \vspace appears. So, for example, if you have
>> the following snippet:
>>
>> Here is a vspace\vspace{1cm} after this line. How can be seen it only
>> appears after the first line of this paragraph.
>>
>> Supposing that the first line ends at the word "appears", the output would be
>>
>> Here is a vspace after this line. How can be seen it only appears
>>  ↕ 1cm
>> after the first line of this paragraph.
>>
>> Thus, given that after that \vspace the line ends, one can think that 
>> a following blank line is irrelevant, but apparently it is not so.
>> Moreover, the subsequent \vspace is not followed by a blank line
>> (and this is not due to the above hunk).
> I had always wondered how \vspace in paragraphs worked. Now I understand
> and I also understand how LyX cannot display them exactly like the PDF
> output because LyX does not know where the line break will be.
>
>> Anyway, neither the change is desired, nor I can explain how that hunk
>> slipped in, so I am going to revert it.
> Thanks for the quick fix.
>
> By the way, if anyone is curious how I noticed that difference in
> spacing, it is thanks to the open-source "diffpdf" tool.
>
> I compared the PDF that stable LyX produced for me of my paper with the
> PDF that master LyX produces, with the following command:
>
> diffpdf mwe_before.pdf mwe_after.pdf
>
> Setting compare to "Appearance" leads to the attached screenshot:
>
> I find diffpdf to be a valuable tool when testing.

I can see why!!

Richard

Re: [LyX/master] Sort the language nesting mess with polyglossia

[Richard Heck]

On 05/29/2017 05:29 PM, Enrico Forestieri wrote:
> On Mon, May 29, 2017 at 11:15:01PM +0200, Enrico Forestieri wrote:
>> Anyway, neither the change is desired, nor I can explain how that hunk
>> slipped in, so I am going to revert it.
> Reverted at 90c423a6.
>
> Richard, I fear this slipped in also in stable, so you may want to
> revert it there, too.

I've done so.

Richard

Re: [LyX/master] Sort the language nesting mess with polyglossia

[Scott Kostyshak]

On Mon, May 29, 2017 at 11:15:01PM +0200, Enrico Forestieri wrote:

> However, given the semantics of \vspace, that change should not have an
> effect. Indeed, when used within a paragraph, the vertical space is added
> after the line in which the \vspace appears. So, for example, if you have
> the following snippet:
> 
> Here is a vspace\vspace{1cm} after this line. How can be seen it only
> appears after the first line of this paragraph.
> 
> Supposing that the first line ends at the word "appears", the output would be
> 
> Here is a vspace after this line. How can be seen it only appears
>  ↕ 1cm
> after the first line of this paragraph.
> 
> Thus, given that after that \vspace the line ends, one can think that 
> a following blank line is irrelevant, but apparently it is not so.
> Moreover, the subsequent \vspace is not followed by a blank line
> (and this is not due to the above hunk).

I had always wondered how \vspace in paragraphs worked. Now I understand
and I also understand how LyX cannot display them exactly like the PDF
output because LyX does not know where the line break will be.

> Anyway, neither the change is desired, nor I can explain how that hunk
> slipped in, so I am going to revert it.

Thanks for the quick fix.

By the way, if anyone is curious how I noticed that difference in
spacing, it is thanks to the open-source "diffpdf" tool.

I compared the PDF that stable LyX produced for me of my paper with the
PDF that master LyX produces, with the following command:

diffpdf mwe_before.pdf mwe_after.pdf

Setting compare to "Appearance" leads to the attached screenshot:

I find diffpdf to be a valuable tool when testing.

Scott


signature.asc
Description: PGP signature

Re: [LyX/master] Sort the language nesting mess with polyglossia

[Enrico Forestieri]

On Mon, May 29, 2017 at 11:15:01PM +0200, Enrico Forestieri wrote:
> 
> Anyway, neither the change is desired, nor I can explain how that hunk
> slipped in, so I am going to revert it.

Reverted at 90c423a6.

Richard, I fear this slipped in also in stable, so you may want to
revert it there, too.

-- 
Enrico

Re: [LyX/master] Sort the language nesting mess with polyglossia

[Enrico Forestieri]

On Mon, May 29, 2017 at 12:23:29PM -0400, Scott Kostyshak wrote:
> 
> I think (if I did git bisect correctly) this commit caused a change in
> LaTeX and PDF output that I would like to make sure is expected.
> 
> See the attached mwe.lyx file (in 2.2.x format). mwe_before.tex and
> mwe_before.pdf are the outputs before this commit, and mwe_after.tex and
> mwe_after.pdf are the outputs after this commit.
> 
> Note the differences in vertical spacing around "point2".
> 
> The only difference in the .tex files is that after this commit there is
> no empty line after the \vspace command.
> 
> Is this change desired?

At first, it seemed strange to me that a commit dealing with language
nesting could have such an effect on pretty unrelated code. But, going
through the commit, I discovered the following hunk:

> @@ -1000,7 +1103,7 @@ void TeXOnePar(Buffer const & buf,
> // prevent unwanted whitespace
> os << '%';
> if (!os.afterParbreak() && !last_was_separator)
> -   os << '\n';
> +   os << breakln;
> }
> 
> // if this is a CJK-paragraph and the next isn't, close CJK
> 

which is responsible for the change. I really don't remember why I made
that change, which seems completely unrelated.

However, given the semantics of \vspace, that change should not have an
effect. Indeed, when used within a paragraph, the vertical space is added
after the line in which the \vspace appears. So, for example, if you have
the following snippet:

Here is a vspace\vspace{1cm} after this line. How can be seen it only
appears after the first line of this paragraph.

Supposing that the first line ends at the word "appears", the output would be

Here is a vspace after this line. How can be seen it only appears
 ↕ 1cm
after the first line of this paragraph.

Thus, given that after that \vspace the line ends, one can think that 
a following blank line is irrelevant, but apparently it is not so.
Moreover, the subsequent \vspace is not followed by a blank line
(and this is not due to the above hunk).

Anyway, neither the change is desired, nor I can explain how that hunk
slipped in, so I am going to revert it.

-- 
Enrico

Re: [LyX/master] Sort the language nesting mess with polyglossia

[Scott Kostyshak]

On Sat, Sep 24, 2016 at 03:25:42AM +0200, Enrico Forestieri wrote:
> commit 3bc08a76c42cd350a3141f00f37082bc9fab8967
> Author: Enrico Forestieri 
> Date:   Sat Sep 24 03:15:02 2016 +0200
> 
> Sort the language nesting mess with polyglossia
> 
> When using polyglossia, lyx was making a real mess when changing
> language inside nested insets. The \begin{language} and
> \end{language} commands were not well paired such that they could
> easily occur just before and after the start or end of an
> environment. Of course this was causing latex errors such that
> "\begin{otherlanguage} ended by \end{environment}".
> There may still be some cases I did not take into account.

I think (if I did git bisect correctly) this commit caused a change in
LaTeX and PDF output that I would like to make sure is expected.

See the attached mwe.lyx file (in 2.2.x format). mwe_before.tex and
mwe_before.pdf are the outputs before this commit, and mwe_after.tex and
mwe_after.pdf are the outputs after this commit.

Note the differences in vertical spacing around "point2".

The only difference in the .tex files is that after this commit there is
no empty line after the \vspace command.

Is this change desired?

Scott
#LyX 2.2 created this file. For more info see http://www.lyx.org/
\lyxformat 508
\begin_document
\begin_header
\save_transient_properties true
\origin unavailable
\textclass beamer
\use_default_options true
\maintain_unincluded_children false
\language english
\language_package default
\inputencoding auto
\fontencoding global
\font_roman "default" "default"
\font_sans "default" "default"
\font_typewriter "default" "default"
\font_math "auto" "auto"
\font_default_family default
\use_non_tex_fonts false
\font_sc false
\font_osf false
\font_sf_scale 100 100
\font_tt_scale 100 100
\graphics default
\default_output_format default
\output_sync 0
\bibtex_command default
\index_command default
\paperfontsize default
\spacing single
\use_hyperref false
\papersize default
\use_geometry true
\use_package amsmath 1
\use_package amssymb 1
\use_package cancel 1
\use_package esint 1
\use_package mathdots 1
\use_package mathtools 1
\use_package mhchem 1
\use_package stackrel 1
\use_package stmaryrd 1
\use_package undertilde 1
\cite_engine basic
\cite_engine_type default
\biblio_style plain
\use_bibtopic false
\use_indices false
\paperorientation portrait
\suppress_date false
\justification true
\use_refstyle 1
\index Index
\shortcut idx
\color #008000
\end_index
\secnumdepth 3
\tocdepth 3
\paragraph_separation indent
\paragraph_indentation default
\quotes_language english
\papercolumns 1
\papersides 1
\paperpagestyle default
\tracking_changes false
\output_changes false
\html_math_output 0
\html_css_as_file 0
\html_be_strict false
\end_header

\begin_body

\begin_layout Frame
\begin_inset Argument 4
status open

\begin_layout Plain Layout
One more extension
\end_layout

\end_inset


\begin_inset Separator latexpar
\end_inset


\end_layout

\begin_deeper
\begin_layout Standard
A common theme in this presentation has been about how OLS is not good at...
 I propose...:
\begin_inset VSpace 0.5cm
\end_inset


\end_layout

\begin_layout Quote
point 1
\end_layout

\begin_layout Standard
\begin_inset VSpace 0.5cm
\end_inset

point2:
\end_layout

\begin_layout Enumerate
point2.1
\end_layout

\begin_layout Enumerate
point2.2
\end_layout

\end_deeper
\end_body
\end_document


mwe_after.pdf
Description: Adobe PDF document


mwe_after.tex
Description: TeX document


mwe_before.pdf
Description: Adobe PDF document


mwe_before.tex
Description: TeX document


signature.asc
Description: PGP signature

Re: [LyX/master] Sort the language nesting mess with polyglossia

[Enrico Forestieri]

On Tue, Sep 27, 2016 at 01:02:04PM +0200, Guillaume Munch wrote:
> Le 24/09/2016 à 03:25, Enrico Forestieri a écrit :
> >commit 3bc08a76c42cd350a3141f00f37082bc9fab8967
> >Author: Enrico Forestieri 
> >Date:   Sat Sep 24 03:15:02 2016 +0200
> >
> >Sort the language nesting mess with polyglossia
> >
> >When using polyglossia, lyx was making a real mess when changing
> >language inside nested insets. The \begin{language} and
> >\end{language} commands were not well paired such that they could
> >easily occur just before and after the start or end of an
> >environment. Of course this was causing latex errors such that
> >"\begin{otherlanguage} ended by \end{environment}".
> >There may still be some cases I did not take into account.
> 
> Hi, this commit introduced the following new warning in master with the
> default gcc configuration:
> 
> ../../src/output_latex.cpp: In function ‘void lyx::TeXOnePar(const
> lyx::Buffer&, const lyx::Text&, lyx::pit_type, lyx::otexstream&, const
> lyx::OutputParams&, const string&, int, int)’:
> ../../src/output_latex.cpp:986:7: warning: suggest parentheses around ‘&&’
> within ‘||’ [-Wparentheses]
>&& nextpar->getDepth() == par.getDepth()
>^

Thanks. Fixed at f476d9c8.

-- 
Enrico

Re: [LyX/master] Sort the language nesting mess with polyglossia

[Guillaume Munch]

Le 24/09/2016 à 03:25, Enrico Forestieri a écrit :

commit 3bc08a76c42cd350a3141f00f37082bc9fab8967
Author: Enrico Forestieri 
Date:   Sat Sep 24 03:15:02 2016 +0200

Sort the language nesting mess with polyglossia

When using polyglossia, lyx was making a real mess when changing
language inside nested insets. The \begin{language} and
\end{language} commands were not well paired such that they could
easily occur just before and after the start or end of an
environment. Of course this was causing latex errors such that
"\begin{otherlanguage} ended by \end{environment}".
There may still be some cases I did not take into account.


Hi, this commit introduced the following new warning in master with the 
default gcc configuration:


../../src/output_latex.cpp: In function ‘void lyx::TeXOnePar(const 
lyx::Buffer&, const lyx::Text&, lyx::pit_type, lyx::otexstream&, const 
lyx::OutputParams&, const string&, int, int)’:
../../src/output_latex.cpp:986:7: warning: suggest parentheses around 
‘&&’ within ‘||’ [-Wparentheses]

   && nextpar->getDepth() == par.getDepth()
   ^