Re: [LyX/master] Fix export with polyglossia and uninvert a manual

[Guenter Milde]

On 2015-10-29, Guenter Milde wrote:

> -> We need a framework to handle such "generic" problems like "Xe/LuaTeX
>with TeX fonts".

>Suggestion:

>* Code changes/commits that break tests with the default export method of
>  important documents require immediate action.

>* Problems with other export routes can be handled later (but should not
>  be forgotten).

   -> open a bug report.

Günter

Re: [LyX/master] Fix export with polyglossia and uninvert a manual

[Guenter Milde]

On 2015-10-28, Scott Kostyshak wrote:
> On Mon, Oct 26, 2015 at 10:59:05AM +, Guenter Milde wrote:
>> On 2015-10-26, Scott Kostyshak wrote:
>> > On Sun, Oct 25, 2015 at 11:17:15AM +0100, Georg Baum wrote:
>> >> Scott Kostyshak wrote:

...

>> >> If some features of a document are implemented in a babel specific way
>> >> (even if the feature would in theory not need babel), then we should
>> >> rather declare that explicitly like in 78c60de9bb IMHO, than to
>> >> pretend that polyglossia can be used, but silently change the output.

>> > Ah this is a good point. Dang it. In this case I think it makes even
>> > more sense now to remove the babel-specific preamble code, which there
>> > seems to be support for from others as well. 

>> In this case, I would rather not remove the code but change it to be usable
>> without babel:

>> * if there is only one language in the document (or if the autrefname can be
>>   the same in all used languages), do the (r)definitions directly:

>> * if there is more than one language and autoreferencenames must be language
>>   speecific, define them for both, babel and polyglossia.


>> > I would prefer to leave the babel conditional in because of my reason
>> > above, until we come to a decision and action on whether the
>> > babel-specific code should be removed (e.g. fixing #9817?). That said, I
>> > do understand that two wrongs doesn't make a right so let me know if you
>> > believe strongly that I should revert.

I believe, the babel conditional should only stay, if the file with this
conditional not only compiles but also the result is OK.

* Math.lyx uses packages that either fail to compile or produce wrong output
  with XeTeX/LuaTeX (with both, TeX fonts and non-TeX fonts) in any language
  (e.g. mhchem)
  Also, some accents (e.g. \t) in "mathematical text" are missing in the
  output.
  
  -> invert all */Math.lyx tests with XeTeX and LuaTeX.
  -> don't add partial fixes (they may be outdated once the real problem is
 solved).
  -> consider a safeguard in LyX against using Xe/LuaTeX with non-compatible
 packages.
  
* fr/UserGuide.lyx compiles but has wrong output for LuaTeX with TeX fonts:
  Eg. the ToC is called "Table des matiÃĺres" (should be "Table des matières").
  This is a bug in Babel (or frenchb.ldf), where detecting
  XeTeX/LuaTeX assumes that non-TeX fonts are used and the encoding is utf8. 
  (It affects other Babel languages as well.)

  -> invert all doc/fr/*.lyx tests with XeTeX/LuaTeX and TeX fonts.
  -> don't add partial fixes.
  -> consider a safeguard in LyX against using Xe/LuaTeX with non-compatible
 packages.

* es/Customization.lyx
  Same problem with LuaTeX and TeX fonts: ToC becomes "ÃŊndice general".

  -> invert all doc/es/*.lyx tests with XeTeX/LuaTeX and TeX fonts.
  ...
  
>> I believe it would be better to suspend testing the problematic files
>> with "non-TeX fonts" until the issues are solved.

> Ideally code changes would not be committed before broken tests are
> fixed or the regressions caused by the commits are fixed. That should
> all be done in one commit.

We cannot expect this for cases where there are several reasons for failure
affecting a large number of documents. (Just consider the size of a single
commit fixing all cases of "missing character".)

-> We need a framework to handle such "generic" problems like "Xe/LuaTeX
   with TeX fonts".

   Suggestion:

   * Code changes/commits that break tests with the default export method of
 important documents require immediate action.
 
   * Problems with other export routes can be handled later (but should not
 be forgotten).
 
Also, not every developer should be expected to run the export tests for
every small commit - however, there should be a commitment to fix
problems introduced by ones changes.


>> Fixing problems in the Manuals found by automatic testing is a good thing.
>> Modifying the Manuals "somehow" just to please automatic testing is not.

> I agree. I think that in most cases fixing the manuals to compile to
> multiple formats improves the documents and forces us to have clean
> preamble code.

In this case, the preamble code got more complicated, not cleaner.
Also, it takes time to agree on a best praxis.


For the test with non-TeX fonts, the preamble code involving lmodern.sty
must be changed, otherwise 8-bit fonts are used with wrong font encoding!

One possibility would be to remove the special casing altogether,
possibly selecting a better font in the GUI (but this was rejected some
years ago).

Another option is the following patch (similar in all places where this
preamble code is used):

--- a/lib/doc/fr/UserGuide.lyx
+++ b/lib/doc/fr/UserGuide.lyx
@@ -15,11 +15,11 @@
 % email: lyx-d...@lists.lyx.org
 
 \usepackage{ifpdf} % part of the hyperref bundle
-\ifpdf % if pdflatex is used
-
+\ifpdf % if pdflatex or xelatex or lualatex is used
+\@ifpackageloaded{fontspec}{}{% non-tex-fonts default to LModern
  % set fonts for 

Re: [LyX/master] Fix export with polyglossia and uninvert a manual

[Scott Kostyshak]

On Thu, Oct 29, 2015 at 10:42:20PM -0400, Scott Kostyshak wrote:
> On Thu, Oct 29, 2015 at 01:38:29PM +, Guenter Milde wrote:
> > On 2015-10-29, Guenter Milde wrote:
> > 
> > > -> We need a framework to handle such "generic" problems like "Xe/LuaTeX
> > >with TeX fonts".
> > 
> > >Suggestion:
> > 
> > >* Code changes/commits that break tests with the default export method 
> > > of
> > >  important documents require immediate action.
> > 
> > >* Problems with other export routes can be handled later (but should 
> > > not
> > >  be forgotten).
> > 
> >-> open a bug report.
> 
> Seems reasonable but I would emphasize that the above is with regard to
> breaking other export routes *for that document*. Often if a document's
> export for a non-default route is broken it is because a bug was
> introduced in LyX and not because a document-specific change was made
> that is not supported by the route. This is why I think there is a high
> signal/noise ratio in the export tests for the non-default formats. The
> most annoying part is setting them correctly to begin with.

I added this explanation to the test documentation at 1a4d5b3c. Please
edit it accordingly.

Scott

Re: [LyX/master] Fix export with polyglossia and uninvert a manual

[Scott Kostyshak]

On Thu, Oct 29, 2015 at 01:38:29PM +, Guenter Milde wrote:
> On 2015-10-29, Guenter Milde wrote:
> 
> > -> We need a framework to handle such "generic" problems like "Xe/LuaTeX
> >with TeX fonts".
> 
> >Suggestion:
> 
> >* Code changes/commits that break tests with the default export method of
> >  important documents require immediate action.
> 
> >* Problems with other export routes can be handled later (but should not
> >  be forgotten).
> 
>-> open a bug report.

Seems reasonable but I would emphasize that the above is with regard to
breaking other export routes *for that document*. Often if a document's
export for a non-default route is broken it is because a bug was
introduced in LyX and not because a document-specific change was made
that is not supported by the route. This is why I think there is a high
signal/noise ratio in the export tests for the non-default formats. The
most annoying part is setting them correctly to begin with.

But yes, if it is only with regard to the document, those "bugs" can sit
around.

Scott

Re: [LyX/master] Fix export with polyglossia and uninvert a manual

[Scott Kostyshak]

On Thu, Oct 29, 2015 at 09:04:10AM +, Guenter Milde wrote:

> * Math.lyx uses packages that either fail to compile or produce wrong output
>   with XeTeX/LuaTeX (with both, TeX fonts and non-TeX fonts) in any language
>   (e.g. mhchem)
>   Also, some accents (e.g. \t) in "mathematical text" are missing in the
>   output.
>   
>   -> invert all */Math.lyx tests with XeTeX and LuaTeX.
>   -> don't add partial fixes (they may be outdated once the real problem is
>  solved).
>   -> consider a safeguard in LyX against using Xe/LuaTeX with non-compatible
>  packages.
>   
> * fr/UserGuide.lyx compiles but has wrong output for LuaTeX with TeX fonts:
>   Eg. the ToC is called "Table des matiÃĺres" (should be "Table des 
> matières").
>   This is a bug in Babel (or frenchb.ldf), where detecting
>   XeTeX/LuaTeX assumes that non-TeX fonts are used and the encoding is utf8. 
>   (It affects other Babel languages as well.)
> 
>   -> invert all doc/fr/*.lyx tests with XeTeX/LuaTeX and TeX fonts.
>   -> don't add partial fixes.
>   -> consider a safeguard in LyX against using Xe/LuaTeX with non-compatible
>  packages.
> 
> * es/Customization.lyx
>   Same problem with LuaTeX and TeX fonts: ToC becomes "ÃŊndice general".
> 
>   -> invert all doc/es/*.lyx tests with XeTeX/LuaTeX and TeX fonts.
>   ...

Thanks for taking a look at these, Günter. I will follow your advice and
invert the problematic files and create a bug report that we can
summarize the important information in.

> >> I believe it would be better to suspend testing the problematic files
> >> with "non-TeX fonts" until the issues are solved.
> 
> > Ideally code changes would not be committed before broken tests are
> > fixed or the regressions caused by the commits are fixed. That should
> > all be done in one commit.
> 
> We cannot expect this for cases where there are several reasons for failure
> affecting a large number of documents. (Just consider the size of a single
> commit fixing all cases of "missing character".)

What happened does not seem good either. Because of the commit several
regressions went unnoticed because the tests were broken. I do not mean
that the bugs must always be fixed, but we could for example invert the
tests affected by the change. This would at least keep the testing
output clean and useful. We could then create a bug report so that
inverting the tests does not mean we forget about the bug. The tests are
meant to notify us that there is a problem. After they do their job of
notifying us, we need to make sure we don't forget about the problem
(e.g. make a bug report) and move on.

> -> We need a framework to handle such "generic" problems like "Xe/LuaTeX
>with TeX fonts".
> 
>Suggestion:
> 
>* Code changes/commits that break tests with the default export method of
>  important documents require immediate action.
>  
>* Problems with other export routes can be handled later (but should not
>  be forgotten).
>  
> Also, not every developer should be expected to run the export tests for
> every small commit - however, there should be a commitment to fix
> problems introduced by ones changes.
> 
> 
> >> Fixing problems in the Manuals found by automatic testing is a good thing.
> >> Modifying the Manuals "somehow" just to please automatic testing is not.
> 
> > I agree. I think that in most cases fixing the manuals to compile to
> > multiple formats improves the documents and forces us to have clean
> > preamble code.
> 
> In this case, the preamble code got more complicated, not cleaner.
> Also, it takes time to agree on a best praxis.

Yes, good point.

> For the test with non-TeX fonts, the preamble code involving lmodern.sty
> must be changed, otherwise 8-bit fonts are used with wrong font encoding!
> 
> One possibility would be to remove the special casing altogether,
> possibly selecting a better font in the GUI (but this was rejected some
> years ago).
> 
> Another option is the following patch (similar in all places where this
> preamble code is used):

I'm fine with this change, but my opinion should not carry much weight
since I really don't understand this well.

Scott

Re: [LyX/master] Fix export with polyglossia and uninvert a manual

[Scott Kostyshak]

On Mon, Oct 26, 2015 at 10:59:05AM +, Guenter Milde wrote:
> On 2015-10-26, Scott Kostyshak wrote:
> > On Sun, Oct 25, 2015 at 11:17:15AM +0100, Georg Baum wrote:
> >> Scott Kostyshak wrote:
> 
> >> > -% increase link area for cross-references and autoname them,
> >> > -\AtBeginDocument{\renewcommand{\ref}[1]{\mbox{\autoref{#1
> >> > -\addto\extrasfrench{%
> >> > - \renewcommand*{\equationautorefname}[1]{}%
> >> > - \renewcommand{\sectionautorefname}{sec.\negthinspace}%
> >> > - \renewcommand{\subsectionautorefname}{sec.\negthinspace}%
> >> > - \renewcommand{\subsubsectionautorefname}{sec.\negthinspace}%
> >> > +\@ifpackageloaded{babel}{
> >> > + % increase link area for cross-references and autoname them,
> >> > + \AtBeginDocument{\renewcommand{\ref}[1]{\mbox{\autoref{#1
> >> > + \addto\extrasfrench{%
> >> > +  \renewcommand*{\equationautorefname}[1]{}%
> >> > +  \renewcommand{\sectionautorefname}{sec.\negthinspace}%
> >> > +  \renewcommand{\subsectionautorefname}{sec.\negthinspace}%
> >> > +  \renewcommand{\subsubsectionautorefname}{sec.\negthinspace}%
> >> > + }
> >> >  }
> 
> ...
> 
> >> If some features of a document are implemented in a babel specific way
> >> (even if the feature would in theory not need babel), then we should
> >> rather declare that explicitly like in 78c60de9bb IMHO, than to
> >> pretend that polyglossia can be used, but silently change the output.
> 
> > Ah this is a good point. Dang it. In this case I think it makes even
> > more sense now to remove the babel-specific preamble code, which there
> > seems to be support for from others as well. 
> 
> In this case, I would rather not remove the code but change it to be usable
> without babel:
> 
> * if there is only one language in the document (or if the autrefname can be
>   the same in all used languages), do the (r)definitions directly:
>   
>  % increase link area for cross-references and autoname them,
>  \AtBeginDocument{\renewcommand{\ref}[1]{\mbox{\autoref{#1
>  \renewcommand*{\equationautorefname}[1]{}%
>  \renewcommand{\sectionautorefname}{sec.~}%
>  \renewcommand{\subsectionautorefname}{sec.~}%
>  \renewcommand{\subsubsectionautorefname}{sec.~}%
>   
>   (or in the \AtBeginDocument, if this works better).
>   
> * if there is more than one language and autoreferencenames must be language
>   speecific, define them for both, babel and polyglossia.
>   
>   Proof of concept:
>   
> % increase link area for cross-references and autoname them,
> \AtBeginDocument{\renewcommand{\ref}[1]{\mbox{\autoref{#1
> \renewcommand*{\equationautorefname}[1]{}%
> \@ifpackageloaded{babel}{
>  \addto\extrasfrench{%
>   \renewcommand{\sectionautorefname}{sec.~}%
>   % ...
>  }
>  \addto\extrasgerman{%
>   \renewcommand{\sectionautorefname}{Abschnitt~}%
>   % ...
>  }
> }   
> \@ifpackageloaded{polyglossia}{
>  \addto\captionsfrench{%
>   \renewcommand{\sectionautorefname}{sec.~}%
>   % ...
>  }
>  \addto\captionsgerman{%
>   \renewcommand{\sectionautorefname}{Abschnitt~}%
>   % ...
>  }
> }   
> 
>   (Not tested with polyglossia. The polyglossia manual recommends using
>   \gappto from the etoolbox package but says \addto is also available.)
> 
> > I would prefer to leave the babel conditional in because of my reason
> > above, until we come to a decision and action on whether the
> > babel-specific code should be removed (e.g. fixing #9817?). That said, I
> > do understand that two wrongs doesn't make a right so let me know if you
> > believe strongly that I should revert.
> 
> I believe it would be better to suspend testing the problematic files
> with "non-TeX fonts" until the issues are solved.

Ideally code changes would not be committed before broken tests are
fixed or the regressions caused by the commits are fixed. That should
all be done in one commit.
> 
> Fixing problems in the Manuals found by automatic testing is a good thing.
> Modifying the Manuals "somehow" just to please automatic testing is not.

I agree. I think that in most cases fixing the manuals to compile to
multiple formats improves the documents and forces us to have clean
preamble code.

Scott

Re: [LyX/master] Fix export with polyglossia and uninvert a manual

[Kornel Benko]

Am Mittwoch, 28. Oktober 2015 um 11:22:31, schrieb Scott Kostyshak 

> On Mon, Oct 26, 2015 at 10:59:05AM +, Guenter Milde wrote:
> > On 2015-10-26, Scott Kostyshak wrote:
> > > On Sun, Oct 25, 2015 at 11:17:15AM +0100, Georg Baum wrote:
> > >> Scott Kostyshak wrote:
> > 
> > >> > -% increase link area for cross-references and autoname them,
> > >> > -\AtBeginDocument{\renewcommand{\ref}[1]{\mbox{\autoref{#1
> > >> > -\addto\extrasfrench{%
> > >> > - \renewcommand*{\equationautorefname}[1]{}%
> > >> > - \renewcommand{\sectionautorefname}{sec.\negthinspace}%
> > >> > - \renewcommand{\subsectionautorefname}{sec.\negthinspace}%
> > >> > - \renewcommand{\subsubsectionautorefname}{sec.\negthinspace}%
> > >> > +\@ifpackageloaded{babel}{
> > >> > + % increase link area for cross-references and autoname them,
> > >> > + \AtBeginDocument{\renewcommand{\ref}[1]{\mbox{\autoref{#1
> > >> > + \addto\extrasfrench{%
> > >> > +  \renewcommand*{\equationautorefname}[1]{}%
> > >> > +  \renewcommand{\sectionautorefname}{sec.\negthinspace}%
> > >> > +  \renewcommand{\subsectionautorefname}{sec.\negthinspace}%
> > >> > +  \renewcommand{\subsubsectionautorefname}{sec.\negthinspace}%
> > >> > + }
> > >> >  }
> > 
> > ...
> > 
> > >> If some features of a document are implemented in a babel specific way
> > >> (even if the feature would in theory not need babel), then we should
> > >> rather declare that explicitly like in 78c60de9bb IMHO, than to
> > >> pretend that polyglossia can be used, but silently change the output.
> > 
> > > Ah this is a good point. Dang it. In this case I think it makes even
> > > more sense now to remove the babel-specific preamble code, which there
> > > seems to be support for from others as well. 
> > 
> > In this case, I would rather not remove the code but change it to be usable
> > without babel:
> > 
> > * if there is only one language in the document (or if the autrefname can be
> >   the same in all used languages), do the (r)definitions directly:
> >   
> >  % increase link area for cross-references and autoname them,
> >  \AtBeginDocument{\renewcommand{\ref}[1]{\mbox{\autoref{#1
> >  \renewcommand*{\equationautorefname}[1]{}%
> >  \renewcommand{\sectionautorefname}{sec.~}%
> >  \renewcommand{\subsectionautorefname}{sec.~}%
> >  \renewcommand{\subsubsectionautorefname}{sec.~}%
> >   
> >   (or in the \AtBeginDocument, if this works better).
> >   
> > * if there is more than one language and autoreferencenames must be language
> >   speecific, define them for both, babel and polyglossia.
> >   
> >   Proof of concept:
> >   
> > % increase link area for cross-references and autoname them,
> > \AtBeginDocument{\renewcommand{\ref}[1]{\mbox{\autoref{#1
> > \renewcommand*{\equationautorefname}[1]{}%
> > \@ifpackageloaded{babel}{
> >  \addto\extrasfrench{%
> >   \renewcommand{\sectionautorefname}{sec.~}%
> >   % ...
> >  }
> >  \addto\extrasgerman{%
> >   \renewcommand{\sectionautorefname}{Abschnitt~}%
> >   % ...
> >  }
> > }   
> > \@ifpackageloaded{polyglossia}{
> >  \addto\captionsfrench{%
> >   \renewcommand{\sectionautorefname}{sec.~}%
> >   % ...
> >  }
> >  \addto\captionsgerman{%
> >   \renewcommand{\sectionautorefname}{Abschnitt~}%
> >   % ...
> >  }
> > }   
> > 
> >   (Not tested with polyglossia. The polyglossia manual recommends using
> >   \gappto from the etoolbox package but says \addto is also available.)
> > 
> > > I would prefer to leave the babel conditional in because of my reason
> > > above, until we come to a decision and action on whether the
> > > babel-specific code should be removed (e.g. fixing #9817?). That said, I
> > > do understand that two wrongs doesn't make a right so let me know if you
> > > believe strongly that I should revert.
> > 
> > I believe it would be better to suspend testing the problematic files
> > with "non-TeX fonts" until the issues are solved.
> 
> Ideally code changes would not be committed before broken tests are
> fixed or the regressions caused by the commits are fixed. That should
> all be done in one commit.
> > 
> > Fixing problems in the Manuals found by automatic testing is a good thing.
> > Modifying the Manuals "somehow" just to please automatic testing is not.
> 
> I agree. I think that in most cases fixing the manuals to compile to
> multiple formats improves the documents and forces us to have clean
> preamble code.
> 
> Scott

We should also create a directory containing some uncompilable lyx-files which 
are supposed
to compile in the future due to corrected handling in lyx.

How about "development/attic/testdocs"?

They would be the first to test with newly fixed lyx. If some of them compile, 
they can be removed.

Kornel


signature.asc
Description: This is a digitally signed message part.

Re: [LyX/master] Fix export with polyglossia and uninvert a manual

[Scott Kostyshak]

On Wed, Oct 28, 2015 at 04:36:26PM +0100, Kornel Benko wrote:

> We should also create a directory containing some uncompilable lyx-files 
> which are supposed
> to compile in the future due to corrected handling in lyx.
> 
> How about "development/attic/testdocs"?
> 
> They would be the first to test with newly fixed lyx. If some of them 
> compile, they can be removed.

Why removed? Don't we want to make sure they keep compiling? Or you mean
that they should be moved back to the main documentation folder?

I agree that we need a folder for documents that we don't want in our
main documentation. I think we should have both documents that we expect
to compile in there (e.g. to prevent regressions) and documents that we
hope compile in the future. The reason for this mixing and not having
e.g. two separate folders is because often one document will compile for
one format and not for another. That's why we can signal such things
with the framework we already have set up, with INVERTED_SEE-README.
Although it feels a little hackish, I actually think it works quite
well.

Scott

Re: [LyX/master] Fix export with polyglossia and uninvert a manual

[Kornel Benko]

Am Mittwoch, 28. Oktober 2015 um 11:45:46, schrieb Scott Kostyshak 

> On Wed, Oct 28, 2015 at 04:36:26PM +0100, Kornel Benko wrote:
> 
> > We should also create a directory containing some uncompilable lyx-files 
> > which are supposed
> > to compile in the future due to corrected handling in lyx.
> > 
> > How about "development/attic/testdocs"?
> > 
> > They would be the first to test with newly fixed lyx. If some of them 
> > compile, they can be removed.
> 
> Why removed? Don't we want to make sure they keep compiling? Or you mean
> that they should be moved back to the main documentation folder?

Sorry for the confusion. I mean e.g. manual files which do not compile because 
lyx creates erroneous tex file.
(No ERT or bad preamble code)
If such file now compiles, then we don't need it anymore. We have still the 
manual.

> I agree that we need a folder for documents that we don't want in our
> main documentation. I think we should have both documents that we expect
> to compile in there (e.g. to prevent regressions) and documents that we
> hope compile in the future. The reason for this mixing and not having
> e.g. two separate folders is because often one document will compile for
> one format and not for another.

Under 'it compiles' I meant "document compiles for _all_ possible formats"

> That's why we can signal such things
> with the framework we already have set up, with INVERTED_SEE-README.
> Although it feels a little hackish, I actually think it works quite
> well.

Hm, yes.

> Scott

Kornel

signature.asc
Description: This is a digitally signed message part.

Re: [LyX/master] Fix export with polyglossia and uninvert a manual

[Scott Kostyshak]

On Wed, Oct 28, 2015 at 05:16:55PM +0100, Kornel Benko wrote:
> Am Mittwoch, 28. Oktober 2015 um 11:45:46, schrieb Scott Kostyshak 
> 
> > On Wed, Oct 28, 2015 at 04:36:26PM +0100, Kornel Benko wrote:
> > 
> > > We should also create a directory containing some uncompilable lyx-files 
> > > which are supposed
> > > to compile in the future due to corrected handling in lyx.
> > > 
> > > How about "development/attic/testdocs"?
> > > 
> > > They would be the first to test with newly fixed lyx. If some of them 
> > > compile, they can be removed.
> > 
> > Why removed? Don't we want to make sure they keep compiling? Or you mean
> > that they should be moved back to the main documentation folder?
> 
> Sorry for the confusion. I mean e.g. manual files which do not compile 
> because lyx creates erroneous tex file.
> (No ERT or bad preamble code)
> If such file now compiles, then we don't need it anymore. We have still the 
> manual.

Ah I see what you mean.

> > I agree that we need a folder for documents that we don't want in our
> > main documentation. I think we should have both documents that we expect
> > to compile in there (e.g. to prevent regressions) and documents that we
> > hope compile in the future. The reason for this mixing and not having
> > e.g. two separate folders is because often one document will compile for
> > one format and not for another.
> 
> Under 'it compiles' I meant "document compiles for _all_ possible formats"

OK.

Scott

Re: [LyX/master] Fix export with polyglossia and uninvert a manual

[Guenter Milde]

On 2015-10-28, Kornel Benko wrote:
> Am Mittwoch, 28. Oktober 2015 um 11:45:46, schrieb Scott Kostyshak 
> 
>> On Wed, Oct 28, 2015 at 04:36:26PM +0100, Kornel Benko wrote:

>> > We should also create a directory containing some uncompilable lyx-files 
>> > which are supposed
>> > to compile in the future due to corrected handling in lyx.

...

> Under 'it compiles' I meant "document compiles for _all_ possible formats"

I would restrict this to "all formats that are expected to work".

* Some features, symbols and packags are simply not available in all formats.
* Some combinations (like XeteX + TeX fonts are currently extesively tested
  butg rarely used in real live.
  
OTOH, I suggest testing the documents not only with LyX's default mix of
8-bit latex-input-encodings, but also with settings 
\inputencoding ascii and
\inputencoding utf8
which are the ones most useful for exported LaTeX files these days.

Günter

Re: [LyX/master] Fix export with polyglossia and uninvert a manual

[Scott Kostyshak]

On Wed, Oct 28, 2015 at 05:51:28PM +, Guenter Milde wrote:
> On 2015-10-28, Kornel Benko wrote:
> > Am Mittwoch, 28. Oktober 2015 um 11:45:46, schrieb Scott Kostyshak 
> > 
> >> On Wed, Oct 28, 2015 at 04:36:26PM +0100, Kornel Benko wrote:
> 
> >> > We should also create a directory containing some uncompilable lyx-files 
> >> > which are supposed
> >> > to compile in the future due to corrected handling in lyx.
> 
> ...
> 
> > Under 'it compiles' I meant "document compiles for _all_ possible formats"
> 
> I would restrict this to "all formats that are expected to work".
> 
> * Some features, symbols and packags are simply not available in all formats.
> * Some combinations (like XeteX + TeX fonts are currently extesively tested
>   butg rarely used in real live.
>   
> OTOH, I suggest testing the documents not only with LyX's default mix of
> 8-bit latex-input-encodings, but also with settings 
> \inputencoding ascii and
> \inputencoding utf8
> which are the ones most useful for exported LaTeX files these days.

OK good to know. Yes it would be nice to extend the tests by setting
certain document settings.

Scott

Re: [LyX/master] Fix export with polyglossia and uninvert a manual

[Kornel Benko]

Am Montag, 26. Oktober 2015 um 10:59:05, schrieb Guenter Milde 

> BTW: what does the term "invert/revert" mean in the autotest context?

Essentially the same: The test is (for now) expected to fail, so that it does 
not
appear in the list of failed tests.

If the test fails, it means that in the meantime lyx produces better output and 
we can
remove this test from our list specified in 
"development/autotests/revertedTests".

Kornel

signature.asc
Description: This is a digitally signed message part.

Re: [LyX/master] Fix export with polyglossia and uninvert a manual

[Guenter Milde]

On 2015-10-26, Scott Kostyshak wrote:
> On Sun, Oct 25, 2015 at 11:17:15AM +0100, Georg Baum wrote:
>> Scott Kostyshak wrote:

>> > -% increase link area for cross-references and autoname them,
>> > -\AtBeginDocument{\renewcommand{\ref}[1]{\mbox{\autoref{#1
>> > -\addto\extrasfrench{%
>> > - \renewcommand*{\equationautorefname}[1]{}%
>> > - \renewcommand{\sectionautorefname}{sec.\negthinspace}%
>> > - \renewcommand{\subsectionautorefname}{sec.\negthinspace}%
>> > - \renewcommand{\subsubsectionautorefname}{sec.\negthinspace}%
>> > +\@ifpackageloaded{babel}{
>> > + % increase link area for cross-references and autoname them,
>> > + \AtBeginDocument{\renewcommand{\ref}[1]{\mbox{\autoref{#1
>> > + \addto\extrasfrench{%
>> > +  \renewcommand*{\equationautorefname}[1]{}%
>> > +  \renewcommand{\sectionautorefname}{sec.\negthinspace}%
>> > +  \renewcommand{\subsectionautorefname}{sec.\negthinspace}%
>> > +  \renewcommand{\subsubsectionautorefname}{sec.\negthinspace}%
>> > + }
>> >  }

...

>> If some features of a document are implemented in a babel specific way
>> (even if the feature would in theory not need babel), then we should
>> rather declare that explicitly like in 78c60de9bb IMHO, than to
>> pretend that polyglossia can be used, but silently change the output.

> Ah this is a good point. Dang it. In this case I think it makes even
> more sense now to remove the babel-specific preamble code, which there
> seems to be support for from others as well. 

In this case, I would rather not remove the code but change it to be usable
without babel:

* if there is only one language in the document (or if the autrefname can be
  the same in all used languages), do the (r)definitions directly:
  
 % increase link area for cross-references and autoname them,
 \AtBeginDocument{\renewcommand{\ref}[1]{\mbox{\autoref{#1
 \renewcommand*{\equationautorefname}[1]{}%
 \renewcommand{\sectionautorefname}{sec.~}%
 \renewcommand{\subsectionautorefname}{sec.~}%
 \renewcommand{\subsubsectionautorefname}{sec.~}%
  
  (or in the \AtBeginDocument, if this works better).
  
* if there is more than one language and autoreferencenames must be language
  speecific, define them for both, babel and polyglossia.
  
  Proof of concept:
  
% increase link area for cross-references and autoname them,
\AtBeginDocument{\renewcommand{\ref}[1]{\mbox{\autoref{#1
\renewcommand*{\equationautorefname}[1]{}%
\@ifpackageloaded{babel}{
 \addto\extrasfrench{%
  \renewcommand{\sectionautorefname}{sec.~}%
  % ...
 }
 \addto\extrasgerman{%
  \renewcommand{\sectionautorefname}{Abschnitt~}%
  % ...
 }
}   
\@ifpackageloaded{polyglossia}{
 \addto\captionsfrench{%
  \renewcommand{\sectionautorefname}{sec.~}%
  % ...
 }
 \addto\captionsgerman{%
  \renewcommand{\sectionautorefname}{Abschnitt~}%
  % ...
 }
}   

  (Not tested with polyglossia. The polyglossia manual recommends using
  \gappto from the etoolbox package but says \addto is also available.)

> I would prefer to leave the babel conditional in because of my reason
> above, until we come to a decision and action on whether the
> babel-specific code should be removed (e.g. fixing #9817?). That said, I
> do understand that two wrongs doesn't make a right so let me know if you
> believe strongly that I should revert.

I believe it would be better to suspend testing the problematic files
with "non-TeX fonts" until the issues are solved.

Fixing problems in the Manuals found by automatic testing is a good thing.
Modifying the Manuals "somehow" just to please automatic testing is not.

BTW: what does the term "invert/revert" mean in the autotest context?


Günter

Re: [LyX/master] Fix export with polyglossia and uninvert a manual

[Georg Baum]

Scott Kostyshak wrote:

> -% increase link area for cross-references and autoname them,
> -\AtBeginDocument{\renewcommand{\ref}[1]{\mbox{\autoref{#1
> -\addto\extrasfrench{%
> - \renewcommand*{\equationautorefname}[1]{}%
> - \renewcommand{\sectionautorefname}{sec.\negthinspace}%
> - \renewcommand{\subsectionautorefname}{sec.\negthinspace}%
> - \renewcommand{\subsubsectionautorefname}{sec.\negthinspace}%
> +\@ifpackageloaded{babel}{
> + % increase link area for cross-references and autoname them,
> + \AtBeginDocument{\renewcommand{\ref}[1]{\mbox{\autoref{#1
> + \addto\extrasfrench{%
> +  \renewcommand*{\equationautorefname}[1]{}%
> +  \renewcommand{\sectionautorefname}{sec.\negthinspace}%
> +  \renewcommand{\subsectionautorefname}{sec.\negthinspace}%
> +  \renewcommand{\subsubsectionautorefname}{sec.\negthinspace}%
> + }
>  }

If you switch between babel and polyglossia you will suddenly get different 
references in the PDF, which I would not expect at all as a user.

If some features of a document are implemented in a babel specific way (even 
if the feature would in theory not need babel), then we should rather 
declare that explicitly like in 78c60de9bb IMHO, than to pretend that 
polyglossia can be used, but silently change the output.


Georg

Re: [LyX/master] Fix export with polyglossia and uninvert a manual

[Scott Kostyshak]

On Sun, Oct 25, 2015 at 11:17:15AM +0100, Georg Baum wrote:
> Scott Kostyshak wrote:
> 
> > -% increase link area for cross-references and autoname them,
> > -\AtBeginDocument{\renewcommand{\ref}[1]{\mbox{\autoref{#1
> > -\addto\extrasfrench{%
> > - \renewcommand*{\equationautorefname}[1]{}%
> > - \renewcommand{\sectionautorefname}{sec.\negthinspace}%
> > - \renewcommand{\subsectionautorefname}{sec.\negthinspace}%
> > - \renewcommand{\subsubsectionautorefname}{sec.\negthinspace}%
> > +\@ifpackageloaded{babel}{
> > + % increase link area for cross-references and autoname them,
> > + \AtBeginDocument{\renewcommand{\ref}[1]{\mbox{\autoref{#1
> > + \addto\extrasfrench{%
> > +  \renewcommand*{\equationautorefname}[1]{}%
> > +  \renewcommand{\sectionautorefname}{sec.\negthinspace}%
> > +  \renewcommand{\subsectionautorefname}{sec.\negthinspace}%
> > +  \renewcommand{\subsubsectionautorefname}{sec.\negthinspace}%
> > + }
> >  }
> 
> If you switch between babel and polyglossia you will suddenly get different 
> references in the PDF, which I would not expect at all as a user.
> 
> If some features of a document are implemented in a babel specific way (even 
> if the feature would in theory not need babel), then we should rather 
> declare that explicitly like in 78c60de9bb IMHO, than to pretend that 
> polyglossia can be used, but silently change the output.

Ah this is a good point. Dang it. In this case I think it makes even
more sense now to remove the babel-specific preamble code, which there
seems to be support for from others as well. There are several benefits
to removing the babel-specific preamble code. One of them that I'm
currently focused on is that if we use "Always babel" we will miss out
on a lot of tests. Basically, any document that Uwe has edited will
contain babel-specific preamble code and thus cannot be used as a test
of XeTeX or LuaTeX with non-TeX fonts.

I would prefer to leave the babel conditional in because of my reason
above, until we come to a decision and action on whether the
babel-specific code should be removed (e.g. fixing #9817?). That said, I
do understand that two wrongs doesn't make a right so let me know if you
believe strongly that I should revert.

Scott