Re: problems with quotes

[Jean-Marc Lasgouttes]

Le 18/12/2016 à 12:21, Jürgen Spitzmüller a écrit :

Would you like to have a go? After all, inset-forall is your child ;-)


This particular child was born in 2009, and I'd say that in lfun-years, 
it is very much adult. But in memory of the good old days, I'll try to 
have a look.


JMarc

Re: problems with quotes

[Jürgen Spitzmüller]

Am Sonntag, den 18.12.2016, 20:36 + schrieb Guenter Milde:
> We could also take the symbol from T1 

Good idea.

> or the typewriter font:

IMHO mixing fonts is not a good idea.

I think I'll go for

\DeclareTextSymbolDefault{\textquotedbl}{T1}

Thanks,
Jürgen

signature.asc
Description: This is a digitally signed message part

Re: problems with quotes

[Guenter Milde]

On 2016-12-18, Jürgen Spitzmüller wrote:
> Am Montag, den 12.12.2016, 11:01 + schrieb Guenter Milde:
>> > > As opposed to that, I would rather introduce a plain quote style
>> > > and ditch the straight quote special char 

>> Seconded. This could also solve the issue with the currently missing
>> straight single quote/apostrophe.

> Günter, do you have an idea how straight double quotes can be produced
> with OT1 font encoding? \textquotedbl is unavailable, and \char`\"
> (what we use currently for the straight quote in OT1) does not produce
> a straight quotation mark, but a closing English quotation mark.
...

We could also take the symbol from T1 or the typewriter font:

\documentclass[]{article}
\usepackage{textcomp}
% \DeclareTextSymbol{\textquotedbl}{OT1}{`\"} % curly
\ProvideTextCommand{\textquotedbl}{OT1}{\texttt{\char'42}}
% \DeclareTextSymbolDefault{\textquotedbl}{T1}

\begin{document}

Straight quotes with OT1:
\textquotesingle{} and \textquotedbl.

\end{document}


Mark, that using the symbol from T1 does not require loading fontenc.

For the straight quotes, I'd prefer the typewriter font.
In a recent post, I asked about other characters missing in OT1 where I
suggest to either leave this to the user or use T1 characters as substitute.

Günter

Re: problems with quotes

[Jürgen Spitzmüller]

Am Montag, den 12.12.2016, 11:01 + schrieb Guenter Milde:
> > > As opposed to that, I would rather introduce a plain quote style
> > > and
> > > ditch the straight quote special char 
> 
> Seconded. This could also solve the issue with the currently missing
> straight single quote/apostrophe.

Günter, do you have an idea how straight double quotes can be produced
with OT1 font encoding? \textquotedbl is unavailable, and \char`\"
(what we use currently for the straight quote in OT1) does not produce
a straight quotation mark, but a closing English quotation mark.
(\textquotesingle, OTOH, is provided by textcomp, and works in OT1).

One option would be to define our own text command for OT1, e.g.

\DeclareTextCommandDefault{\textquotedouble}{\textquotesingle\kern-
.25ex\textquotesingle}

Jürgen

signature.asc
Description: This is a digitally signed message part

Re: problems with quotes

[Jürgen Spitzmüller]

Am Sonntag, den 18.12.2016, 12:18 +0100 schrieb Jean-Marc Lasgouttes:
> Le 18/12/2016 à 11:33, Jürgen Spitzmüller a écrit :
> > Am Sonntag, den 18.12.2016, 11:20 +0100 schrieb Jürgen Spitzmüller:
> > > That would have to be more upstream (selection-forall, in line
> > > with
> > > buffer-forall)
> > 
> > inset-forall-select, for that matter.
> 
> Or change inset-forall to operate on selection when there is one.

Yes, that would probably be what people expect.

Would you like to have a go? After all, inset-forall is your child ;-)

Jürgen

> 
> JMarc
> 

signature.asc
Description: This is a digitally signed message part

Re: problems with quotes

[Jean-Marc Lasgouttes]

Le 18/12/2016 à 11:33, Jürgen Spitzmüller a écrit :

Am Sonntag, den 18.12.2016, 11:20 +0100 schrieb Jürgen Spitzmüller:

That would have to be more upstream (selection-forall, in line with
buffer-forall)


inset-forall-select, for that matter.


Or change inset-forall to operate on selection when there is one.

JMarc

Re: problems with quotes

[Kornel Benko]

Am Sonntag, 18. Dezember 2016 um 11:33:16, schrieb Jürgen Spitzmüller 

> Am Sonntag, den 18.12.2016, 11:20 +0100 schrieb Jürgen Spitzmüller:
> > That would have to be more upstream (selection-forall, in line with
> > buffer-forall)
> 
> inset-forall-select, for that matter.
> 
> Jürgen

Sounds good.

Kornel

signature.asc
Description: This is a digitally signed message part.

Re: problems with quotes

[Jürgen Spitzmüller]

Am Sonntag, den 18.12.2016, 11:20 +0100 schrieb Jürgen Spitzmüller:
> That would have to be more upstream (selection-forall, in line with
> buffer-forall)

inset-forall-select, for that matter.

Jürgen

signature.asc
Description: This is a digitally signed message part

Re: problems with quotes

[Jürgen Spitzmüller]

Am Sonntag, den 18.12.2016, 11:11 +0100 schrieb Kornel Benko:
> Yes, this works too, thanks. Could we please get also a function for
> a selected region?

That would have to be more upstream (selection-forall, in line with
buffer-forall)

Jürgen

> 
>   Kornel

signature.asc
Description: This is a digitally signed message part

Re: problems with quotes

[Kornel Benko]

Am Sonntag, 18. Dezember 2016 um 10:42:17, schrieb Jürgen Spitzmüller 

> Am Sonntag, den 18.12.2016, 10:38 +0100 schrieb Kornel Benko:
> > Playing with quotes for a while now.
> > It seems impossible to change quotes for a whole selected region at
> > once.
> > From my POV, that would be nice. Or did I miss something?
> 
> This might be possible via Adv S

Thanks, yes it is.

> For a whole document, it is now easily possible with the lfun changes I
> just submitted.
> 
> Jürgen
> 

Yes, this works too, thanks. Could we please get also a function for a selected 
region?

Kornel

signature.asc
Description: This is a digitally signed message part.

Re: problems with quotes

[Jürgen Spitzmüller]

Am Sonntag, den 18.12.2016, 10:38 +0100 schrieb Kornel Benko:
> Playing with quotes for a while now.
> It seems impossible to change quotes for a whole selected region at
> once.
> From my POV, that would be nice. Or did I miss something?

This might be possible via Adv S

For a whole document, it is now easily possible with the lfun changes I
just submitted.

Jürgen

> 
>   Kornel

signature.asc
Description: This is a digitally signed message part

Re: problems with quotes

[Kornel Benko]

Am Samstag, 10. Dezember 2016 um 17:26:44, schrieb Jürgen Spitzmüller 

> Am Samstag, den 10.12.2016, 16:27 +0100 schrieb Jürgen Spitzmüller:
> > > With non-TeX fonts and babel, \guillemotleft, \gullemotright,
> > > \guilsinglleft, and \guilsinglright are taken from CM: this results
> > > in
> > > non-matching characters if the document font is not CM. (Exception:
> > > correct «» if the text language is french.)
> > 
> > I don't see this.
> 
> OK, I do now. Fixed in master.
> 
> Jürgen

Playing with quotes for a while now.
It seems impossible to change quotes for a whole selected region at once.
From my POV, that would be nice. Or did I miss something?

Kornel

signature.asc
Description: This is a digitally signed message part.

Re: problems with quotes

[Helge Hafting]

Den 09. des. 2016 11:46, skrev Jean-Marc Lasgouttes:


And pardon me for failing to be impressed by the modernity of people 
having unicode keyboards (how many keys does your keyboard have?) and 
selection of a character in a list or 65535 others in some GUI where I 
have to guess what is the category of the character I am looking for. 
Unicode is great, but it is not a religion I want to get enrolled in.
Nobody has all of unicode, but it is common to have what you need for 
your own language. LyX is not the only place for text, it is nice being 
able to type common symbols into email or vim as well. Text is not 
«ascii», it is «language». So a computer keyboard should have easy 
access to the symbols of the language in use. Especially now that 
unicode lets computers support the symbols of just about any language 
one might wish to type.


For Norwegian, «» is the preferred quote style (used in typesetting 
books & newspapers.)  These “ ” are also common, and preferred for 
nested quotes. So naturally, I have all of these accessible on my 
keyboard - I can «effortlessly type a “nested” quote in email» for example.


It doesn't matter much to me if LyX uses the symbols as-is or converts 
them to some kind of inset, as long as they work.  The LyX file tend to 
be more readable with unicode symbols - but perhaps not for those with a 
bad terminal font. I have decided to use a terminal font with the 
symbols I need though, I won't suffer a limited terminal.

Re: problems with quotes

[Jürgen Spitzmüller]

Am Montag, den 12.12.2016, 20:41 + schrieb Guenter Milde:
> Would it also be possible to override the "open/close" automatism?
> Currently, I have to 
>   write something, 
>   insert a quote, 
>   go back and delete what was written, 
>   go behind the quote again
> if I want a closing quote following a space for some reason (e.g.
> working
> around the missing "padding" of single quotes in French with Babel).
> 
> But maybe this does not need to be done with quote-insert if the
> second (and
> more important) wishlist item were implemented:
> 
>   Make the Quote inset mutable: a right-click should open a menu
> allowing to
>   set the arguments 
> position: open/close,
> level:primary/secondary, and 
> style:straight, english, french, ...

Yes, this is on my TODO list.
 
> And, when adding styles, I'd like to see one more special style:
> "default",
> which would use the document-wide style setting at time of export:
> 
> * changing the style of all quotes in one go from "german" to
> "french",
>   say would become simple.
>   
> * quotes would adapt in style: this comes handy when a language uses
>   alternative quoting styles, e.g. copying from a document with
> "german"
>   quotes to a document with "french" style adapts the style of quotes
>   automatically. If this is not desired (e.g. for quotes in foreign
> language
>   text parts), the existing "fixed" styles can be set in the context
> menu.

This is the "dynamic quotes" feature requested here (by me 13 years
ago):
http://www.lyx.org/trac/ticket/1499

> The code would become a lot simpler, if macros were used instead
> of ligatures for all LaTeX export:
> 
> - simpler conditonals,
> - no need to insert {} to prevent unwanted ligatures.

This is actually not a very tricky part of the code. More tricky was to
correctly terminate the macros in the given context space vs. {} cs.
nothing, which is done now in master.

Jürgen

signature.asc
Description: This is a digitally signed message part

Re: problems with quotes

[Jürgen Spitzmüller]

Am Montag, den 12.12.2016, 22:12 + schrieb Guenter Milde:
> Characters. Unfortunately, with LuaTeX it does this not only for
> French,
> but also English or German text --- I consider this a Polyglossia
> bug.
> 
>   English «is ‹fine›».
>   French «‹is› fine» spaced as it should.
>   English again «is ‹not fine›». Still spaces inside guillemets.
> 
> becomes:
> 
>   English «is ‹fine›».
>   French « ‹ is › fine » spaced as it should.
>   English again « is ‹ not fine › ». Still spaces inside guillemets.
> 
> The padding of guillemets seems to be "sticky".

That's
https://github.com/reutenauer/polyglossia/issues/68

> I suppose, there is no clear preference (similar to the situation in
> German
> with »«, «», or „“ for the primary quotes)

English inner quotes is apparently most used.

> Why not using literal Unicode characters in the TeX source for all
> non-TeX-font documents:
> 
> • no difference with Babel

The specific babel macros (e.g., \og, \fg for French) should be used.
And it does not break documents that make quote chars active (for other
purposes). 

> However, we need "hardcoded" spaces for text export (copy to external
> source), HTML and possibly as workaround for the single guillemet
> with
> Babel.

I don't know. Babel does not do it, and nobody seems to care.

> > > Babel (and Polyglossia with LuaTeX) will add the spaces if the
> > > text
> > > language is french by default (the user can supress this with
> > > preamble
> > > code). However, they will not add more space, if there is already
> > > space, so we don't need to care for "double space".
> > I am not sure about that. There is currently a bug with the
> > polyglossia/csquotes combination since both add space that adds up
> > in
> > the result:
> > https://github.com/reutenauer/polyglossia/issues/141
> 
> With Babel-French, a narrow no-break space and the Babel-inserted
> space
> around double guillemet add up (!).
> 
> With Polyglossia, spaces are not added.

Did you read that polyglossia ticket?

> 
> 
> I added test documents to #10451.

Thanks.

Jürgen

> 
> Thanks,
> Günter
> 
> 
> 
> 

signature.asc
Description: This is a digitally signed message part

Re: problems with quotes

[Guenter Milde]

Dear Jürgen,

spacing/padding of French quotes is a wide field...

On 2016-12-11, Jürgen Spitzmüller wrote:
> Am Samstag, den 10.12.2016, 19:44 + schrieb Guenter Milde:

>> More interestingly, it also put spaces between single guillemots!

> Yes, sure. This is what polyglossia does as well (with literal call-
> them-whatever-you-like). 

Characters. Unfortunately, with LuaTeX it does this not only for French,
but also English or German text --- I consider this a Polyglossia bug.

  English «is ‹fine›».
  French «‹is› fine» spaced as it should.
  English again «is ‹not fine›». Still spaces inside guillemets.

becomes:

  English «is ‹fine›».
  French « ‹ is › fine » spaced as it should.
  English again « is ‹ not fine › ». Still spaces inside guillemets.

The padding of guillemets seems to be "sticky".


> Also babel-french has support for that via
> \frquote and InnerGuillSingle=true, but that's quite hidden. 

Yes, it does not work for the LICR macros or literal characters, so no help
for the quote inset.

> It seems that single guillemets are rather unusual in French
> typesetting.

> \csquotes has four different styles for french quotes, none of them
> using single guillemets for nested quotations.
...

> These styles seem to be in accordance to how the Wikipedia has it:
> https://en.wikipedia.org/wiki/Quotation_mark#French

The French wikipedia cites 3 styles for secondary level quotes:

with literal Unicode characters and narrow no-break spaces (0x202F):Bug:
The narrow no-break space is present in the lyx source file but not
visible in the GUI!


Première méthode : « L’ouvreuse m’a dit : “Donnez-moi votre ticket.” Je
   le lui ai donné. » 

Seconde méthode : « L’ouvreuse m’a dit : « Donnez-moi votre ticket. » Je
  le lui ai donné. » 

Troisième méthode : « L’ouvreuse m’a dit : ‹ Donnez-moi votre ticket. ›
    Je le lui ai donné. » 

https://fr.wikipedia.org/wiki/Guillemet#Double_ou_triple_niveau_de_citation

The single guillemets are also listed as secondary quotes in French in

# http://en.wikipedia.org/wiki/Non-English_usage_of_quotation_marks
# http://de.wikipedia.org/wiki/Anf%C3%BChrungszeichen#Andere_Sprachen

I suppose, there is no clear preference (similar to the situation in German
with »«, «», or „“ for the primary quotes)

As « „“ » and « “” » are common in other languages, too

  'el':   u'«»“”',
  'es':   u'«»“”',
  'it':   u'«»“”',
  'pt':   u'«»“”',
  'ru':   u'«»„“',
  'ro':   u'„”«»' or u'«»„”',
  'pl':   u'„”«»' or u'«»“”',

it would be nice to have them as quoting style options.

>> Jean-Mark, do you think this is the right way to do it?


>> However, with XeTeX and Polyglossia spacing is more than wrong:
>> Spaces are missing with double guillemots and the opening single one
>> but
>> present before a closing single guillemot.

> It's correct for me (recent TL 2016) with the literal things. That's
> why I propose to use that for polyglossia (only).

Why not using literal Unicode characters in the TeX source for all
non-TeX-font documents:

• no difference with Babel
• no change (same bug) with Polyglossia + LuaTeX
• better (correct padding) with Polyglossia + XeTeX




>> I suggest to provide different styles:
>>  
>>   "french" with hard-coded spaces inside guillemots (in all
>>languages)
>>   "swiss" without spaces (in all languages).

> I think we should leave the spacing to the language packages (and/or
> packages/classes). 

Fine.
...

However, we need "hardcoded" spaces for text export (copy to external
source), HTML and possibly as workaround for the single guillemet with
Babel.


>> Babel (and Polyglossia with LuaTeX) will add the spaces if the text
>> language is french by default (the user can supress this with preamble
>> code). However, they will not add more space, if there is already
>> space, so we don't need to care for "double space".

> I am not sure about that. There is currently a bug with the
> polyglossia/csquotes combination since both add space that adds up in
> the result:
> https://github.com/reutenauer/polyglossia/issues/141

With Babel-French, a narrow no-break space and the Babel-inserted space
around double guillemet add up (!).

With Polyglossia, spaces are not added.


I added test documents to #10451.

Thanks,
Günter

Re: problems with quotes

[Guenter Milde]

On 2016-12-12, Jürgen Spitzmüller wrote:

>> >> As opposed to that, I would rather introduce a plain quote style and
>> >> ditch the straight quote special char
...
>> >> (and a language/style option "RequirePlainQuote").

>> Do we really require a separate option or could we just use the new
>> "straight" style with the existing QuoteStyle keyword in lib/languages?

> This would be needed if a language or style would not allow other than
> straight quotes. The straight style itself would just be yet another style
> to select (globally).
> IOW the option would enforce plain quotes in the given context (such as
> PassThru) even if a different quote style is selected in Document Settings.




> Additionally, I would extend the quote-insert lfun to select a given style.
> Then we can keep a way to insert straight quote from the menu.

Fine. 

Would it also be possible to override the "open/close" automatism?
Currently, I have to 
  write something, 
  insert a quote, 
  go back and delete what was written, 
  go behind the quote again
if I want a closing quote following a space for some reason (e.g. working
around the missing "padding" of single quotes in French with Babel).

But maybe this does not need to be done with quote-insert if the second (and
more important) wishlist item were implemented:

  Make the Quote inset mutable: a right-click should open a menu allowing to
  set the arguments 
position: open/close,
level:primary/secondary, and 
style:straight, english, french, ...

And, when adding styles, I'd like to see one more special style: "default",
which would use the document-wide style setting at time of export:

* changing the style of all quotes in one go from "german" to "french",
  say would become simple.
  
* quotes would adapt in style: this comes handy when a language uses
  alternative quoting styles, e.g. copying from a document with "german"
  quotes to a document with "french" style adapts the style of quotes
  automatically. If this is not desired (e.g. for quotes in foreign language
  text parts), the existing "fixed" styles can be set in the context menu.
  

>> > BTW I am not so sure anymore that the Hebrew restriction is correct.
>> > According to what I read on the Web, Hebrew uses typographic quotation
>> > marks as well, just RTL, either ``so'' or ``so,, (and they seem to be
>> > supported by babel-hebrew).

>> > https://en.wikipedia.org/wiki/Hebrew_punctuation#Quotation_marks

>> This means the special casing

>> // In PassThru and Hebrew, we use straight quotes
>> if (pass_thru_ || context_lang_ == "he_IL")
>> and
>> || runparams.local_font->language()->lang() == "hebrew";


>> in QuoteInset should be restrichted to PassThru, right?


> Yes. Additionally, however, we'd need to output macros
> (\textquote[dbl]{left,right}) instead of the ligatures, since the ligatures
> are (apparently) not supported in the HE font encodings.

Fine with me. 

> The attached patch works fine for me.

The code would become a lot simpler, if macros were used instead
of ligatures for all LaTeX export:

- simpler conditonals,
- no need to insert {} to prevent unwanted ligatures.


>> This would also obsolete the lyx2lyx conversion for Hebrew quotes.

> No, since in LyX 2.2.x and earlier, the LaTeX output does not compile. So
> the current reversion routine, which produces the LyX 2.2.x state, seems OK
> to me (an alternative would be to output the respective macros in ERT).

OK


Thanks,

Günter

Re: problems with quotes

[Jürgen Spitzmüller]

2016-12-12 12:01 GMT+01:00 Guenter Milde :

> On 2016-12-11, Jürgen Spitzmüller wrote:
>
> >> As opposed to that, I would rather introduce a plain quote style and
> >> ditch the straight quote special char
>
> Seconded. This could also solve the issue with the currently missing
> straight single quote/apostrophe.
>

Yes, good point.


> >> (and a language/style option "RequirePlainQuote").
>
> Do we really require a separate option or could we just use the new
> "straight" style with the existing QuoteStyle keyword in lib/languages?
>

This would be needed if a language or style would not allow other than
straight quotes. The straight style itself would just be yet another style
to select (globally).
IOW the option would enforce plain quotes in the given context (such as
PassThru) even if a different quote style is selected in Document Settings.

Additionally, I would extend the quote-insert lfun to select a given style.
Then we can keep a way to insert straight quote from the menu.


> > BTW I am not so sure anymore that the Hebrew restriction is correct.
> > According to what I read on the Web, Hebrew uses typographic quotation
> > marks as well, just RTL, either ``so'' or ``so,, (and they seem to be
> > supported by babel-hebrew).
>
> > https://en.wikipedia.org/wiki/Hebrew_punctuation#Quotation_marks
>
> Yes, indeed.
>
> This means the special casing
>
> // In PassThru and Hebrew, we use straight quotes
> if (pass_thru_ || context_lang_ == "he_IL")
> and
> || runparams.local_font->language()->lang() == "hebrew";
>
>
> in QuoteInset should be restrichted to PassThru, right?
>

Yes. Additionally, however, we'd need to output macros
(\textquote[dbl]{left,right}) instead of the ligatures, since the ligatures
are (apparently) not supported in the HE font encodings.

The attached patch works fine for me.


>
> This would also obsolete the lyx2lyx conversion for Hebrew quotes.
>

No, since in LyX 2.2.x and earlier, the LaTeX output does not compile. So
the current reversion routine, which produces the LyX 2.2.x state, seems OK
to me (an alternative would be to output the respective macros in ERT).

Jürgen


>
> Günter
>
>
diff --git a/src/insets/InsetQuotes.cpp b/src/insets/InsetQuotes.cpp
index 6e19a1b2f6..7653caede8 100644
--- a/src/insets/InsetQuotes.cpp
+++ b/src/insets/InsetQuotes.cpp
@@ -85,6 +85,13 @@ char const * const latex_quote_ot1[2][5] = {
 "\\guillemotleft{}", "\\guillemotright{}" }
 };
 
+char const * const latex_quote_nonlatin[2][5] = {
+   { "\\quotesinglbase ",  "\\textquoteleft{}", "\\textquoteright{}",
+"\\guilsinglleft{}", "\\guilsinglright{}" },
+  { "\\quotedblbase ", "\\textquotedblleft{}", "\\textquotedblright{}",
+"\\guillemotleft{}", "\\guillemotright{}" }
+};
+
 char const * const latex_quote_babel[2][5] = {
{ "\\glq ",  "'", "`", "\\flq{}", "\\frq{}" },
   { "\\glqq ", "''", "``", "\\flqq{}", "\\frqq{}" }
@@ -192,8 +199,8 @@ void InsetQuotes::parseString(string const & s)
 
 docstring InsetQuotes::displayString() const
 {
-   // In PassThru and Hebrew, we use straight quotes
-   if (pass_thru_ || context_lang_ == "he_IL")
+   // In PassThru, we use straight quotes
+   if (pass_thru_)
return (times_ == DoubleQuotes) ? from_ascii("\"") : 
from_ascii("'");
 
int const index = quote_index[side_][language_];
@@ -257,11 +264,8 @@ void InsetQuotes::latex(otexstream & os, OutputParams 
const & runparams) const
const int quoteind = quote_index[side_][language_];
string qstr;
 
-   // In some context, we output plain quotes
-   bool const force_plain = 
-   runparams.pass_thru
-   || runparams.local_font->language()->lang() == "hebrew";
-   if (force_plain)
+   // In pass-thru context, we output plain quotes
+   if (runparams.pass_thru)
qstr = (times_ == DoubleQuotes) ? "\"" : "'";
else if (language_ == FrenchQuotes && times_ == DoubleQuotes
&& prefixIs(runparams.local_font->language()->code(), "fr")
@@ -272,10 +276,16 @@ void InsetQuotes::latex(otexstream & os, OutputParams 
const & runparams) const
qstr = "\\og "; //the spaces are important here
else
qstr = " \\fg{}"; //and here
-   } else if (fontenc_ == "T1" && !runparams.use_polyglossia) {
+   } else if (fontenc_ == "T1"
+  && !runparams.local_font->language()->internalFontEncoding()
+  && !runparams.use_polyglossia) {
// Quotation marks for T1 font encoding
// (using ligatures)
qstr = latex_quote_t1[times_][quoteind];
+   } else if (runparams.local_font->language()->internalFontEncoding()) {
+   // Quotation marks for internal font encodings
+   // (ligatures not featured)
+   qstr = latex_quote_nonlatin[times_][quoteind];
 #ifdef DO_USE_DEFAULT_LANGUAGE
} 

Re: problems with quotes

[Guenter Milde]

On 2016-12-11, Jürgen Spitzmüller wrote:

>> As opposed to that, I would rather introduce a plain quote style and
>> ditch the straight quote special char 

Seconded. This could also solve the issue with the currently missing
straight single quote/apostrophe.

>> (and a language/style option "RequirePlainQuote"). 

Do we really require a separate option or could we just use the new
"straight" style with the existing QuoteStyle keyword in lib/languages?



> BTW I am not so sure anymore that the Hebrew restriction is correct.
> According to what I read on the Web, Hebrew uses typographic quotation
> marks as well, just RTL, either ``so'' or ``so,, (and they seem to be
> supported by babel-hebrew).

> https://en.wikipedia.org/wiki/Hebrew_punctuation#Quotation_marks

Yes, indeed. 

This means the special casing

// In PassThru and Hebrew, we use straight quotes
if (pass_thru_ || context_lang_ == "he_IL")
and
|| runparams.local_font->language()->lang() == "hebrew";


in QuoteInset should be restrichted to PassThru, right?

This would also obsolete the lyx2lyx conversion for Hebrew quotes.

Günter

Re: problems with quotes

[Jürgen Spitzmüller]

Am Sonntag, den 11.12.2016, 10:18 +0100 schrieb Jürgen Spitzmüller:
> > Once we are at adding new quote styles, a separate style for Hebrew
> > seems
> > better suited than auto-converting any Quote-inset to straight
> > quotes
> > in
> > Hebrew.
> 
> As opposed to that, I would rather introduce a plain quote style and
> ditch the straight quote special char (and a language/style option
> "RequirePlainQuote"). However, I have no time to do that now.

BTW I am not so sure anymore that the Hebrew restriction is correct.
According to what I read on the Web, Hebrew uses typographic quotation
marks as well, just RTL, either ``so'' or ``so,, (and they seem to be
supported by babel-hebrew).

https://en.wikipedia.org/wiki/Hebrew_punctuation#Quotation_marks

Jürgen

> 
> Jürgen

signature.asc
Description: This is a digitally signed message part

Re: problems with quotes

[Jürgen Spitzmüller]

Am Samstag, den 10.12.2016, 19:44 + schrieb Guenter Milde:
> > Hm, interestingly enough, 
> > \guillemotleft French \guillemotright
> > outputs the proper spacing with fontspec/LuaTeX.
> 
> More interestingly, it also put spaces between single guillemots!

Yes, sure. This is what polyglossia does as well (with literal call-
them-whatever-you-like). Also babel-french has support for that via
\frquote and InnerGuillSingle=true, but that's quite hidden. It seems
that single guillemets are rather unusual in French typesetting.

\csquotes has four different styles for french quotes, none of them
using single guillemets for nested quotations. Rather than that,
English double quotations marks are used by default (french=quotes,
also the default definition of babel-french's \ogii and \fgii), as in 
« texte “intégré” »
Another style (french=guillemets*) uses double quotation marks for
nested quotes, but merges the closing ones:
« texte « intégré »
(this is supported in babel-french via \frquote*{})
These styles seem to be in accordance to how the Wikipedia has it:
https://en.wikipedia.org/wiki/Quotation_mark#French

> 
> Jean-Mark, do you think this is the right way to do it?
> 
> However, with XeTeX and Polyglossia spacing is more than wrong:
> Spaces are missing with double guillemots and the opening single one
> but
> present before a closing single guillemot.

It's correct for me (recent TL 2016) with the literal things. That's
why I propose to use that for polyglossia (only).

> 
> I suggest to provide different styles:
>  
>   "french" with hard-coded spaces inside guillemots (in all
> languages)
>   
>   "swiss" without spaces (in all languages).
>   
> The default style for French language would be "french", other
> languages
> currently using "french" chould switch to "swiss" in lib/languages.
> 
> This way, the user can configure the spacing via
> Document>Settings>Language>Quote_Style.

I think we should leave the spacing to the language packages (and/or
packages/classes). I am not sure we gain anything with a dedicated
"French" style and hardcoding of spacing. However, the specifics of
French inner quotation marks might be a reason for a separate style.

> 
> Babel (and Polyglossia with LuaTeX) will add the spaces if the text
> language
> is french by default (the user can supress this with preamble code).
> However, they will not add more space, if there is already space, so
> wehttps://github.com/reutenauer/polyglossia/issues/141
> don't need to care for "double space".

I am not sure about that. There is currently a bug with the
polyglossia/csquotes combination since both add space that adds up in
the result:
https://github.com/reutenauer/polyglossia/issues/141

> Once we are at adding new quote styles, a separate style for Hebrew
> seems
> better suited than auto-converting any Quote-inset to straight quotes
> in
> Hebrew.

As opposed to that, I would rather introduce a plain quote style and
ditch the straight quote special char (and a language/style option
"RequirePlainQuote"). However, I have no time to do that now.

Jürgen

signature.asc
Description: This is a digitally signed message part

Re: problems with quotes

[Guenter Milde]

On 2016-12-09, Jean-Marc Lasgouttes wrote:
> Le 07/12/2016 à 18:49, Guenter Milde a écrit :

>> The current behaviour of "insert-quote" LFUN is usually called "smart
>> quotes". This can be done without Quote insets.

> Sure. What the Quote inset brings us, as Juergen pointed out already, is 
> some semantics. But it is not semantics that we really exploit now.

Yes, this is why I asked whether we could simplify LyX by using Unicode
instead of an inset or want to keep the inset for future wishlist
features. 

If the decision is to keep the inset, I have some suggestions (see below).

>> Only if we want to keep the type of quotes configurable also after
>> insertion, "dynamic" Quote insets are required.
>> But even then, the existing "static" Quote insets should be converted to
>> Unicode in the source files with lyx2lyx.

> I am not so sure about that. 

Nor am I any more.

> The example of inner spacein french quotes is interesting in this
> repect. Sure, we can take the lazy approach and add inde spaces by
> hand. 

I would not add the spaces by hand but via the quote-insert function.
The quote style determines what is inserted:

1. set the quote-style to "french"
2. press " (Shift-2 on my keyboard)

(BTW: should there be spaces also with single quillemets? Polyglossia with
 LuaTeX adds them, Babel only adds spaces to double ones.)

> This lead french text to be full of ugly grey 
> rectangles (the spaces) on screen in word processors. 

Not in my LyX, there the space has the same background color as the rest.

> But it is not really a quote and a space. In some sense the space is
> part of the quote, like the spacing in math typesetting.

With my idea of inserting quote and space via "quote-insert", this would
them be 2 characters, i.e. deleting would require 2 keypresses ...
With a quote inset you can delete this in one press.

OTOH, you can search for a text part with quoted words if the quotes are
characters but not if they are insets.

> And pardon me for failing to be impressed by the modernity of people 
> having unicode keyboards (how many keys does your keyboard have?)

105

> and selection of a character in a list or 65535 others in some GUI
> where I have to guess what is the category of the character I am
> looking for.  

In my text-editor, I search in the "Unicode 9.0.0 final names list" which is
just 48771 lines long.

But as I said above, I won't abadon the quote-insert LFUN and its "smart
quotes" feature, where one common keybinding can be used for all types of
typographical quotes.

In case the quote inset is to stay:

Suggestion for an improved Quote inset
==

Arguments and values

level:   primary# "regular" (in English double) quotes
 secondary  # "inner" (in English single) quotes
 
function:opening# left in LTR and right in RTL languages
 closing# right in LTR and left in RTL languages
  
style:   default# use document-wide setting (\quote_language)
 # auto # use language-dependent style
 straight   # or neutral or dumb ASCII quotes ("" '')
 english# 
 german
 ...

* values can be changed via context menu

* "style" describes sets of 4 strings for primary-opening,
  primary-closing, secondary-opening, secondary-closing, e.g.
  
  'german': ('„', '“', '‚', '‘'),
  'french': ('« ', ' »', '‹ ', ' ›'), # with narrow no-break space
  ...
  
* use Unicode characters for GUI rendering and export (copy to external,
  HTML export, LaTeX export)
  
* visual clue (color, frame, or similar) in the GUI to mark it as inset.


Günter

Re: problems with quotes

[Guenter Milde]

On 2016-12-10, Jürgen Spitzmüller wrote:

> [-- Type: text/plain, Encoding: quoted-printable --]

> Am Samstag, den 10.12.2016, 16:27 +0100 schrieb Jürgen Spitzmüller:
>> > No auto-spaces around quotes with polyglossia

>> The only supported way here seem to be literal glyphs. 

Do you mean literal characters?

  The ISO/IEC 10646 (Unicode) International Standard defines character, or
  abstract character as "a member of a set of elements used for the
  organisation, control, or representation of data".
  
  The term glyph is used to describe a particular visual appearance of a
  character. Many computer fonts consist of glyphs that are indexed by the
  numerical code of the corresponding character.
  
  -- https://en.wikipedia.org/wiki/Character_(computing)
  
  In typography, a glyph is "the specific shape, design, or representation
  of a character".
  
  -- https://en.wikipedia.org/wiki/Glyph#cite_ref-1


>> I suggest we output those if polyglossia is used.

> Hm, interestingly enough, 
> \guillemotleft French \guillemotright
> outputs the proper spacing with fontspec/LuaTeX.

More interestingly, it also put spaces between single guillemots!

Jean-Mark, do you think this is the right way to do it?

However, with XeTeX and Polyglossia spacing is more than wrong:
Spaces are missing with double guillemots and the opening single one but
present before a closing single guillemot.

I suggest to provide different styles:
 
  "french" with hard-coded spaces inside guillemots (in all languages)
  
  "swiss" without spaces (in all languages).
  
The default style for French language would be "french", other languages
currently using "french" chould switch to "swiss" in lib/languages.

This way, the user can configure the spacing via
Document>Settings>Language>Quote_Style.

Babel (and Polyglossia with LuaTeX) will add the spaces if the text language
is french by default (the user can supress this with preamble code).
However, they will not add more space, if there is already space, so we
don't need to care for "double space".


Once we are at adding new quote styles, a separate style for Hebrew seems
better suited than auto-converting any Quote-inset to straight quotes in
Hebrew.

Thanks,

Günter

Re: problems with quotes

[Jürgen Spitzmüller]

Am Samstag, den 10.12.2016, 16:27 +0100 schrieb Jürgen Spitzmüller:
> > With non-TeX fonts and babel, \guillemotleft, \gullemotright,
> > \guilsinglleft, and \guilsinglright are taken from CM: this results
> > in
> > non-matching characters if the document font is not CM. (Exception:
> > correct «» if the text language is french.)
> 
> I don't see this.

OK, I do now. Fixed in master.

Jürgen

signature.asc
Description: This is a digitally signed message part

Re: problems with quotes

[Jürgen Spitzmüller]

Am Samstag, den 10.12.2016, 16:27 +0100 schrieb Jürgen Spitzmüller:
> > No auto-spaces around quotes with polyglossia
> 
> The only supported way here seem to be literal glyphs. I suggest we
> output those if polyglossia is used.

Hm, interestingly enough, 
\guillemotleft French \guillemotright
outputs the proper spacing with fontspec/LuaTeX.

Jürgen

signature.asc
Description: This is a digitally signed message part

Re: problems with quotes

[Jürgen Spitzmüller]

Am Samstag, den 10.12.2016, 13:39 + schrieb Guenter Milde:
> With non-TeX fonts and babel, \guillemotleft, \gullemotright,
> \guilsinglleft, and \guilsinglright are taken from CM: this results
> in
> non-matching characters if the document font is not CM. (Exception:
> correct «» if the text language is french.)

I don't see this.

> No auto-spaces around quotes with polyglossia

The only supported way here seem to be literal glyphs. I suggest we
output those if polyglossia is used.

Jürgen

signature.asc
Description: This is a digitally signed message part

Re: problems with quotes

[Jürgen Spitzmüller]

Am Samstag, den 10.12.2016, 14:45 + schrieb Guenter Milde:
> Qould you apply the same fix to HTML export?

Done.

Jürgen

signature.asc
Description: This is a digitally signed message part

Re: problems with quotes

[Guenter Milde]

On 2016-12-10, Jürgen Spitzmüller wrote:

> [-- Type: text/plain, Encoding: quoted-printable --]

> Am Samstag, den 10.12.2016, 13:39 + schrieb Guenter Milde:
>> On screen, the spaces are shown 
>>   
>> * for all quotes, not just guillemets.

> Fixed in master.

Thanks a lot.

Qould you apply the same fix to HTML export?

(Actually, as LyXHTML uses utf8 encoding, both on-screen display and HTML
export could use the same characters.)

Günter

Re: problems with quotes

[Jürgen Spitzmüller]

Am Samstag, den 10.12.2016, 13:39 + schrieb Guenter Milde:
> On screen, the spaces are shown 
>   
> * for all quotes, not just guillemets.

Fixed in master.

Jürgen

signature.asc
Description: This is a digitally signed message part

Re: problems with quotes

[Jürgen Spitzmüller]

Am Samstag, den 10.12.2016, 13:39 + schrieb Guenter Milde:
> On screen, the spaces are shown 
> 
> * if the document language is set to French but not if only the text
>   language is French.

Not as of e8c932ff.

Jürgen

signature.asc
Description: This is a digitally signed message part

Re: problems with quotes

[Guenter Milde]

On 2016-12-09, Jean-Marc Lasgouttes wrote:
> Le 08/12/2016 à 00:09, Guenter Milde a écrit :

>> * consistency: currently, if a user sees a guillemot « on screen, it
>>   can be a literal character or a Quote inset and the LaTeX export can
>>   be

>>   "«" or "\guillemotleft" (depending on the "inputencoding")
>>   "<<" (for Quote inset, even if « is supported by the encoding)

> Concerning this particular part: it is a remnant of the situation for 
> french language 10+ years ago whaere we had to competing solutions. So 
> it should indeed be cleaned-up with vigor.

This is (partially) cleaned up already:

literal « »:
   converted to \guillemotleft, \guillemotright if required,
   kept if supported by encoding.
   
Quote inset fld frd:
   converted to \og \fr if language is french and language package babel,
   converted to \guillemotleft, \guillemotright else.
   
This works nice with Babel and 8-bit fonts.

With non-TeX fonts and babel, \guillemotleft, \gullemotright,
\guilsinglleft, and \guilsinglright are taken from CM: this results in
non-matching characters if the document font is not CM. (Exception:
correct «» if the text language is french.)

No auto-spaces around quotes with polyglossia or HTML

BTW: LyX (Built from git commit hash e4c46abe)  crashes with non-TeX fonts:
   
(  1) /usr/local/src/lyxtest/bin/lyx2.3: 
lyx::frontend::Alert::doError(std::__cxx11::basic_string const&, 
std::__cxx11::basic_string const&, bool)
(  2) /usr/local/src/lyxtest/bin/lyx2.3:

On screen, the spaces are shown 

* if the document language is set to French but not if only the text
  language is French.
  
* for all quotes, not just guillemets.


> However, it would need to double check whether the handling of « as 
> defined in some language agnostic unicode package caters for the need 
> that we have for French language (to be frank, I do not have the answer 
> here, but there is a lot of code in french.ldf).

Currently, no auto-space around literal quote characters:

Suggestions:

a) differentiate «french» (with hard-coded small spaces) and «swiss» (without)?

   Do you also want a space around quoted text for single guillemets?
   
   +1 works independent of Babel
   
   -1 double space, if user inserts a space, too.

b) preamble code: go=«, gc=» for encodings supporting «» or preamble code
   \renewcommand{\guillemotleft}{\og} \renewcommand{\guillemotright}{\fg}?

   Problematic, because «» will be turned into English quotes in other
   languages:

   If French quote characters are available on your keyboard, you can use
   them, to get proper spacing in L A TEX 2 ε see option og=«, fg=» p. 8.
   \og and \fg can be used outside French, they typeset then English
   quotes “ and ”.

   — `texdoc babel-french`

   It should work with Babel when redefining \guillemotleft and
   \guillemotright to \og fg in \extrasfrench and reset it in
   \noextrasfrench.
  
   +1  it does not matter, whether you input the spaces
   or not: Babel inserts them if required and adapts them to the
   correct size if they are there.
   
   -1 does not work with Polyglossia nor with HTML.

  
Günter  

Re: problems with quotes

[Jean-Marc Lasgouttes]

Le 08/12/2016 à 00:09, Guenter Milde a écrit :

* consistency: currently, if a user sees a guillemot « on screen, it can be
  a literal character or a Quote inset and the LaTeX export can be

  "«" or "\guillemotleft" (depending on the "inputencoding")
  "<<" (for Quote inset, even if « is supported by the encoding)


Concerning this particular part: it is a remnant of the situation for 
french language 10+ years ago whaere we had to competing solutions. So 
it should indeed be cleaned-up with vigor.


However, it would need to double check whether the handling of « as 
defined in some language agnostic unicode package catters for the need 
that we have for French language (to be frank, I do not have the answer 
here, but there is a lot of code in french.ldf).


JMarc

Re: problems with quotes

[Jean-Marc Lasgouttes]

Le 07/12/2016 à 18:49, Guenter Milde a écrit :

The current behaviour of "insert-quote" LFUN is usually called "smart
quotes". This can be done without Quote insets.


Sure. What the Quote inset brings us, as Juergen pointed out already, is 
some semantics. But it is not semantics that we really exploit now.



Only if we want to keep the type of quotes configurable also after
insertion, "dynamic" Quote insets are required.
But even then, the existing "static" Quote insets should be converted to
Unicode in the source files with lyx2lyx.


I am not so sure about that. The example of inner spacein french quotes 
is interesting in this repect. Sure, we can take the lazy approach and 
add inde spaces by hand. This lead french text to be full of ugly grey 
rectangles (the spaces) on screen in word processors. But it is not 
really a quote and a space. In some sense the space is part of the 
quote, like the spacing in math typesetting.


And pardon me for failing to be impressed by the modernity of people 
having unicode keyboards (how many keys does your keyboard have?) and 
selection of a character in a list or 65535 others in some GUI where I 
have to guess what is the category of the character I am looking for. 
Unicode is great, but it is not a religion I want to get enrolled in.


JMarc

Re: problems with quotes

[Guenter Milde]

Dear Jean-Marc,

On 2016-12-08, Jean-Marc Lasgouttes wrote:
> Le 08/12/2016 à 00:09, Guenter Milde a écrit :
>> * consistency: currently, if a user sees a guillemot « on screen, it can be
>>   a literal character or a Quote inset and the LaTeX export can be

>>   "«" or "\guillemotleft" (depending on the "inputencoding")
>>   "<<" (for Quote inset, even if « is supported by the encoding)

> This code is old and shall be audited. But this has nothing to do with 
> keeping or not a quote inset.

Yes and no:

* Consistency in the LaTeX output can be achieved also when keeping the
  Quote inset.

* Other inconsistencies remain or must be overcome with additional
  complexity in the LyX code, e.g.
  
  Edit>Find finds the literal character but not the Quote inset  


OTOH, the current functionality (except for not using literal
characters in the LaTeX output) can be achieved with literal Unicode
characters:

* The "smart" part of the quote-insert function and the \quotes_language
  setting control insertion, not export. Once inserted, a Quote insert is
  as immutable as a literal character.
  
  -> changing the appearance of Quote insets document-wide is harder than
  changing literal quote characters as the regular find/replace does not
  work. 



Replacing the Quote inset would simplify the LyX code without loss of
existing features.

Keeping the Quote inset would allow to add feature requests:

* "dynamic" Quotes where the appearance is determinded by the
  \quotes_language setting at export time.
  
* "configurable" Quotes (with context-menu to modify settings)


Günter

Re: problems with quotes

[Guenter Milde]

Dear Jürgen,

On 2016-12-08, Jürgen Spitzmüller wrote:
> Am Mittwoch, den 07.12.2016, 23:09 + schrieb Guenter Milde:

>> However, given that other non-ASCII characters are exported as-is, too
>> using literal characters for typographical quotes makes the generated
>> document's source more consistent and easier to read.

>> Also:
>> +1 get rid of the dependance on "TeX ligatures" with non-TeX fonts

> I am not sure we do not need these ligatures in other cases.

Nor am I, but it would be a step in the right direction in any case.
IMO, LyX should not actively use these "archaic" conventions intended
for easier input with an american keyboard.



>> > Querying fontenc is not really something problematic.

>> However, if we can do without, the code becomes simpler and less
>> error-prone.

My remark was about LyX code

> Not in all cases. My documents, for instance, will break, since I have
> activated » and « for the use with csquotes, as many csquotes users do:

> \MakeAutoQuote{»}{«}

> So literal guillemot glyphs change in output depending on the context.
> In parallel, I am using the LyX quotes for static quotes, and if I
> enter a guillemot there, I expect it to come out as a guillemet in any
> case, thus it needs to be output as a macro or ligature.

OK. I understand the question about "active chars" now.

You set the "\quotes_language" to french or danish and press " for a
guillemet in the output and » or « for a "csquotes auto quote". 

In LyX, you cannot distinguish one from the other unless you open the
"source pane". For the average user, different behaviour in the output
will be a bad surprise!


>> > I really do not see what we will gain here.

>> * consistency: currently, if a user sees a guillemot « on screen, it
>> can be
>>   a literal character or a Quote inset and the LaTeX export can be 
>>   
>>   "«" or "\guillemotleft" (depending on the "inputencoding")
>>   "<<" (for Quote inset, even if « is supported by the encoding)

> What is crucial for most LyX users is the output, not the intermediate
> LaTeX source.

... unless 

* the differences lead to hard to detect errors in the output like
  csquotes ignoring a quote that looks like the one set in
  \MakeAutoQuote{»}{«}

* users share documents with co-authors working with LaTeX

* users try to find the string "hallo «Welt»" somewhere in the document.

Günter

Re: problems with quotes

[Jean-Marc Lasgouttes]

Le 08/12/2016 à 00:09, Guenter Milde a écrit :

* consistency: currently, if a user sees a guillemot « on screen, it can be
  a literal character or a Quote inset and the LaTeX export can be

  "«" or "\guillemotleft" (depending on the "inputencoding")
  "<<" (for Quote inset, even if « is supported by the encoding)


This code is old and shall be audited. But this has nothing to do with 
keeping or not a quote inset.


JMarc

Re: problems with quotes

[Jürgen Spitzmüller]

Am Mittwoch, den 07.12.2016, 21:14 + schrieb Guenter Milde:
> If quotes come in pairs, it can be implemented similar to brackets.
> (Also for brackets there is a chance of mismatch due to other uses
> like
>   1) in enumerations
>   2) in smilies :-)
> )

Really? Do you have a patch for that? I doubt it will be as easy as
with the inset.

> If quotes don't come in pairs (e.g. the Finnish quotes) it does not
> make
> much sense to highlight matches.

Why? Also Finish has opening and closing quotes (functionally), even if
they are allographs (formally). 
(Which is yet another argument pro inset: the distinction of form and
meaning).


> However, at least in English, a typographical apostrophe and a
> typographical
> single left quotation mark are considered the same *character* (not
> just
> glyph):
> 
> The Unicode standard says:
> 
> 0027  APOSTROPHE
>   * neutral (vertical) glyph with mixed usage
>   * 2019 is preferred for apostrophe
>   * preferred characters in English for paired quotation marks
> are
>     2018 & 2019
> 
> 2018  LEFT SINGLE QUOTATION MARK
>   = single turned comma quotation mark
>   * this is the preferred character (as opposed to 201B)
> 2019  RIGHT SINGLE QUOTATION MARK
>   = single comma quotation mark
>   * this is the preferred character to use for apostrophe

Which shows that apostrophe and single quotation mark _are_ different
glyphs.


> Expectiong the user to inserting this character with different
> keybindings depending on the use case is not user friendly.

Why? There is an apostrophe key and a quotation mark key.

Still not convinced (or even less convinced the more I start pondering
it) ...

Jürgen

signature.asc
Description: This is a digitally signed message part

Re: problems with quotes

[Jürgen Spitzmüller]

Am Mittwoch, den 07.12.2016, 23:09 + schrieb Guenter Milde:
> However, given that other non-ASCII characters are exported as-is,
> too
> using literal characters for typographical quotes makes the generated
> document's source more consistent and easier to read.
> 
> Also:
> +1 get rid of the dependance on "TeX ligatures" with non-TeX fonts

I am not sure we do not need these ligatures in other cases.

> > > +1 no need to query "fontenc": Unicode charaters are replaced by
> > >    lib/unicodesymbols - this already works reliably for all quote
> > > characters
> > >    in question.
> > Querying fontenc is not really something problematic.
> 
> However, if we can do without, the code becomes simpler and less
> error-prone.

Not in all cases. My documents, for instance, will break, since I have
activated » and « for the use with csquotes, as many csquotes users do:

\MakeAutoQuote{»}{«}

So literal guillemot glyphs change in output depending on the context.
In parallel, I am using the LyX quotes for static quotes, and if I
enter a guillemot there, I expect it to come out as a guillemet in any
case, thus it needs to be output as a macro or ligature.

> > I really do not see what we will gain here.
> 
> * consistency: currently, if a user sees a guillemot « on screen, it
> can be
>   a literal character or a Quote inset and the LaTeX export can be 
>   
>   "«" or "\guillemotleft" (depending on the "inputencoding")
>   "<<" (for Quote inset, even if « is supported by the encoding)

What is crucial for most LyX users is the output, not the intermediate
LaTeX source.
 
> * remove code doubling: currently, shortcomings of the OT1 font
> encoding
>   are handled separately for Quote-inset export and Unicode character
> export

This is really just a small code junk.

Jürgen

> I.e. even if we decide to keep the Quotes inset, export as Unicode
> characters makes the code simpler and the exported documents more
> consistent.
> 
> Günter
> 

signature.asc
Description: This is a digitally signed message part

Re: problems with quotes

[Guenter Milde]

On 2016-12-07, Jürgen Spitzmüller wrote:

> [-- Type: text/plain, Encoding: quoted-printable --]

> Am Mittwoch, den 07.12.2016, 17:53 + schrieb Guenter Milde:
>> +1 "the right thing" for non-TeX fonts
>> +1 "the right thing" for HTML (LyX generates utf8-encoded HTML
>> documents)

> Macros/HTML tags are not "the wrong thing".

However, given that other non-ASCII characters are exported as-is, too
using literal characters for typographical quotes makes the generated
document's source more consistent and easier to read.

Also:
+1 get rid of the dependance on "TeX ligatures" with non-TeX fonts


>> +1 no need to query "fontenc": Unicode charaters are replaced by
>>    lib/unicodesymbols - this already works reliably for all quote
>> characters
>>    in question.

> Querying fontenc is not really something problematic.

However, if we can do without, the code becomes simpler and less
error-prone.

> I really do not see what we will gain here.

* consistency: currently, if a user sees a guillemot « on screen, it can be
  a literal character or a Quote inset and the LaTeX export can be 
  
  "«" or "\guillemotleft" (depending on the "inputencoding")
  "<<" (for Quote inset, even if « is supported by the encoding)
  
* remove code doubling: currently, shortcomings of the OT1 font encoding
  are handled separately for Quote-inset export and Unicode character export
  
I.e. even if we decide to keep the Quotes inset, export as Unicode
characters makes the code simpler and the exported documents more consistent.

Günter

Re: problems with quotes

[Jean-Pierre Chrétien]

Le 7 décembre 2016 23:06:11 CET, Guenter Milde  a écrit :
>On 2016-12-07, Jürgen Spitzmüller wrote:
>> Am Mittwoch, den 07.12.2016, 17:28 + schrieb Guenter Milde
>
>> What about extended features such as quote balancing check or
>> automatic nesting (of inner/outer quotes)? 
>
In French, guillemets are not necessarily balanced when paragraphs are quoted.

-- 
Jean-Pierre

Re: problems with quotes

[Guenter Milde]

On 2016-12-07, Jürgen Spitzmüller wrote:
> Am Mittwoch, den 07.12.2016, 17:28 + schrieb Guenter Milde:

>> The only possible problem would be to make sure that the
>> unicodesymbols replacements are done *after* coverting the Quote inset
>> to a quote character.

> Can you assure that the opening/closing quote mechanism will still work
> reliably? 

I cannot assure this for the current mechanism in LyX because I don't
know the details. The common smartquotes mechanism relies on word
boundaries -- independent of Quote insets. 

> What about extended features such as quote balancing check or
> automatic nesting (of inner/outer quotes)? 

While the current functionality can be maintained without Quote insets,
possible future enhancements may gain from the complex handling of quotes
via insets.

For each of them, the advantages and drawbacks of a Quote inset should be 
weighed up against the added complexity of a Quote inset.

I agree that this should be done before taking a decision to remove the
Quote inset framework.

> What if quotation mark glyphs are made active?

No change, as already now literal quote characters are in common use. 
In 8-bit LaTeX, all typographical quote characters are active (as all
non-ASCII characters).

The problematic character is the neutral "ASCII quote" (0022 QUOTATION
MARK). However, handling of this character will not be affected by
superseding Quote insets with literal quote characters:

* »"« is currently not used in Quote-inset export,
* the default binding for the »"« key (the quote-insert LFUN) remains
* it is inset as-is in some cases by the quote-insert LFUN already now.
  (Outside of verbatim context, inserting \textquotedbl{} is preferable.)


Günter

Re: problems with quotes

[Guenter Milde]

On 2016-12-07, Jürgen Spitzmüller wrote:

> [-- Type: text/plain, Encoding: quoted-printable --]

> Am Mittwoch, den 07.12.2016, 19:58 + schrieb Guenter Milde:
>> I am not sure the Quote inset brings benefits for this feature.

> I doubt it is possible to implement it without the inset.

If quotes come in pairs, it can be implemented similar to brackets.
(Also for brackets there is a chance of mismatch due to other uses like
  1) in enumerations
  2) in smilies :-)
)
If quotes don't come in pairs (e.g. the Finnish quotes) it does not make
much sense to highlight matches.

>> +1 it may distinguish single left quote and apostrophe.

>> -1 it may still mix them, when the apostrophe is inserted with the
>>    insert-quote LFUN.

> Which is _not_ an apostrophe (and thus the marking would actually be a
> feature).

However, at least in English, a typographical apostrophe and a typographical
single left quotation mark are considered the same *character* (not just
glyph):

The Unicode standard says:

0027APOSTROPHE
* neutral (vertical) glyph with mixed usage
* 2019 is preferred for apostrophe
* preferred characters in English for paired quotation marks are
  2018 & 2019

2018LEFT SINGLE QUOTATION MARK
= single turned comma quotation mark
* this is the preferred character (as opposed to 201B)
2019RIGHT SINGLE QUOTATION MARK
= single comma quotation mark
* this is the preferred character to use for apostrophe

Expectiong the user to inserting this character with different
keybindings depending on the use case is not user friendly.

The situation is different in German (or Czech, Slovak, Slovenian), where
the preferred "typographical" apostrophe character (2019 RIGHT SINGLE
QUOTATION MARK) differs from both single quotation marks ‚‘ (201A SINGLE
LOW-9 QUOTATION MARK, 2018 LEFT SINGLE QUOTATION MARK).
   
>> -1 strange results for documents with a mix of literal Unicode quotes
>> and
>>    Quote insets

> It would only be supported for quote insets of course.

In case it is only implemented for the quote insets...

>>    Mind, that some autors prefer use of literal Unicode quotes
>> (accessible
>>    e.g.via AltGr+... over LyX's smartquotes keybinding.

> Then they don't get the feature.

... which IMO is an artificial limitation when OTOH supporting brackets.

>>    Also quotes in imported text parts are not converted to insets
>> (and cannot be converted without guessing).

> But this can (should) be changed.

The guessing may be improved but will remain a guessing (see tex2lyx).
This is not a real problem, as long as back-conversion to the original
character is ensured and no other features depend on the correct guess.

Günter

Re: problems with quotes

[Jürgen Spitzmüller]

Am Mittwoch, den 07.12.2016, 19:58 + schrieb Guenter Milde:
> I am not sure the Quote inset brings benefits for this feature.

I doubt it is possible to implement it without the inset.

> +1 it may distinguish single closing quote and apostrophe.
> 
> -1 it may still mix them, when the apostrophe is inserted with the
>    insert-quote LFUN.

Which is _not_ an apostrophe (and thus the marking would actually be a
feature).
  
> -1 strange results for documents with a mix of literal Unicode quotes
> and
>    Quote insets

It would only be supported for quote insets of course.

>    Mind, that some autors prefer use of literal Unicode quotes
> (accessible
>    e.g.via AltGr+... over LyX's smartquotes keybinding.

Then they don't get the feature.

>    Also quotes in imported text parts are not converted to insets
> (and
>    cannot be converted withnout guessing).

But this can (should) be changed.

Jürgen

signature.asc
Description: This is a digitally signed message part

Re: problems with quotes

[Guenter Milde]

On 2016-12-07, Jürgen Spitzmüller wrote:
> Am Mittwoch, den 07.12.2016, 14:02 +0100 schrieb Jean-Marc Lasgouttes:

> What I'd like to see eventually is some kind of quote balancing check,
> such that if you enter a closing quote, the text back to the respective
> opening quote is highlighted. Same for brackets.

As many text editors do... 

I am not sure the Quote inset brings benefits for this feature.

+1 it may distinguish single closing quote and apostrophe.

-1 it may still mix them, when the apostrophe is inserted with the
   insert-quote LFUN.
   
-1 strange results for documents with a mix of literal Unicode quotes and
   Quote insets
 
   Mind, that some autors prefer use of literal Unicode quotes (accessible
   e.g.via AltGr+... over LyX's smartquotes keybinding.
   Also quotes in imported text parts are not converted to insets (and
   cannot be converted withnout guessing).
   
Günter   

Re: problems with quotes

[Jürgen Spitzmüller]

Am Mittwoch, den 07.12.2016, 17:49 + schrieb Guenter Milde:
> > What bugs would remain once this switch to unicode is done in the
> > inset? 
> > Except of course that the code is still complicated.
> 
> Data loss when converting to a "listings" or ERT or URL inset.
> http://www.lyx.org/trac/ticket/7309

Hm. Looking at this bug I think this can only be properly solved _with_
the quote inset (which can output the "correct" quoation mark in the
different contexts, i.e. listings vs. ERT vs. other verbatim insets.

Jürgen

signature.asc
Description: This is a digitally signed message part

Re: problems with quotes

[Jürgen Spitzmüller]

Am Mittwoch, den 07.12.2016, 17:28 + schrieb Guenter Milde:
> The only possible problem would be to make sure that the
> unicodesymbols
> replacements are done *after* coverting the Quote inset to a quote
> character.

Can you assure that the opening/closing quote mechanism will still work
reliably? What about extended features such as quote balancing check or
automatic nesting (of inner/outer quotes)? What if quotation mark
glyphs are made active?

Jürgen

signature.asc
Description: This is a digitally signed message part

Re: problems with quotes

[Jürgen Spitzmüller]

Am Mittwoch, den 07.12.2016, 17:53 + schrieb Guenter Milde:
> +1 "the right thing" for non-TeX fonts
> +1 "the right thing" for HTML (LyX generates utf8-encoded HTML
> documents)

Macros/HTML tags are not "the wrong thing".

> +1 no need to query "fontenc": Unicode charaters are replaced by
>    lib/unicodesymbols - this already works reliably for all quote
> characters
>    in question.

Querying fontenc is not really something problematic.

I really do not see what we will gain here.

Jürgen

signature.asc
Description: This is a digitally signed message part

Re: problems with quotes

[Guenter Milde]

On 2016-12-07, Jürgen Spitzmüller wrote:
> 2016-12-07 13:27 GMT+01:00 Guenter Milde :

>> * Adjoining closing Single + double quote becomes double + single quote
>>   (for English, Swedish and German, LaTeX export as ''').

>> * French double quotes are converted to << >> in the LaTeX file and to
>>   double inverted question/interrogation marks in the output, if the font
>>   encoding is set to [None] or OT1 but the global default is T1. (test
>>   for lyxrc.fontenc instead of the document-specific fontenc setting in
>>   InsetQuotes.cpp).

>> * Quote type ignored for LyXHTML: always "English" quotes used.


> Fixing these issues in the inset was trivial, so I went ahead and did that
> (in master).

Even without getting rid of the inset altogether, a better fix would be to
convert it to Unicode (not LaTeX ligatures, not LaTeX macros, not HTML
entities). 

+1 "the right thing" for non-TeX fonts
+1 "the right thing" for HTML (LyX generates utf8-encoded HTML documents)
+1 no need to query "fontenc": Unicode charaters are replaced by
   lib/unicodesymbols - this already works reliably for all quote characters
   in question.
   
Günter   

Re: problems with quotes

[Guenter Milde]

On 2016-12-07, Jean-Marc Lasgouttes wrote:
> Le 07/12/2016 à 13:27, Guenter Milde a écrit :

...
>> The Quotes inset shows a number of problems that could easily be solved
>> by using literal Unicode characters instead

>> Most important:

>> * Adjoining closing Single + double quote becomes double + single quote
>>   (for English, Swedish and German, LaTeX export as ''').

>> * French double quotes are converted to << >> in the LaTeX file and to
>>   double inverted question/interrogation marks in the output, if the font
>>   encoding is set to [None] or OT1 but the global default is T1. (test
>>   for lyxrc.fontenc instead of the document-specific fontenc setting in
>>   InsetQuotes.cpp).

> Would these two problems be solved just by making the quote inset always 
> use unicode on output? 

Yes.

In addition, it would automatically also solve the HTML output (we use
utf8-encoded characters anyway, so this would be the better choice for
quotes as well) and the output for non-TeX fonts: Uses TeX-Ligatures and
LICR macros instead of Unicode characters.

> The use of unicode is orthogonal to the use of a 
> quote inset, right?

Not fully orthogonal:

* use of Unicode can be combined with Quote insets (i.e. the insets
  are converted to Unicode characters at export), or
  
* use of Unicode can replace the Quote insets (i.e. the "insert quote" LFUN
  directly inserts Unicode characters).

> I agree that currently the quote inset is not very helpful (only french 
> spacing is important here). If however some smart quote handling gets 
> introduced, then the inset will be mandatory.

The current behaviour of "insert-quote" LFUN is usually called "smart
quotes". This can be done without Quote insets.

Only if we want to keep the type of quotes configurable also after
insertion, "dynamic" Quote insets are required.
But even then, the existing "static" Quote insets should be converted to
Unicode in the source files with lyx2lyx.

> What bugs would remain once this switch to unicode is done in the inset? 
> Except of course that the code is still complicated.

Data loss when converting to a "listings" or ERT or URL inset.
http://www.lyx.org/trac/ticket/7309

Günter

Re: problems with quotes

[Guenter Milde]

On 2016-12-07, Jürgen Spitzmüller wrote:

> [-- Type: text/plain, Encoding: quoted-printable --]

> 2016-12-07 13:27 GMT+01:00 Guenter Milde:

>> Suggestion
>> ==

>> Let the “quote” LFUN insert literal Unicode characters


> Sound like a good idea (but probably needs good testing).

The only possible problem would be to make sure that the unicodesymbols
replacements are done *after* coverting the Quote inset to a quote character.

>> Use the Quotes inset for "dynamic quotes" (the remaining
>> part of #1499) and Unicode for "static quotes":

>  especially if this is also implemented.

Well. Especially if this is *not* implemented, we could make LyX simpler
without loss of functionality!

Günter

Re: problems with quotes

[Jürgen Spitzmüller]

Am Mittwoch, den 07.12.2016, 14:02 +0100 schrieb Jean-Marc Lasgouttes:
> I agree that currently the quote inset is not very helpful (only
> franch 
> spacing is important here). If however some smart quote handling
> gets 
> introduced, then the inset will be mandatory.

What I'd like to see eventually is some kind of quote balancing check,
such that if you enter a closing quote, the text back to the respective
opening quote is highlighted. Same for brackets.

Jürgen

signature.asc
Description: This is a digitally signed message part

Re: problems with quotes

[Jürgen Spitzmüller]

2016-12-07 13:27 GMT+01:00 Guenter Milde :

> * Adjoining closing Single + double quote becomes double + single quote
>   (for English, Swedish and German, LaTeX export as ''').
>
> * French double quotes are converted to << >> in the LaTeX file and to
>   double inverted question/interrogation marks in the output, if the font
>   encoding is set to [None] or OT1 but the global default is T1. (test
>   for lyxrc.fontenc instead of the document-specific fontenc setting in
>   InsetQuotes.cpp).
>
> * Quote type ignored for LyXHTML: always "English" quotes used.
>

Fixing these issues in the inset was trivial, so I went ahead and did that
(in master).

Jürgen

Re: problems with quotes

[Jean-Marc Lasgouttes]

Le 07/12/2016 à 13:27, Guenter Milde a écrit :

Dear LyX developers,

I am asking for comments and suggestions regarding Ticket #10451:


The Quotes inset shows a number of problems that could easily be solved
by using literal Unicode characters instead

Most important:

* Adjoining closing Single + double quote becomes double + single quote
  (for English, Swedish and German, LaTeX export as ''').

* French double quotes are converted to << >> in the LaTeX file and to
  double inverted question/interrogation marks in the output, if the font
  encoding is set to [None] or OT1 but the global default is T1. (test
  for lyxrc.fontenc instead of the document-specific fontenc setting in
  InsetQuotes.cpp).


Would these two problems be solved just by making the quote inset always 
use unicode on output? The use of unicode is orthogonal to the use of a 
quote inset, right?


I agree that currently the quote inset is not very helpful (only franch 
spacing is important here). If however some smart quote handling gets 
introduced, then the inset will be mandatory.


What bugs would remain once this switch to unicode is done in the inset? 
Except of course that the code is still complicated.



* Quote type ignored for LyXHTML: always "English" quotes used.


This should be easy/trivial to fix.

JMarc

Re: problems with quotes

[Jürgen Spitzmüller]

2016-12-07 13:27 GMT+01:00 Guenter Milde:

> Suggestion
> ==
>
> Let the “quote” LFUN insert literal Unicode characters
>

Sound like a good idea (but probably needs good testing).


> Use the Quotes inset for "dynamic quotes" (the remaining
> part of #1499) and Unicode for "static quotes":
>

 especially if this is also implemented.

Jürgen