Re: svn commit: r362148 - head/contrib/nvi/common

2020-06-22 Thread Yuri Pankov

Gleb Smirnoff wrote:

   Yuri, Zhihao,

this commit totally broke Russian input for me in nvi. After
exiting edit mode, nvi immediately converts all text to ???.

I don't have any special settings in my environment. All I have
is "russian" class for my user which yields in these environment
variables:

declare -x LANG="ru_RU.UTF-8"
declare -x MM_CHARSET="UTF-8"
declare -x XTERM_LOCALE="ru_RU.UTF-8"

I'm already digging into that problem, but may be you have
a clue immediately.


My bad, yes, I see the problem, looking into it.


On Sat, Jun 13, 2020 at 02:11:02PM +, Yuri Pankov wrote:
Y> Author: yuripv
Y> Date: Sat Jun 13 14:11:02 2020
Y> New Revision: 362148
Y> URL: https://svnweb.freebsd.org/changeset/base/362148
Y>
Y> Log:
Y>   nvi: fallback to ISO8859-1 as last resort
Y>
Y>   Current logic of using user's locale encoding that is UTF-8 doesn't make
Y>   much sense if we already failed the looks_utf8() check and skipped
Y>   encoding set using "fileencoding" as being UTF-8 as well; fallback to
Y>   ISO8859-1 in that case.
Y>
Y>   Reviewed by:Zhihao Yuan 
Y>   Differential Revision:  https://reviews.freebsd.org/D24919
Y>
Y> Modified:
Y>   head/contrib/nvi/common/exf.c
Y>
Y> Modified: head/contrib/nvi/common/exf.c
Y> 
==
Y> --- head/contrib/nvi/common/exf.c Sat Jun 13 09:16:07 2020(r362147)
Y> +++ head/contrib/nvi/common/exf.c Sat Jun 13 14:11:02 2020(r362148)
Y> @@ -1237,7 +1237,10 @@ file_encinit(SCR *sp)
Y>   }
Y>
Y>   /*
Y> -  * Detect UTF-8 and fallback to the locale/preset encoding.
Y> +  * 1. Check for valid UTF-8.
Y> +  * 2. Check if fallback fileencoding is set and is NOT UTF-8.
Y> +  * 3. Check if user locale's encoding is NOT UTF-8.
Y> +  * 4. Use ISO8859-1 as last resort.
Y>*
Y>* XXX
Y>* A manually set O_FILEENCODING indicates the "fallback
Y> @@ -1246,9 +1249,13 @@ file_encinit(SCR *sp)
Y>*/
Y>   if (looks_utf8(buf, blen) > 1)
Y>   o_set(sp, O_FILEENCODING, OS_STRDUP, "utf-8", 0);
Y> - else if (!O_ISSET(sp, O_FILEENCODING) ||
Y> - !strcasecmp(O_STR(sp, O_FILEENCODING), "utf-8"))
Y> + else if (O_ISSET(sp, O_FILEENCODING) &&
Y> + strcasecmp(O_STR(sp, O_FILEENCODING), "utf-8") != 0)
Y> + /* Use fileencoding as is */ ;
Y> + else if (strcasecmp(codeset(), "utf-8") != 0)
Y>   o_set(sp, O_FILEENCODING, OS_STRDUP, codeset(), 0);
Y> + else
Y> + o_set(sp, O_FILEENCODING, OS_STRDUP, "iso8859-1", 0);
Y>
Y>   conv_enc(sp, O_FILEENCODING, 0);
Y>  #endif

___
svn-src-all@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"


Re: svn commit: r362148 - head/contrib/nvi/common

2020-06-22 Thread Gleb Smirnoff
  Yuri, Zhihao,

this commit totally broke Russian input for me in nvi. After
exiting edit mode, nvi immediately converts all text to ???.

I don't have any special settings in my environment. All I have
is "russian" class for my user which yields in these environment
variables:

declare -x LANG="ru_RU.UTF-8"
declare -x MM_CHARSET="UTF-8"
declare -x XTERM_LOCALE="ru_RU.UTF-8"

I'm already digging into that problem, but may be you have
a clue immediately.

On Sat, Jun 13, 2020 at 02:11:02PM +, Yuri Pankov wrote:
Y> Author: yuripv
Y> Date: Sat Jun 13 14:11:02 2020
Y> New Revision: 362148
Y> URL: https://svnweb.freebsd.org/changeset/base/362148
Y> 
Y> Log:
Y>   nvi: fallback to ISO8859-1 as last resort
Y>   
Y>   Current logic of using user's locale encoding that is UTF-8 doesn't make
Y>   much sense if we already failed the looks_utf8() check and skipped
Y>   encoding set using "fileencoding" as being UTF-8 as well; fallback to
Y>   ISO8859-1 in that case.
Y>   
Y>   Reviewed by:   Zhihao Yuan 
Y>   Differential Revision: https://reviews.freebsd.org/D24919
Y> 
Y> Modified:
Y>   head/contrib/nvi/common/exf.c
Y> 
Y> Modified: head/contrib/nvi/common/exf.c
Y> 
==
Y> --- head/contrib/nvi/common/exf.cSat Jun 13 09:16:07 2020
(r362147)
Y> +++ head/contrib/nvi/common/exf.cSat Jun 13 14:11:02 2020
(r362148)
Y> @@ -1237,7 +1237,10 @@ file_encinit(SCR *sp)
Y>  }
Y>  
Y>  /*
Y> - * Detect UTF-8 and fallback to the locale/preset encoding.
Y> + * 1. Check for valid UTF-8.
Y> + * 2. Check if fallback fileencoding is set and is NOT UTF-8.
Y> + * 3. Check if user locale's encoding is NOT UTF-8.
Y> + * 4. Use ISO8859-1 as last resort.
Y>   *
Y>   * XXX
Y>   * A manually set O_FILEENCODING indicates the "fallback
Y> @@ -1246,9 +1249,13 @@ file_encinit(SCR *sp)
Y>   */
Y>  if (looks_utf8(buf, blen) > 1)
Y>  o_set(sp, O_FILEENCODING, OS_STRDUP, "utf-8", 0);
Y> -else if (!O_ISSET(sp, O_FILEENCODING) ||
Y> -!strcasecmp(O_STR(sp, O_FILEENCODING), "utf-8"))
Y> +else if (O_ISSET(sp, O_FILEENCODING) &&
Y> +strcasecmp(O_STR(sp, O_FILEENCODING), "utf-8") != 0)
Y> +/* Use fileencoding as is */ ;
Y> +else if (strcasecmp(codeset(), "utf-8") != 0)
Y>  o_set(sp, O_FILEENCODING, OS_STRDUP, codeset(), 0);
Y> +else
Y> +o_set(sp, O_FILEENCODING, OS_STRDUP, "iso8859-1", 0);
Y>  
Y>  conv_enc(sp, O_FILEENCODING, 0);
Y>  #endif
Y> ___
Y> svn-src-all@freebsd.org mailing list
Y> https://lists.freebsd.org/mailman/listinfo/svn-src-all
Y> To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"

-- 
Gleb Smirnoff
___
svn-src-all@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"