On Thu, Feb 07, 2013 at 12:45:04PM +0100, Werner Fink wrote:
> On Thu, Feb 07, 2013 at 12:18:31PM +0100, Irek Szczesniak wrote:
> > On Thu, Feb 7, 2013 at 12:16 PM, Dr. Werner Fink <[email protected]> wrote:
> > > Hi,
> > >
> > > just to say the change
> > >
> > > 11-05-27 A bug with command substitution with the shift jis locale has
> > > been
> > > fixed.
> > >
> > > has been removed from ksh93u+ 2012-08-01 as well as from ksh93v found in
> > > the
> > > git repository. Please re-add the change as e.g. found in the attachment.
> > > Maybe together some more comments to avoid such drops. Even if the code
> > > works
> > > for e.g. UTF-8 ... Shift-JIS is special as it includes ASCII characters in
> > > its multi byte encoding. Without this patch my own regression check will
> > > trow out errors (see second attachment).
> >
> > Didn't the patch cause problems with some kind of command
> > substitutions which included multibyte characters? I recall Roland and
> > David running into such problems with the XML parsing scripts.
>
> Hmmm ... then it would be better to extend this patch. I've a lot of
> Japaneese users with old ksh scripts mostly written in Shift-JIS encoding.
Just tried to detect all multi byte locales which are not UTF-8 with
the attached patch. Don't know if this does it better with XML.
Nevertheless the balance to next buffer area with Shift-JIS text
should not break the multi byte characters.
Werner
--
"Having a smoking section in a restaurant is like having
a peeing section in a swimming pool." -- Edward Burr
--- src/cmd/ksh93/sh/macro.c
+++ src/cmd/ksh93/sh/macro.c 2013-02-07 13:26:07.897952986 +0000
@@ -54,6 +54,7 @@
#if SHOPT_MULTIBYTE
# undef isascii
# define isacii(c) ((c)<=UCHAR_MAX)
+# include <lc.h>
#else
# define mbchar(p) (*(unsigned char*)p++)
#endif /* SHOPT_MULTIBYTE */
@@ -2026,6 +2027,10 @@ static void comsubst(Mac_t *mp,register
struct _mac_ savemac;
int savtop = stktell(stkp);
char lastc=0, *savptr = stkfreeze(stkp,0);
+#if SHOPT_MULTIBYTE
+ const Lc_t *lc=lcinfo(LC_CTYPE)->lc;
+ wchar_t lastw=0;
+#endif /* SHOPT_MULTIBYTE */
int was_history = sh_isstate(SH_HISTORY);
int was_verbose = sh_isstate(SH_VERBOSE);
int was_interactive = sh_isstate(SH_INTERACTIVE);
@@ -2209,6 +2214,17 @@ static void comsubst(Mac_t *mp,register
}
else if(lastc)
{
+#if SHOPT_MULTIBYTE
+ if(lastw)
+ {
+ int n;
+ char mb[8];
+ n = mbconv(mb, lastw);
+ mac_copy(mp,mb,n);
+ lastw = 0;
+ }
+ else
+#endif /* SHOPT_MULTIBYTE */
mac_copy(mp,&lastc,1);
lastc = 0;
}
@@ -2220,6 +2236,17 @@ static void comsubst(Mac_t *mp,register
ssize_t len = 1;
/* can't write past buffer so save last character */
+#if SHOPT_MULTIBYTE
+ if ((len = mbsize(str))>1 && !(lc->flags & LC_utf8))
+ {
+ len = mb2wc(lastw,str,len);
+ if (len < 0)
+ {
+ lastw = 0;
+ len = 1;
+ }
+ }
+#endif /* SHOPT_MULTIBYTE */
c -= len;
lastc = str[c];
str[c] = 0;
@@ -2240,6 +2267,17 @@ static void comsubst(Mac_t *mp,register
}
if(lastc)
{
+#if SHOPT_MULTIBYTE
+ if(lastw)
+ {
+ int n;
+ char mb[8];
+ n = mbconv(mb, lastw);
+ mac_copy(mp,mb,n);
+ lastw = 0;
+ }
+ else
+#endif /* SHOPT_MULTIBYTE */
mac_copy(mp,&lastc,1);
lastc = 0;
}
_______________________________________________
ast-developers mailing list
[email protected]
http://lists.research.att.com/mailman/listinfo/ast-developers
