That is:
- libedit has a wchar_t* buffer (el->el_line.buffer) and el_line calls
ct_encode_string to convert it to a char*.
- ct_encode_string calls wctomb which it expects to make UTF-8 but in
fact because setlocale has not been called it outputs ASCII.
- el_line then uses ct_enc_width which assumes UTF-8 and returns 2. So
the offset is adjusted by 2 even though only 1 byte was filled in.
- ftp obviously isn't happy about having a position after a \0, so it
goes boom.
The setlocale() change below will only fix the problem if LC_CTYPE or
LC_ALL is set to UTF-8. ftp still cores if pasting UTF-8 in C locale.
I think the right fix is for libedit to use the return value of wctomb
to adjust the offset rather than assuming UTF-8 and working out the
width itself.
Perhaps something like this (very lightly tested):
Index: chartype.c
===================================================================
RCS file: /cvs/src/lib/libedit/chartype.c,v
retrieving revision 1.4
diff -u -p -r1.4 chartype.c
--- chartype.c 17 Nov 2011 20:14:24 -0000 1.4
+++ chartype.c 31 Oct 2012 00:13:12 -0000
@@ -44,6 +44,8 @@
#define CT_BUFSIZ 1024
#ifdef WIDECHAR
+protected ssize_t ct_encode_char1(char *, size_t, Char);
+
protected void
ct_conv_buff_resize(ct_buffer_t *conv, size_t mincsize, size_t minwsize)
{
@@ -178,27 +180,25 @@ ct_decode_argv(int argc, const char *arg
protected size_t
ct_enc_width(Char c)
{
- /* UTF-8 encoding specific values */
- if (c < 0x80)
- return 1;
- else if (c < 0x0800)
- return 2;
- else if (c < 0x10000)
- return 3;
- else if (c < 0x110000)
- return 4;
- else
- return 0; /* not a valid codepoint */
+ char s[MB_CUR_MAX];
+
+ return ct_encode_char1(s, sizeof s, c);
}
protected ssize_t
ct_encode_char(char *dst, size_t len, Char c)
{
- ssize_t l = 0;
if (len < ct_enc_width(c))
return -1;
- l = ct_wctomb(dst, c);
+ return ct_encode_char1(dst, len, c);
+}
+protected ssize_t
+ct_encode_char1(char *dst, size_t len, Char c)
+{
+ ssize_t l = 0;
+
+ l = ct_wctomb(dst, c);
if (l < 0) {
ct_wctomb_reset;
l = 0;
On Tue, Oct 30, 2012 at 11:56:18PM +0000, Nicholas Marriott wrote:
> Hi
>
> The buffer isn't zero-terminated, it's the result of calling wctomb to
> convert the internal wchar_t* that libedit has into a char*.
>
> libedit works out the offset in el_line with ct_enc_width which rather
> foolishly makes the assumption that wctomb will convert to UTF-8, but
> ftp doesn't call setlocale so it just leaves it as ASCII.
>
> Try this:
>
> Index: main.c
> ===================================================================
> RCS file: /cvs/src/usr.bin/ftp/main.c,v
> retrieving revision 1.85
> diff -u -p -r1.85 main.c
> --- main.c 26 Aug 2012 02:16:02 -0000 1.85
> +++ main.c 30 Oct 2012 23:52:34 -0000
> @@ -67,6 +67,7 @@
>
> #include <ctype.h>
> #include <err.h>
> +#include <locale.h>
> #include <netdb.h>
> #include <pwd.h>
> #include <stdio.h>
> @@ -90,6 +91,8 @@ main(volatile int argc, char *argv[])
> char *outfile = NULL;
> const char *errstr;
> int dumb_terminal = 0;
> +
> + setlocale(LC_CTYPE, "");
>
> ftpport = "ftp";
> httpport = "http";
>
>
>
>
>
> On Tue, Oct 30, 2012 at 10:31:16PM +0100, Otto Moerbeek wrote:
> > On Tue, Oct 30, 2012 at 10:17:12PM +0100, Otto Moerbeek wrote:
> >
> > > On Tue, Oct 30, 2012 at 08:59:27PM +0100, Juan Francisco Cantero Hurtado
> > > wrote:
> > >
> > > > On Tue, Oct 30, 2012 at 09:31:58AM +0100, Otto Moerbeek wrote:
> > > > > On Mon, Oct 29, 2012 at 06:43:13PM +0100, Juan Francisco Cantero
> > > > > Hurtado wrote:
> > > > >
> > > > > > Chris Cappuccio sent me a mail saying he can't see the characters,
> > > > > > only
> > > > > > a question mark.
> > > > > >
> > > > > > I'm linking each character to their wikipedia page, so you can
> > > > > > copy-paste the character.
> > > > > >
> > > > > > On Thu, Oct 25, 2012 at 05:07:34AM +0200, Juan Francisco Cantero
> > > > > > Hurtado wrote:
> > > > > > > This afternoon I was downloading a tarball from a OpenBSD mirror.
> > > > > > > I
> > > > > > > press the key "?" and after the tab key. ftp crashed with a
> > > > > > > segfault.
> > > > >
> > > > > Please also include your environment settings. It is likely locale
> > > > > plays a role here.
> > > > >
> > > > > At least env | grep LC
> > > > >
> > > >
> > > > I've tried the bug in amd64 without locales and also with
> > > > LC_TIME="es_ES.ISO8859-1" LC_CTYPE="en_US.UTF-8".
> > > >
> > > > The i386 system was a clean installation in a virtual machine.
> > >
> > > I can now reproduce using a terminal that accepts more than just low
> > > ascii.
> > >
> > > What I see is that when complete() is called the cursor position in
> > > the EditLine struct is not what it is supposed to be, it points a
> > > couple of bytes beyond the terminating NUL while it is supposed to
> > > point to the NUL. That causes confusing in the scanner, getting the
> > > argument list count wrong.
> >
> > Ehh, the buffer is not NUL terminated, but observation still holds:
> > the cursor position is a couple of bytes further than it
> > should be.
> >
> > >
> > > The root of the problem seems to be inside the editline lib.
> > >
> > > Cc:ing nicm@, maybe he has a clue
> > >
> > > -Otto
> > >
> > >
> > > >
> > > > >
> > > > > > https://en.wikipedia.org/wiki/%C2%BA
> > > > > > >
> > > > > > > Steps for reproduce:
> > > > > > > # ftp ftp.fr.openbsd.org
> > > > > > > user and password
> > > > > > > ascii art
> > > > > > > ftp> cd pub/Open? <- Here press the tab key
> > > > > > https://en.wikipedia.org/wiki/%C2%BA
> > > > > > > segmentation fault (core dumped) ftp ftp.fr.openbsd.org
> > > > > > >
> > > > > > > It also crashes with the letter "?" and "?".
> > > > > > https://en.wikipedia.org/wiki/%C3%81
> > > > > > https://en.wikipedia.org/wiki/%C3%91
> > > > > > >
> > > > > > > Tested in:
> > > > > > > - A snapshot from yesterday. i386. root account. console/ksh
> > > > > > > without
> > > > > > > locales.
> > > > > > > - A snapshot from a few days ago. amd64. user. urxvt/zsh with utf8
> > > > > > > locales.
> > > > > > >
> > > > > > > I also tested the bug in a remote session with OpenBSD 4.7 and
> > > > > > > ftp works
> > > > > > > without problems.
> > > > > > >
> > > > > > > I've updated the code of usr.bin/ftp to 2012-10-01 and 2012-01-01
> > > > > > > and
> > > > > > > tried both versions. ftp also crashes.
> > > > > > >
> > > > > > > Backtrace:
> > > > > > > Thread 1 (process 3436):
> > > > > > > #0 memcpy (dst0=0x9d4160, src0=Variable "src0" is not available.
> > > > > > > ) at /usr/src/lib/libc/string/bcopy.c:115
> > > > > > > #1 0x000000000040432b in complete (el=Variable "el" is not
> > > > > > > available.
> > > > > > > ) at /usr/src/usr.bin/ftp/complete.c:313
> > > > > > > #2 0x000000000041eb84 in el_wgets (el=0x20da64800,
> > > > > > > nread=0x7f7ffffe3ebc) at read.c:612
> > > > > > > #3 0x000000000041ef8d in el_gets (el=0x20da64800, nread=Variable
> > > > > > > "nread" is not available.
> > > > > > > ) at eln.c:78
> > > > > > > #4 0x000000000040e55f in cmdscanner (top=Variable "top" is not
> > > > > > > available.
> > > > > > > ) at /usr/src/usr.bin/ftp/main.c:465
> > > > > > > #5 0x000000000040eb7c in main (argc=1, argv=0x7f7ffffe4398) at
> > > > > > > /usr/src/usr.bin/ftp/main.c:369
> > > > > > >
> > > > > > > Let me know if it's necessary more info or whatever :)
> > > > > > >
> > > > > > > Cheers.
> > > > > > >
> > > > > >
> > > >
> > > > --
> > > > Juan Francisco Cantero Hurtado http://juanfra.info
