Bugs item #2925674, was opened at 2010-01-04 10:37
Message generated for change (Comment added) made by mbucc
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=2925674&group_id=56967
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: SQL/Core
Group: Clients CVS Head
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Mark Bucciarelli (mbucc)
Assigned to: Niels Nes (nielsnes)
Summary: mclient dump fails on UTF-8 char and gives no error
Initial Comment:
This is the Nov. 09 branch, built from CVS.
The mclient -D option stops dead when it hits a field with a UTF-8 character.
In addition, the return value to the shell is zero, indicating no error.
See attached test files that create this scenario on my machine.
Note that I can run
$ cat test6.sql
just fine, so it's not something related to the locale setting for my
terminal.
(Furthermore, I believe if piping to stdout to a file, it shouldn't even
matter what the locale setting for the terminal is.)
This is a really bad one--if you are using dump for backups mclient
gives no indication that there was an error.
----------------------------------------------------------------------
Comment By: Mark Bucciarelli (mbucc)
Date: 2010-01-04 15:03
Message:
Thank you for changing mclient error value!
However, I disagree that mclient is behaving properly.
For example, what if I want to pipe the mclient output to a program that
understands UTF-8?
My terminal locale is completely irrelevant in this case.
----------------------------------------------------------------------
Comment By: Sjoerd Mullender (sjoerd)
Date: 2010-01-04 14:52
Message:
I agree that the error code of mclient is not right. I just fixed that in
the code.
mclient uses the LC_CTYPE value to determine the encoding of the terminal.
In your case that is "C", so that means that you (implicitly) told mclient
that it can only write ASCII data. Converting the e with diaeresis to
ASCII is not possible, so mclient bails.
If your terminal can do more than ASCII, you should set your locale to
reflect that and specify LC_CTYPE=en_US.UTF-8 in your environment.
Mclient is doing the right thing in this respect (apart from the error
code which I fixed).
----------------------------------------------------------------------
Comment By: Mark Bucciarelli (mbucc)
Date: 2010-01-04 14:14
Message:
Also, I think mclient in this case must exit with
a value of 1 not 0. (I don't care so much about
the text of the error message).
Otherwise, there is no way to tell that the
dump failed, which is really bad for a
production system with automated backups.
I'll try and work up a patch.
----------------------------------------------------------------------
Comment By: Mark Bucciarelli (mbucc)
Date: 2010-01-04 14:08
Message:
I can cat the test6.sql just fine so my terminal handles UTF8.
And the attached test script pipes the output to a file--shouldn't that
always work irrespective of my terminal's local is? I'll pretty sure you
can do something like
$ cat file_with_utf8.txt > another_file.txt
even with a locale of C.
To answer your question:
$locale
LANG=en_US.UTF-8
LC_CTYPE="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_COLLATE="C"
LC_MONETARY="C"
LC_MESSAGES="C"
LC_PAPER="C"
LC_NAME="C"
LC_ADDRESS="C"
LC_TELEPHONE="C"
LC_MEASUREMENT="C"
LC_IDENTIFICATION="C"
LC_ALL=C
$
----------------------------------------------------------------------
Comment By: Sjoerd Mullender (sjoerd)
Date: 2010-01-04 13:58
Message:
What is your locale? Give the command
locale
in a shell and post the result.
mclient uses your locale to determine how to translate the UTF-8
characters it gets from the server to whatever your terminal window uses.
If your locale is e.g. C, then there is no way to translate the "e with
diaeresis" in the table, so mclient complains about that (the write error
you reported). I admit, the error message could be improved, but there is
nothing else mclient can do.
If your locale settings are indeed C (or Posix), then you can tell mclient
which encoding to use to overrule this by using the -E (--encoding) option.
You can, for example, use
mclient -Eutf-8 -lsql ...
if you know your terminal window can handle UTF-8.
----------------------------------------------------------------------
Comment By: Mark Bucciarelli (mbucc)
Date: 2010-01-04 12:20
Message:
Sorry, this was against an old (two weeks?) checkout of CVS head.
When I ran the test files against 2009_Nov, the test passed. I'll leave
it open
in case it really is an issue for HEAD.
----------------------------------------------------------------------
Comment By: Mark Bucciarelli (mbucc)
Date: 2010-01-04 10:39
Message:
If you run inside mclient interactive shell, it also fails:
sql>select * from city;
+------+------+------+------+------------------------+------------------------+------+------+------+
| id | name | regi | int1 | lat | long
| int2 | int3 | int4 |
: : : on : : :
: : : :
+======+======+======+======+========================+========================+======+======+======+
| 198 | Gjak | Euro | 0 | 20.430799484252901 |
42.380298614502003 | 0 | 0 | null |
: :1 tuple
write error
sql>\D
START TRANSACTION;
SET SCHEMA "test";
CREATE TABLE "test"."city" (
"id" int,
"name" varchar(100),
"region" varchar(100),
"int1" int,
"lat" double,
"long" double,
"int2" int,
"int3" int,
"int4" int
);
COPY 1 RECORDS INTO "test"."city" FROM stdin USING DELIMITERS
'\t','\n','"';
198 "Gjakovsql>
----------------------------------------------------------------------
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=2925674&group_id=56967
------------------------------------------------------------------------------
This SF.Net email is sponsored by the Verizon Developer Community
Take advantage of Verizon's best-in-class app development support
A streamlined, 14 day to market process makes app distribution fast and easy
Join now and get one step closer to millions of Verizon customers
http://p.sf.net/sfu/verizon-dev2dev
_______________________________________________
Monetdb-bugs mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/monetdb-bugs
