[patch] glib20, UTF-8 and string collation
I have seen recent commit WRT string collation in devel/glib20 by
marcus, so I have decided to check if there is an interest to fix SEGV
in g_utf8_collate when it is given 8-bit non-UTF-8 string(s) to collate.
Good (but by no means only) example of this would be using Evolution to
open mailbox with the mix of KOI-8, CP1251 and UTF-8 message subjects
and order them by the subject. Admittedly, I do not know whether there
are special symbols that trigger the situation or any mix would do. vova
at fbsd ru posted test case mailbox under the link below.
Full discussion including my first approach to fix this problem could be
found here
http://bugzilla.gnome.org/show_bug.cgi?id=492389
Slightly different approach is attached to this E-mail.
Without either patch, my Evolution will core dump on start-up.
First patch was rejected by gnome folks with the recommendation Don't
do that, which, unfortunately, is not that easy to follow ;)
Any comments from people with the knowledge of gnome, UTF-8 and string
collation will be greatly appreciated.
I am not subscribed to ports@, please, make sure to keep me on CC list
when replying.
--
Alexandre Sunny Kovalenko
--- glib/gunidecomp.c.BAK 2008-01-09 09:07:46.0 -0500
+++ glib/gunidecomp.c 2008-01-09 09:17:04.0 -0500
@@ -22,6 +22,7 @@
#include config.h
#include stdlib.h
+#include string.h
#include glib.h
#include gunidecomp.h
@@ -528,6 +529,14 @@
result = g_ucs4_to_utf8 (result_wc, -1, NULL, NULL, NULL);
g_free (result_wc);
+ // Upstream callers rely on the returned pointer to be valid
+ // and produce core if it does not (witness collation routine).
+#define NOT_VALID_UTF8_STRING *** This was not a valid UTF-8 string ***
+ if(!result)
+ {
+result = g_malloc(strlen(NOT_VALID_UTF8_STRING) + 1);
+strcpy(result, NOT_VALID_UTF8_STRING);
+ }
return result;
}
___
freebsd-ports@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: [patch] glib20, UTF-8 and string collation
On Wed, 2008-01-09 at 10:53 -0500, Alexandre Sunny Kovalenko wrote: I have seen recent commit WRT string collation in devel/glib20 by marcus, so I have decided to check if there is an interest to fix SEGV in g_utf8_collate when it is given 8-bit non-UTF-8 string(s) to collate. Any commits I have made in the area of UTF-8 are completely accidental. I am not the UTF-8 guy. Both bland and jylefort have expressed interest in this. Perhaps one of them will comment. Good (but by no means only) example of this would be using Evolution to open mailbox with the mix of KOI-8, CP1251 and UTF-8 message subjects and order them by the subject. Admittedly, I do not know whether there are special symbols that trigger the situation or any mix would do. vova at fbsd ru posted test case mailbox under the link below. Full discussion including my first approach to fix this problem could be found here http://bugzilla.gnome.org/show_bug.cgi?id=492389 Slightly different approach is attached to this E-mail. I'm not sure why the malloc and copy. Why not just use g_strdup()? Without either patch, my Evolution will core dump on start-up. First patch was rejected by gnome folks with the recommendation Don't do that, which, unfortunately, is not that easy to follow ;) Any comments from people with the knowledge of gnome, UTF-8 and string collation will be greatly appreciated. I'm hoping someone like bland chimes in here. Touching glib in such a way makes me nervous, but bland has had experience, especially with Cyrillic, so maybe he can offer some additional tips or insight. Joe -- PGP Key : http://www.marcuscom.com/pgp.asc signature.asc Description: This is a digitally signed message part
Re: [patch] glib20, UTF-8 and string collation
On Wed, 2008-01-09 at 12:35 -0500, Joe Marcus Clarke wrote:
On Wed, 2008-01-09 at 10:53 -0500, Alexandre Sunny Kovalenko wrote:
I have seen recent commit WRT string collation in devel/glib20 by
marcus, so I have decided to check if there is an interest to fix SEGV
in g_utf8_collate when it is given 8-bit non-UTF-8 string(s) to collate.
Any commits I have made in the area of UTF-8 are completely accidental.
I am not the UTF-8 guy. Both bland and jylefort have expressed interest
in this. Perhaps one of them will comment.
I hope so. Just in case, they would decide to, I have reduced the
situation to the small program below. I get
GLib-CRITICAL **: g_convert: assertion `str != NULL' failed
and no core dump from this simple program, whereas Evolution manages to
pass NULL to strcoll further down in g_utf8_collate and get SEGV for its
pains.
Conversely, if the answer still is Evolution should not have done
that, I will happily crawl back under my rock and keep my patch
locally.
#include stdio.h
#include glib.h
int main(int argc, char *argv[])
{
GString *str1 = g_string_new(\xf3\xcf\xd2\xcf\xcb\x20\xd7\xcf\xd3\xc5
\xcd\xd8\x20\xcf\xc2\xc5\xda\xd8\xd1\xce\x2e\x2e\x2e);
GString *str2 = g_string_new(Goodbye, cruel world!);
printf(%s\n, str1-str);
if(g_utf8_collate(str1-str, str2-str) == 0)
g_print(%s and %s are equal\n, str1-str, str2-str);
else
g_print(%s and %s are NOT equal\n, str1-str, str2-str);
}
built with
gcc `pkg-config --cflags glib-2.0` -g -o testCollate testCollate.c
`pkg-config --libs glib-2.0`
on
7.0-PRERELEASE FreeBSD 7.0-PRERELEASE #0: Sat Jan 5 22:37:29 EST 2008
using
glib-2.14.5 and libiconv-1.11_1
option fix string collation was set during the build of glib.
I'm not sure why the malloc and copy. Why not just use g_strdup()?
Because I don't know much of anything about glib programming, and I was
suspecting that someone will use g_free() on the result, so g_malloc()
sounded like the plausible candidate. I really hope someone with
the necessary knowledge will come up with the right solution, but, for
now, I do what I can.
--
Alexandre Sunny Kovalenko
___
freebsd-ports@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: [patch] glib20, UTF-8 and string collation
On Wed, 2008-01-09 at 19:40 -0500, Alexandre Sunny Kovalenko wrote: On Wed, 2008-01-09 at 12:35 -0500, Joe Marcus Clarke wrote: On Wed, 2008-01-09 at 10:53 -0500, Alexandre Sunny Kovalenko wrote: I have seen recent commit WRT string collation in devel/glib20 by marcus, so I have decided to check if there is an interest to fix SEGV in g_utf8_collate when it is given 8-bit non-UTF-8 string(s) to collate. Any commits I have made in the area of UTF-8 are completely accidental. I am not the UTF-8 guy. Both bland and jylefort have expressed interest in this. Perhaps one of them will comment. I hope so. Just in case, they would decide to, I have reduced the situation to the small program below. I get GLib-CRITICAL **: g_convert: assertion `str != NULL' failed and no core dump from this simple program, whereas Evolution manages to pass NULL to strcoll further down in g_utf8_collate and get SEGV for its pains. That sounds like a no-no for Evolution to be dereferencing a NULL pointer. Hopefully they'd fix this to prevent the problem. Conversely, if the answer still is Evolution should not have done that, I will happily crawl back under my rock and keep my patch locally. I can't imagine you're alone in this. But then again, any Cyrillic mail that comes my way is always spam, so what do I know. Joe -- PGP Key : http://www.marcuscom.com/pgp.asc signature.asc Description: This is a digitally signed message part
Re: [patch] glib20, UTF-8 and string collation
On Wed, 2008-01-09 at 20:16 -0500, Joe Marcus Clarke wrote:
On Wed, 2008-01-09 at 19:40 -0500, Alexandre Sunny Kovalenko wrote:
On Wed, 2008-01-09 at 12:35 -0500, Joe Marcus Clarke wrote:
On Wed, 2008-01-09 at 10:53 -0500, Alexandre Sunny Kovalenko wrote:
I have seen recent commit WRT string collation in devel/glib20 by
marcus, so I have decided to check if there is an interest to fix SEGV
in g_utf8_collate when it is given 8-bit non-UTF-8 string(s) to collate.
Any commits I have made in the area of UTF-8 are completely accidental.
I am not the UTF-8 guy. Both bland and jylefort have expressed interest
in this. Perhaps one of them will comment.
I hope so. Just in case, they would decide to, I have reduced the
situation to the small program below. I get
GLib-CRITICAL **: g_convert: assertion `str != NULL' failed
and no core dump from this simple program, whereas Evolution manages to
pass NULL to strcoll further down in g_utf8_collate and get SEGV for its
pains.
That sounds like a no-no for Evolution to be dereferencing a NULL
pointer. Hopefully they'd fix this to prevent the problem.
It's not Evolution, it is glib, specifically g_utf8_collate, which would
call strcoll(3) blindly on the return of g_utf8_normalize inside
gunicollate.c. And now, I can get core dumped out of this simple program
as well, merely by setting CHARSET=en_US.UTF-8 (I had it is ASCII in the
terminal window, which would trigger different path within
g_utf8_collate).
Conversely, if the answer still is Evolution should not have done
that, I will happily crawl back under my rock and keep my patch
locally.
I can't imagine you're alone in this. But then again, any Cyrillic mail
that comes my way is always spam, so what do I know.
More importantly, it is UTF-8 spam -- in order to trigger this, you need
KOI8-R or CP1251, and in the sorted column to boot. I suspect that
Latin1 or ShiftJIS would do the trick too.
Now, how about this: would you be amenable to this Really Harmless(tm)
patch, which merely adds error checking along the lines used in the same
function, about dozen lines up ;)
--- glib/gunicollate.c.B 2008-01-09 20:48:25.0 -0500
+++ glib/gunicollate.c 2008-01-09 20:49:35.0 -0500
@@ -166,6 +166,9 @@
str1_norm = g_utf8_normalize (str1, -1, G_NORMALIZE_ALL_COMPOSE);
str2_norm = g_utf8_normalize (str2, -1, G_NORMALIZE_ALL_COMPOSE);
+ g_return_val_if_fail (str1_norm != NULL, 0);
+ g_return_val_if_fail (str2_norm != NULL, 0);
+
if (g_get_charset (charset))
{
result = strcoll (str1_norm, str2_norm);
I can add it to your files/extra-patch-glib_gunicollate.c, or package
it separately -- I really hate it when I start Evolution after portupgrade
to write some E-mails real quick, only to find out that I have forgotten
to patch glib... again.
--
Alexandre Sunny Kovalenko
___
freebsd-ports@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: [patch] glib20, UTF-8 and string collation
Alexandre,
The problem you exposed have its roots in Evo code. g_utf8_* stuff
defined to work on *utf-8* strings only and have undefined behaviour on
MBCS strings. It may sound stupid but crashes are allowed in this case
:-) Even we apply your latest patch the true problem solution will be
only postponed. We have to continue with Evo source. Find subject parser
part and ensure that it will be utf-8 encoded string at the end.
All the best,
Alexander.
Alexandre Sunny Kovalenko wrote:
On Wed, 2008-01-09 at 20:16 -0500, Joe Marcus Clarke wrote:
On Wed, 2008-01-09 at 19:40 -0500, Alexandre Sunny Kovalenko wrote:
On Wed, 2008-01-09 at 12:35 -0500, Joe Marcus Clarke wrote:
On Wed, 2008-01-09 at 10:53 -0500, Alexandre Sunny Kovalenko wrote:
I have seen recent commit WRT string collation in devel/glib20 by
marcus, so I have decided to check if there is an interest to fix SEGV
in g_utf8_collate when it is given 8-bit non-UTF-8 string(s) to collate.
Any commits I have made in the area of UTF-8 are completely accidental.
I am not the UTF-8 guy. Both bland and jylefort have expressed interest
in this. Perhaps one of them will comment.
I hope so. Just in case, they would decide to, I have reduced the
situation to the small program below. I get
GLib-CRITICAL **: g_convert: assertion `str != NULL' failed
and no core dump from this simple program, whereas Evolution manages to
pass NULL to strcoll further down in g_utf8_collate and get SEGV for its
pains.
That sounds like a no-no for Evolution to be dereferencing a NULL
pointer. Hopefully they'd fix this to prevent the problem.
It's not Evolution, it is glib, specifically g_utf8_collate, which would
call strcoll(3) blindly on the return of g_utf8_normalize inside
gunicollate.c. And now, I can get core dumped out of this simple program
as well, merely by setting CHARSET=en_US.UTF-8 (I had it is ASCII in the
terminal window, which would trigger different path within
g_utf8_collate).
Conversely, if the answer still is Evolution should not have done
that, I will happily crawl back under my rock and keep my patch
locally.
I can't imagine you're alone in this. But then again, any Cyrillic mail
that comes my way is always spam, so what do I know.
More importantly, it is UTF-8 spam -- in order to trigger this, you need
KOI8-R or CP1251, and in the sorted column to boot. I suspect that
Latin1 or ShiftJIS would do the trick too.
Now, how about this: would you be amenable to this Really Harmless(tm)
patch, which merely adds error checking along the lines used in the same
function, about dozen lines up ;)
--- glib/gunicollate.c.B 2008-01-09 20:48:25.0 -0500
+++ glib/gunicollate.c 2008-01-09 20:49:35.0 -0500
@@ -166,6 +166,9 @@
str1_norm = g_utf8_normalize (str1, -1, G_NORMALIZE_ALL_COMPOSE);
str2_norm = g_utf8_normalize (str2, -1, G_NORMALIZE_ALL_COMPOSE);
+ g_return_val_if_fail (str1_norm != NULL, 0);
+ g_return_val_if_fail (str2_norm != NULL, 0);
+
if (g_get_charset (charset))
{
result = strcoll (str1_norm, str2_norm);
I can add it to your files/extra-patch-glib_gunicollate.c, or package
it separately -- I really hate it when I start Evolution after portupgrade
to write some E-mails real quick, only to find out that I have forgotten
to patch glib... again.
___
freebsd-ports@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: [patch] glib20, UTF-8 and string collation
On Thu, 2008-01-10 at 13:18 +0900, Alexander Nedotsukov wrote:
Alexandre,
The problem you exposed have its roots in Evo code. g_utf8_* stuff
defined to work on *utf-8* strings only and have undefined behaviour on
MBCS strings. It may sound stupid but crashes are allowed in this case
:-) Even we apply your latest patch the true problem solution will be
only postponed. We have to continue with Evo source. Find subject parser
part and ensure that it will be utf-8 encoded string at the end.
rant
While I see this as the noble goal, I fail to understand why asserts are
OK 10 lines above my latest patch, but not in this specific place. Nor
does this latest patch mask any issues in Evo -- you still get
Glib-CRITICAL assert. Hell, you even get assert without the patch if
your CHARSET is ASCII, but core dump if your CHARSET is xx_YY.UTF-8. If
we are talking noble goals here, how about some consistency in the error
handling?
/rant
On the more productive note: I have picked up more glib programming
today than I had before (or cared to) -- it's easy to improve when you
started from the veritable zero ;) However, it might not be sufficient
to do what needs to be done to Evolution. One thing you can help me
with, is to suggest glib function, I can feed arbitrary string of bytes
(either counted or zero-terminated) and it will tell me whether this is
valid UTF-8. g_print() apparently does this somehow, so there must be a
way.
As promised, I'll crawl back under my rock and wait for rainy weekend to
read some Evolution code.
All the best,
Alexander.
BTW: What happened to no top posting rule?
Alexandre Sunny Kovalenko wrote:
On Wed, 2008-01-09 at 20:16 -0500, Joe Marcus Clarke wrote:
On Wed, 2008-01-09 at 19:40 -0500, Alexandre Sunny Kovalenko wrote:
On Wed, 2008-01-09 at 12:35 -0500, Joe Marcus Clarke wrote:
On Wed, 2008-01-09 at 10:53 -0500, Alexandre Sunny Kovalenko wrote:
I have seen recent commit WRT string collation in devel/glib20 by
marcus, so I have decided to check if there is an interest to fix SEGV
in g_utf8_collate when it is given 8-bit non-UTF-8 string(s) to collate.
Any commits I have made in the area of UTF-8 are completely accidental.
I am not the UTF-8 guy. Both bland and jylefort have expressed interest
in this. Perhaps one of them will comment.
I hope so. Just in case, they would decide to, I have reduced the
situation to the small program below. I get
GLib-CRITICAL **: g_convert: assertion `str != NULL' failed
and no core dump from this simple program, whereas Evolution manages to
pass NULL to strcoll further down in g_utf8_collate and get SEGV for its
pains.
That sounds like a no-no for Evolution to be dereferencing a NULL
pointer. Hopefully they'd fix this to prevent the problem.
It's not Evolution, it is glib, specifically g_utf8_collate, which would
call strcoll(3) blindly on the return of g_utf8_normalize inside
gunicollate.c. And now, I can get core dumped out of this simple program
as well, merely by setting CHARSET=en_US.UTF-8 (I had it is ASCII in the
terminal window, which would trigger different path within
g_utf8_collate).
Conversely, if the answer still is Evolution should not have done
that, I will happily crawl back under my rock and keep my patch
locally.
I can't imagine you're alone in this. But then again, any Cyrillic mail
that comes my way is always spam, so what do I know.
More importantly, it is UTF-8 spam -- in order to trigger this, you need
KOI8-R or CP1251, and in the sorted column to boot. I suspect that
Latin1 or ShiftJIS would do the trick too.
Now, how about this: would you be amenable to this Really Harmless(tm)
patch, which merely adds error checking along the lines used in the same
function, about dozen lines up ;)
--- glib/gunicollate.c.B 2008-01-09 20:48:25.0 -0500
+++ glib/gunicollate.c 2008-01-09 20:49:35.0 -0500
@@ -166,6 +166,9 @@
str1_norm = g_utf8_normalize (str1, -1, G_NORMALIZE_ALL_COMPOSE);
str2_norm = g_utf8_normalize (str2, -1, G_NORMALIZE_ALL_COMPOSE);
+ g_return_val_if_fail (str1_norm != NULL, 0);
+ g_return_val_if_fail (str2_norm != NULL, 0);
+
if (g_get_charset (charset))
{
result = strcoll (str1_norm, str2_norm);
I can add it to your files/extra-patch-glib_gunicollate.c, or package
it separately -- I really hate it when I start Evolution after portupgrade
to write some E-mails real quick, only to find out that I have forgotten
to patch glib... again.
--
Alexandre Sunny Kovalenko (Олександр Коваленко)
___
freebsd-ports@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: [patch] glib20, UTF-8 and string collation
Alexandre Sunny Kovalenko wrote: On Thu, 2008-01-10 at 13:18 +0900, Alexander Nedotsukov wrote: Alexandre, The problem you exposed have its roots in Evo code. g_utf8_* stuff defined to work on *utf-8* strings only and have undefined behaviour on MBCS strings. It may sound stupid but crashes are allowed in this case :-) Even we apply your latest patch the true problem solution will be only postponed. We have to continue with Evo source. Find subject parser part and ensure that it will be utf-8 encoded string at the end. rant While I see this as the noble goal, I fail to understand why asserts are OK 10 lines above my latest patch, but not in this specific place. Nor does this latest patch mask any issues in Evo -- you still get Glib-CRITICAL assert. Hell, you even get assert without the patch if your CHARSET is ASCII, but core dump if your CHARSET is xx_YY.UTF-8. If we are talking noble goals here, how about some consistency in the error handling? /rant Those asserts does trivial arguments validation. Yours are slightly more than trivial which may be against existing glib practice. In any case I am against introducing such change locally. It lead to different execution paths in callers compared to other platforms. Also in a current form it suffer form potential memory leaks (guess where). Please save your energy for debates with glib developers if you want to. Even you convince hundredths like me it still will not make a valid argument for them ;-) On the more productive note: I have picked up more glib programming today than I had before (or cared to) -- it's easy to improve when you started from the veritable zero ;) However, it might not be sufficient to do what needs to be done to Evolution. One thing you can help me with, is to suggest glib function, I can feed arbitrary string of bytes (either counted or zero-terminated) and it will tell me whether this is valid UTF-8. g_print() apparently does this somehow, so there must be a way. See g_utf8_validate(). As promised, I'll crawl back under my rock and wait for rainy weekend to read some Evolution code. You better kick Evo authors into $some_painful_part_of_the_body :-) All the best, Alexander. BTW: What happened to no top posting rule? Alexandre Sunny Kovalenko wrote: On Wed, 2008-01-09 at 20:16 -0500, Joe Marcus Clarke wrote: On Wed, 2008-01-09 at 19:40 -0500, Alexandre Sunny Kovalenko wrote: On Wed, 2008-01-09 at 12:35 -0500, Joe Marcus Clarke wrote: On Wed, 2008-01-09 at 10:53 -0500, Alexandre Sunny Kovalenko wrote: I have seen recent commit WRT string collation in devel/glib20 by marcus, so I have decided to check if there is an interest to fix SEGV in g_utf8_collate when it is given 8-bit non-UTF-8 string(s) to collate. Any commits I have made in the area of UTF-8 are completely accidental. I am not the UTF-8 guy. Both bland and jylefort have expressed interest in this. Perhaps one of them will comment. I hope so. Just in case, they would decide to, I have reduced the situation to the small program below. I get GLib-CRITICAL **: g_convert: assertion `str != NULL' failed and no core dump from this simple program, whereas Evolution manages to pass NULL to strcoll further down in g_utf8_collate and get SEGV for its pains. That sounds like a no-no for Evolution to be dereferencing a NULL pointer. Hopefully they'd fix this to prevent the problem. It's not Evolution, it is glib, specifically g_utf8_collate, which would call strcoll(3) blindly on the return of g_utf8_normalize inside gunicollate.c. And now, I can get core dumped out of this simple program as well, merely by setting CHARSET=en_US.UTF-8 (I had it is ASCII in the terminal window, which would trigger different path within g_utf8_collate). Conversely, if the answer still is Evolution should not have done that, I will happily crawl back under my rock and keep my patch locally. I can't imagine you're alone in this. But then again, any Cyrillic mail that comes my way is always spam, so what do I know. More importantly, it is UTF-8 spam -- in order to trigger this, you need KOI8-R or CP1251, and in the sorted column to boot. I suspect that Latin1 or ShiftJIS would do the trick too. Now, how about this: would you be amenable to this Really Harmless(tm) patch, which merely adds error checking along the lines used in the same function, about dozen lines up ;) --- glib/gunicollate.c.B 2008-01-09 20:48:25.0 -0500 +++ glib/gunicollate.c 2008-01-09 20:49:35.0 -0500 @@ -166,6 +166,9 @@ str1_norm = g_utf8_normalize (str1, -1, G_NORMALIZE_ALL_COMPOSE); str2_norm = g_utf8_normalize (str2, -1, G_NORMALIZE_ALL_COMPOSE); + g_return_val_if_fail (str1_norm != NULL, 0); + g_return_val_if_fail (str2_norm != NULL, 0); + if
