On Mon, Dec 8, 2008 at 5:30 PM, Ed Loach <[EMAIL PROTECTED]> wrote:
> Can anyone work out how to find out what possible UTF8 errors there
> might be? Or would you be able to tell some other way if this were
> the problem?
You should try using decode("UTF-8") instead of decode("utf8") in your
checking routine. UTF-8 and utf8 are not equivalent under Encode, the
former is more strict, see
http://perldoc.perl.org/Encode.html#UTF-8-vs.-utf8-vs.-UTF8
encode("utf8", "\x{FFFF_FFFF}", 1); # okay
encode("UTF-8", "\x{FFFF_FFFF}", 1); # croaks
It would also be useful to log these errors, they suggest invalid
byte-sequences in the OSM dataset and it would be useful to fix them
at their source.
_______________________________________________
talk mailing list
[email protected]
http://lists.openstreetmap.org/listinfo/talk
