The way in which Perl supports Unicode, you normally should hardly ever have to call a UTF-8 encoder or decoder explicitely and manually. You just have to make sure that when a UTF-8 string enters Perl, it does so tagged as a UTF-8 string and not as an octet string. How that happens depends on how the string gets into Perl. When opening files, for instance, you can tell Perl the charset to expect or to look at the LC_CTYPE locale.
I've been told that I'm not supposed to ever need to manually call unicode encoders or decoders, but it was the only way I could make the script work.
My script takes filenames as arguments (normally provided by globbing in bash), then it does some manipulations on the filename, and it pipes some output to vorbiscomment. Basically, my music collection stores all the metadata in the filesystem itself, and this script is a bit of glue for moving the metadata out of the filesystem and into the file, for music apps that expect the metadata to be in the file.
What happens is, the filenames end up having non-ASCII characters in them, as some of my music isn't English.
At first, the script was very simple. It would parse the filename, then pipe the info to vorbiscomment, which would store the info. The problem was, XMMS would display (for example) a "u with umlaut" as being two characters, "A with tilde" and "1/4", or something similar. And it would do that with ALL non-ASCII characters. Finding a solution nearly drove me nuts, as writing the data manually into vorbiscomment worked fine.
So then I started futzing around with 'decode_utf8', and just like magic, everything worked. Before I used that function, perl couldn't even print the filenames properly without munging the unicode. It's been a while, but I'm relatively certain I was using perl 5.8.0 at the time.
Now, after months of having the script "just work", all the unicode characters were suddenly turning into the '#' character for no reason. I started futzing around with my script again, trying to solve this problem, and removing all calls to 'decode_utf8' fixed everything up. Now everything "Just Works" like it ought to have been way back when I originally created the script. As far as I know, I haven't upgraded perl since then.
Like I was saying, I don't have a problem: the script works! I was just wondering what the hell happened that suddenly "decode_utf8" was no longer necessary.
Question: What is a quick way in Perl to get a regular expression that matches all Unicode characters in the range U0100..U10FFFF, in other words all non-ASCII Unicode characters?
Would it be possible to do something like this:
/[\x{0x0100}-\x{0x10FFFF}]/
I dunno, it makes sense in my head, but I haven't tested it. If this doesn't work, you might want to try something with 'ord' and testing if the resultant number is within that range.
-- Linux-UTF8: i18n of Linux on all levels Archive: http://mail.nl.linux.org/linux-utf8/
