On Aug 6, 2007, at 4:10 PM, Wilfredo Sánchez Vega wrote:
(Sorry for the lame reply latency.)
On Jul 18, 2007, at 5:24 PM, Roy T. Fielding wrote:
A system less concerned with backwards compatibility is better off
with a requirement of utf-8, though OS X should have made the
filename
encoding a mount option.
I disagree. Having one encoding is far superior to every
application having to first find out what encoding the filesystem
is question is using then using that.
I see no value in having different mount points use different
encodings.
Well, neither do I (now that utf-8 exists), but the fact is that they do
and they aren't necessarily controlled by the same OS.
I assume that the ISO9660-Joliet (CD-ROM) driver does
some form of filename translation automatically from UCS-2.
The underlying volume format can use whatever it wants. Ideally
the format defines what that is. Unfortunately, that's not the
case, but for those that do, yes, converting to UTF-8 is the
responsibility of the file system at the VFS layer.
I suppose that a mount option to tell the filesystem that "this
UFS volume uses encoding X" would be useful, but I maintain that
above the kernel, you really want one encoding, not N. Helping the
kernel know what's underneath is certainly useful.
I agree. But is it the case that non-native mounted filesystems
are name-translated by the kernel? I mean, if OS X did this
consistently
for all mount points, then I would see it as being reasonable for the
OS X applications to reject anything else.
In any case, even with the convention, it is left to the application
to determine how it will treat encoded filenames. The OS X decision
to treat them all as utf-8 is at least consistent. OTOH, this
is just a display convention -- OS X apps should have been designed
to treat the filename internally as an opaque nul-terminated array,
rather than barfing on non-utf8 encodings.
This is difficult in practice. When the open panel sees a file
that is not in UTF-8, there is no reliable way to display anything
sane to the user. I suppose a Linux nerd might say "show me some
hex" or something, but most of our users are not Linux nerds. I
agree that crashing is worse than hex, though.
Actually, it also crashes on valid utf-8 in normal form, because OS X
doesn't follow the standard on normalization. See "man -s 5 utf8":
If more than a single representation of a value exists (for
example,
0x00; 0xC0 0x80; 0xE0 0x80 0x80) the shortest representation is
always
used. Longer ones are detected as an error as they pose a
potential
security risk, and destroy the 1:1 character:octet sequence
mapping.
but OS X requires the longer composition characters over shorter ones.
My guess is that choice was driven by the way the UI allows such
characters to be composed (like "alt-u u" for uumlaut).
Of course, even with these issues, the Mac still kicks ass.
Basically, on Mac OS X, you can, in fact, use whatever characters
you like on UFS and BSD level software tends to cope with that. But
if you aren't using UTF-8, then you aren't writing file name that
are meant for user consumption. ie. that may be OK for a database
(eg. fsfs), though I think that even in that case you can
reasonably stick to ASCII in many cases.
One thing I miss in OS X is an automated way for file archivers
(like unzip) to recognize and convert non-utf-8 filenames
when they are unarchived. I frequently have to do that by hand
after unzipping something from China or Switzerland.
Again, same as with volume formats, if the zip file format
defines the encoding in zip files, then this should be easy
(insofar as encodings are easy) for the software to deal with.
Sadly, it doesn't (filenames are just null-terminated strings). There
are options for conversion from EBCDIC, but nothing to transcode the
filenames in general as they are unzipped. Maybe the zip command
maintainer will take that as an enhancement request.
Subversion
breaks on OS X whenever someone commits a filename with an e-grave,
which is a problem when your main product name is Communiqué.
I wonder if this change in APR would fix that error?
You still have to hope that the inbound encoding is correct (that
is, that svn somehow knows it). On OS X, that's easy; it's UTF-8.
Once other operating systems come into the mix, it'll works as well
as the encodings are defined (and known to svn) on those systems.
What I do currently is define
setenv MM_CHARSET "utf-8"
setenv LANG "en_US.utf-8"
in my shell init file.
....Roy
