Andy Wingo <wi...@pobox.com> writes: > That's the crazy thing: file names on GNU aren't in any encoding! They > are byte strings that may or may not decode to a string, given some > encoding. Granted, they're mostly UTF-8 these days, but users have the > darndest files... [...] > On Tue 03 May 2011 00:18, l...@gnu.org (Ludovic Courtès) writes: >> I think GLib and the like expect UTF-8 as the file name encoding and >> complain otherwise, so UTF-8 might be a better default than locale >> encoding (and it’s certainly wiser to be locale-independent.) > > It's more complicated than that. Here's the old interface that they > used, which attempted to treat paths as utf-8: > > http://developer.gnome.org/glib/unstable/glib-Character-Set-Conversion.html > (search for "file name encoding") > > The new API is abstract, so it allows operations like "get-display-name" > and "get-bytes": > > http://developer.gnome.org/gio/2.28/GFile.html (search for "encoding" > in that page) > > "All GFiles have a basename (get with g_file_get_basename()). These > names are byte strings that are used to identify the file on the > filesystem (relative to its parent directory) and there is no > guarantees that they have any particular charset encoding or even make > any sense at all. If you want to use filenames in a user interface you > should use the display name that you can get by requesting the > G_FILE_ATTRIBUTE_STANDARD_DISPLAY_NAME attribute with > g_file_query_info(). This is guaranteed to be in utf8 and can be used > in a user interface. But always store the real basename or the GFile > to use to actually access the file, because there is no way to go from > a display name to the actual name."
In my opinion, this is a bad approach to take in Guile. When developers are careful to robustly handle filenames with invalid encoding, it will lead to overly complex code. More often, when developers write more straightforward code, it will lead to code that works most of the time but fails badly when confronted with weird filenames. This is the same type of problem that plagues Bourne shell scripts. Let's please not go down that road. There is a better way. We can do a variant of what Python 3 does, described in PEP 383 <http://www.python.org/dev/peps/pep-0383/>. Basically, the idea is to provide alternative versions of scm_{to,from}_stringn that allow arbitrary bytevectors to be turned into strings and back again without any lossage. These alternative versions would be used for operations involving filenames et al, and should probably also be made available to users. Basically the idea is that "invalid bytes" are mapped to code points that will never appear in any valid encoding. PEP 383 maps such bytes to a range of surrogate code points that are reserved for use in UTF-16 surrogate pairs, and are otherwise considered invalid by Unicode. There are other possible mapping schemes as well. See section 3.7 of Unicode Technical Report #36 <http://www.unicode.org/reports/tr36/> for more discussion on this. I can understand why some say that filenames in GNU are not really strings but rather bytevectors. I respectfully disagree. Filenames, environment variables, command-line arguments, etc, are _conceptually_ strings. Let's not muddle that concept just because the transition to Unicode has not yet been completed in the GNU world. Hopefully in the future, these old-style POSIX byte strings will once again become true strings in concept. All that's required for this to happen is for popular software to agree to standardize on the use of UTF-8 for all of these things. This is reasonably likely to happen at some point. In practice, I see no advantage to calling them bytevectors other than to allow lossless storage of oddball filenames. It's not as if any sane user interface is going to display them in hex. Think about it. What are you really going to do with the bytevector version, other than to store it in case you want to convert it back into a filename, environment variable, or command-line argument? Think about the mess that this will make to otherwise simple code. Also think about the obscure bugs that will arise from programmers who balk at this and simply pass around the strings instead. Let's keep things simple. Let's use plain strings for everything that is _conceptually_ a string. Let's instead deal with the occasional ill-encoded-filename by allowing strings to represent these oddballs. Best, Mark