Re: Strip incompatible characters from Windows partitions!
Am 16.05.2008 um 13:05 schrieb Milosz Derezynski: Furthermore, does anyone know how OS X and/or other operating systems handle this issue? On Mac OS X, you can happily create files with names Windows Explorer can't read. Recently it happened to me with a filename with a plain space(!) as the last character. Samba not allowing/normalizing such filenames is i think a good hint that it shouldn't be considered sane to write such filenames to a FAT directory structure. Well, Samba it's self doesn't name or create files. Samba provides whatever the served file system contains. This is fine, as it can be used to copy files from one Linux machine to another Linux box. Markus - - - - - - - - - - - - - - - - - - - Dipl. Ing. Markus Hitter http://www.jump-ing.de/ -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: Strip incompatible characters from Windows partitions!
Am 16.05.2008 um 03:05 schrieb Evan: I don't know where the filename check is supposed to happen, but it isn't happening anywhere. I've tried via the cli, and via nautilus, and neither of them prevent me from using Windows-illegal characters. ... because they are perfectly legal on the OS you're currently running. By what you describe, I'm not sure wether Ubuntu is to blame here. IMHO, the best one could do is to introduce some Windows- compatibility mode. Prohibiting feature X here because it's forbidden there isn't a good idea. Markus - - - - - - - - - - - - - - - - - - - Dipl. Ing. Markus Hitter http://www.jump-ing.de/ -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: Strip incompatible characters from Windows partitions!
On 16/05/2008, Markus Hitter [EMAIL PROTECTED] wrote: Prohibiting feature X here because it's forbidden there isn't a good idea. It is in this case because of Ubuntu's target population and the fact the NTFS / FAT32 are not native Linux file formats. By all means allow advanced users to turn this feature off but by default you should not be allowed to create files which cause such a problem. Alan -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: Strip incompatible characters from Windows partitions!
Scott Kitterman pisze: Doesn't wubi install Ubuntu into an existing Windows partition? Exactly. And then Ubuntu will happily let you create files that you can't read in Windows. It's weird. It just ocurred to me that when you email files, odds are the receiver is using Windows. Perhaps all the mail clients should be patched with similar warnings? That's probably a lot more common than copying from one partition to another. There's no need for that. If you receive an email attachement with a windows-illegal filename then the windows filesystem will refuse to save it under such name and you will be prompted for a name change. -- ## Przemysław Kulczycki Azrael Nightwalker ## # jabber: azrael[na]jabster.pl | tlen: azrael29a # ### www: http://reksio.ftj.agh.edu.pl/~azrael/ ### signature.asc Description: OpenPGP digital signature -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: Strip incompatible characters from Windows partitions!
Le vendredi 16 mai 2008 à 00:06 -0400, Scott Kitterman a écrit : On Thursday 15 May 2008 21:31, Evan wrote: On Thu, May 15, 2008 at 9:14 PM, Scott Kitterman [EMAIL PROTECTED] wrote: I'd say that if there's a bug it's in Windows. I could see a wishlist bug against Ubuntu to provide a way to check for this/suggest changes to avoid problematic filenames, but there is nothing inherently defective with the current behavior. Scott K I agree that there is no inherent problem with the Ubuntu code, and it should really be up to Windows to support more characters. However I can think of several situations where this could cause considerable problems for the end user. We should at the very least provide a warning that Naming a file on this partition with any of the following characters will prevent Windows from opening it. Are you sure you want to continue? Evan Personally I'm against such hand holding. If any such feature is provided, I think it should be off by default. I happen to have some legacy FAT32 and NTFS partitions for various reasons, but the odds that they will ever be read from Windows are very low. I don't think Ubuntu's design should be predicated on the idea that it's an adjunct to using Windows. Sorry for your legacies, but IMHO partitions with a Microfost-ish filesystem are meant to be used with Windows, and if you want to use the full possibilities Unix offers you, just use Unix filesystems. The default should be to be fully Windows-compliant - and you may add an option in /etc/fstab disabling character stripping. Why the hell would you use a Windows filesystem in a Linux-only environment?! I can only think of cases when Windows will have to access one day or another the filesystem: USB keys, external HD, Windows partitions on dual boot... Samba does not provide Windows with invalid characters when sharing files, and Linux must do the same with filesystems. Hope Ubuntu is more modest than you appear to see it. Serve the user, not the ideal technology you dream of in which every character is supported in filenames. When you're working on documents, being able to read it in a conference from your USB key is much more important than being allowed to keep a '?' in its filename, isn't it? -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: Strip incompatible characters from Windows partitions!
Il giorno ven, 16/05/2008 alle 11.42 +0200, Milan Bouchet-Valat ha scritto: Hope Ubuntu is more modest than you appear to see it. Serve the user, not the ideal technology you dream of in which every character is supported in filenames. I keep a copy of my working files in an usb pen. This is FAT because I want to access it from every machine with an usb hole, and my working files (including bzr and hg repositories) are in a subdirectory. I expect that this use case is very common among ubuntu users nowadays and the default behaviour is the only one that does not get in the way. In any case, windows handling of file names has always been broken. You can create files from the command prompt or via FTP that make a whole mess in the graphical file manager.These are bugs in windows. An improvement could perhaps be having a default behavior of warning users only if windows boot files are present in the root of the disk. Vincenzo -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: Strip incompatible characters from Windows partitions!
But clearly this issue can be seen as a limitation of the FAT filesystem, just not yet imposed at the highest level of the filesystem driver (kernel or userland)? Surely ext3 *would* allow a slash in a filename (i guess?), if the userland tools would just let the filesystem driver ever receive such a name, even if it might then break the filesystem since it would create an ambiguity with a subdirectory. I can't be quite sure about this but i assume that at a technical level, a slash could be just part of a filename in ext3, XFS or whatever. If that's the case, what would be the problem of implementing a restriction into the VFAT module akin to the restriction in ext3 not allowing to use slashes in filenames? Whether this restriction is synthetic or not would be IMO up to debate and should find its grounds on whether slashes in ext3 filenames are an artificact of high-level access to the filesystem or not (as we're dealing with this on Linux here it seems to be a good base for debate). Furthermore, does anyone know how OS X and/or other operating systems handle this issue? Samba not allowing/normalizing such filenames is i think a good hint that it shouldn't be considered sane to write such filenames to a FAT directory structure. 2008/5/16, Chris Jones [EMAIL PROTECTED]: Hi Evan wrote: paths are slashes, while Windows has a considerable list (apostrophes, asterisks, etc). The problem is that it doesn't have a list, there are multiple lists and they aren't documented. NTFS will reject some filenames, win16/win32/win64/.net/etc. will reject others (as you have seen by Linux's ability to create files that Windows Explorer refuses to handle). There is no canonical list of filenames to avoid on Windows. See: http://blogs.msdn.com/michkap/archive/2006/11/03/941420.aspx That alone makes this (imho) a basically intractable problem, unfortunately. Cheers, -- Chris Jones -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: Strip incompatible characters from Windows partitions!
On Fri, 16 May 2008 09:03:02 +0100 Alan Milnes [EMAIL PROTECTED] wrote: On 16/05/2008, Markus Hitter [EMAIL PROTECTED] wrote: Prohibiting feature X here because it's forbidden there isn't a good idea. It is in this case because of Ubuntu's target population and the fact the NTFS / FAT32 are not native Linux file formats. By all means allow advanced users to turn this feature off but by default you should not be allowed to create files which cause such a problem. While NTFS is closely associated with Windows, FAT is a defacto standard for portable storage devices. Making and kind of O/S assumptions about FAT is inherently incorrect. For NTFS, I can almost see sense in this and wouldn't be opposed if there were some NTFS spec or standard to base such a design on. I gather from another post in this thread that such a document does not really exist. I feel that you are saying that because I don't also use Windows, I'm not in Ubuntu's target population. I think this is completely wrong. I think fixing Bug 1 is about getting people to stop using Windows, not about using Windows and Ubuntu. I see value in easing transition for new users, but our target is an Ubuntu user, not an Ubuntu/Windows user. Scott K -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: Strip incompatible characters from Windows partitions!
On Fri, 16 May 2008 11:42:28 +0200 Milan Bouchet-Valat [EMAIL PROTECTED] wrote: ... Hope Ubuntu is more modest than you appear to see it. Serve the user, not the ideal technology you dream of in which every character is supported in filenames. When you're working on documents, being able to read it in a conference from your USB key is much more important than being allowed to keep a '?' in its filename, isn't it? First, many external storage devices come pre-formatted with FAT32. It is not at all a Windows specific F/S. Second, my concern isn't with files I manually name (I'm pretty unlikely to use file names that wouldn't be legal under any OS), but systems that generate files/names automatically. I would find it frustrating to get a big stack of Are you sure warnings when copying my maildir files (default storage method for my chosen MUA) to a USB stick. Scott K -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: Strip incompatible characters from Windows partitions!
On 16/05/2008, Scott Kitterman [EMAIL PROTECTED] wrote: While NTFS is closely associated with Windows, FAT is a defacto standard for portable storage devices. Agreed. It is not a Linux 'native' file format though. Making and kind of O/S assumptions about FAT is inherently incorrect. For NTFS, I can almost see sense in this and wouldn't be opposed if there were some NTFS spec or standard to base such a design on. I gather from another post in this thread that such a document does not really exist. That is a problem but the likes of SAMBA have some ways of at least attempting to stop you creating files that then cause problems. I feel that you are saying that because I don't also use Windows, I'm not in Ubuntu's target population. I think this is completely wrong. I think fixing Bug 1 is about getting people to stop using Windows, not about using Windows and Ubuntu. I see value in easing transition for new users, but our target is an Ubuntu user, not an Ubuntu/Windows user. Absolutely agree - that's why we should make this change and allow experienced users to switch it off. We want people to switch and a large percentage of them probably dip their toe in the water through things like dual booting etc. Alan -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: Strip incompatible characters from Windows partitions!
On Fri, 2008-05-16 at 08:24 -0400, Scott Kitterman wrote: FAT is a defacto standard for portable storage devices. Not true anymore, the external disks I have seen that have 300 GB came with NTFS. Anyway, external disks may be a different topic altogether, but what about the Windows system partition that Ubuntu mounts writable by default (IIRC)? -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: Strip incompatible characters from Windows partitions!
On Friday 16 May 2008 12:19, Mario Vukelic wrote: On Fri, 2008-05-16 at 08:24 -0400, Scott Kitterman wrote: FAT is a defacto standard for portable storage devices. Not true anymore, the external disks I have seen that have 300 GB came with NTFS. Anyway, external disks may be a different topic altogether, but what about the Windows system partition that Ubuntu mounts writable by default (IIRC)? OK. So then it's a mistake to assume anything about the target O/S based on they file system type. There was recently some discussion about this on #ubuntu-motu IRC and it seems pretty difficult to actually do this reliably and completely. Scott K -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: Strip incompatible characters from Windows partitions!
This e-mail summarises a discussion in #ubuntu-motu between myself, ScottK and persia. I'll first explain the general problem, then suggest a messy solution to a surprisingly messy problem. Most of these ideas are not my own, and in fact had to be explained to me at some length, so please don't assume that I know what I'm talking about ;) Since there wasn't an NTFS expert available during the conversation, it's possible that the following is only true of FAT filesystems. Characters like '' and '/' are in fact just the tip of the iceberg - see https://bugs.launchpad.net/ubuntu/+source/dosfstools/+bug/49217 for another way that the same problem can bite you. Because there are no proper standards for Windows filesystems, there's no common agreement about how to turn the string of bytes that make up a FAT filename into a string of characters. For example, a Japanese computer might look at a filesystem and assume that all the files are encoded in SHIFT-JIS, while a Western European computer might look at the same filesystem and assume that all the files are encoded in code page 1252. Most irritatingly, FAT filenames can use single-byte encodings (like ASCII), multi-byte (like UTF-8), or double-byte (like UTF-16). This means that a filename might be valid ASCII (perhaps including some disallowed ASCII characters, perhaps not), but which would be garbled nonsense if interpreted as such. The above problems make automatically detecting the character encoding of files in a FAT filesystem at best hard and sometimes impossible. Therefore, there's no general way to tell whether '', '/' etc. are valid characters in a given file in a FAT file. Even if there were a way to work out which characters are allowed, ext2-on-Windows drivers make it possible to have files with disallowed characters in a Windows system. Disallowed characters aren't so much a Windows kernel issue as a pervasive Windows UI issue. The exception that proves the rule is Emacs on Windows. Emacs being Emacs, it pays little attention to the conventions of young upstarts like Microsoft, so can handle files with funnily-named characters just fine. Given the above, my suggestion is that there ought to be a tool that runs identically in Windows and Linux that interactively converts files. It would ask for an initial encoding, target encoding, and target path, then recurse through all the directories rooted in that path, translating files as it goes. Characters that are valid but tend to cause headaches could be automatically converted, or the user could be prompted for a better name. Most of the actual work in this program can be done by iconv, although it might be worth having a punycode mode that minimises incompatibility at the expense of readability. Finally, I would suggest that the Windows version be run straight from the Ubuntu CD, rather than being made available from some website somewhere. As well as making the program a little bit easier to find, it makes a great advert for Linux - it solves the problems that Windows causes. - Andrew -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: Strip incompatible characters from Windows partitions!
Andrew Sayers andrew-ubuntu-devel at pileofstuff.org writes: Since there wasn't an NTFS expert available during the conversation, NTFS is pretty well known and documented, especially filename handling. Windows also do allow the creation of such filenames but it's not so widely known how to do it. When most Windows applications try to read such filenmes then they get just as confused as if the files were created on Linux (google for Windows SFU). It's also documented at http://ntfs-3g.org/support.html#posixfilenames2 -- Why does the driver allow special characters in the filenames? NTFS supports several filename namespaces at the same time: DOS, Win32 and POSIX. While the NTFS-3G driver handles all of them, it always creates new files in the POSIX namespace for maximum portability and interoperability reasons. This means that filenames are case sensitive and all characters are allowed except '/' and '\0'. This is perfectly legal on Windows, though some application may get confused. If you find so then please report it to the developer of the relevant Windows software. Workaround: If case insensitivity handling and/or restriction of special character usage is desirable then you may export the NTFS volume via Samba which supports this functionality the same way as it does for other POSIX file systems. Status: Not NTFS-3G problem. - Regards, Szaka NTFS-3G: http://ntfs-3g.org -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: Strip incompatible characters from Windows partitions!
On Thu, 15 May 2008 20:29:44 -0400 Evan [EMAIL PROTECTED] wrote: Which package would this be filed against? I'd say that if there's a bug it's in Windows. I could see a wishlist bug against Ubuntu to provide a way to check for this/suggest changes to avoid problematic filenames, but there is nothing inherently defective with the current behavior. Scott K -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: Strip incompatible characters from Windows partitions!
On Thu, May 15, 2008 at 9:14 PM, Scott Kitterman [EMAIL PROTECTED] wrote: I'd say that if there's a bug it's in Windows. I could see a wishlist bug against Ubuntu to provide a way to check for this/suggest changes to avoid problematic filenames, but there is nothing inherently defective with the current behavior. Scott K I agree that there is no inherent problem with the Ubuntu code, and it should really be up to Windows to support more characters. However I can think of several situations where this could cause considerable problems for the end user. We should at the very least provide a warning that Naming a file on this partition with any of the following characters will prevent Windows from opening it. Are you sure you want to continue? Evan -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: Strip incompatible characters from Windows partitions!
On Thursday 15 May 2008 21:31, Evan wrote: On Thu, May 15, 2008 at 9:14 PM, Scott Kitterman [EMAIL PROTECTED] wrote: I'd say that if there's a bug it's in Windows. I could see a wishlist bug against Ubuntu to provide a way to check for this/suggest changes to avoid problematic filenames, but there is nothing inherently defective with the current behavior. Scott K I agree that there is no inherent problem with the Ubuntu code, and it should really be up to Windows to support more characters. However I can think of several situations where this could cause considerable problems for the end user. We should at the very least provide a warning that Naming a file on this partition with any of the following characters will prevent Windows from opening it. Are you sure you want to continue? Evan Personally I'm against such hand holding. If any such feature is provided, I think it should be off by default. I happen to have some legacy FAT32 and NTFS partitions for various reasons, but the odds that they will ever be read from Windows are very low. I don't think Ubuntu's design should be predicated on the idea that it's an adjunct to using Windows. Scott K -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: Strip incompatible characters from Windows partitions!
On Thu, 2008-05-15 at 21:14 -0400, Scott Kitterman wrote: but there is nothing inherently defective with the current behavior. I'd agree for any other fs, but the only reason you would use an ntfs partition is because you want to read this in windows. Thus it makes little sense to allow creating filenames that prevent it. -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: Strip incompatible characters from Windows partitions!
On Fri, 16 May 2008 06:36:54 +0200 Mario Vukelic [EMAIL PROTECTED] wrote: þÿOn Thu, 2008-05-15 at 21:14 -0400, Scott Kitterman wrote: but there is nothing inherently defective with the current behavior. I'd agree for any other fs, but the only reason you would use an ntfs partition is because you want to read this in windows. Thus it makes little sense to allow creating filenames that prevent it. Or the system previously had Windows on it. Doesn't wubi install Ubuntu into an existing Windows partition? I don't think it's good design to make assumptions about O/S based on file systems. I think it makes little sense to expend effort to add such hand holding 'features'. If someone does add something (while I don't see the value, who am I to tell you not to volunteer if you do), all I ask is that it not be inflicted on users by default. Scott K -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: Strip incompatible characters from Windows partitions!
On Fri, 2008-05-16 at 00:50 -0400, Scott Kitterman wrote: Doesn't wubi install Ubuntu into an existing Windows partition? Exactly. And then Ubuntu will happily let you create files that you can't read in Windows. It's weird. -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: Strip incompatible characters from Windows partitions!
On Fri, 16 May 2008 06:53:35 +0200 Mario Vukelic [EMAIL PROTECTED] wrote: þÿOn Fri, 2008-05-16 at 00:50 -0400, Scott Kitterman wrote: Doesn't wubi install Ubuntu into an existing Windows partition? Exactly. And then Ubuntu will happily let you create files that you can't read in Windows. It's weird. It just ocurred to me that when you email files, odds are the receiver is using Windows. Perhaps all the mail clients should be patched with similar warnings? That's probably a lot more common than copying from one partition to another. Scott K -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
