Bug#500540: kdebase: automounting vfat (partialy) case sensitive due to utf8 is plain wrong and dangerous
On Monday 17 November 2008 22:00:52 Heinrich Langos wrote: IMHO this is the real solution that you are looking for, but there is (or was?) a problem with making it the default. (See Message#48 for detail Hi Qt-KDE, Work on this RC bug seems to of stalled in the last couple of weeks. Could I ask what your proposed way forward for this bug for lenny? Could be worth discussing on debian-release as well. Thanks, Mark -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#500540: kdebase: automounting vfat (partialy) case sensitive due to utf8 is plain wrong and dangerous
New version of pmount package was uploaded (and unblocked) recently. Sorry to say that, but upgrading to pmount 0.9.18 didn't fix the problem. [EMAIL PROTECTED]:~$ pmount -V 0.9.18 [EMAIL PROTECTED]:~$ dmesg | tail [64822.927396] sd 0:0:0:0: [sda] Write Protect is off [64822.927402] sd 0:0:0:0: [sda] Mode Sense: 68 00 00 08 [64822.927404] sd 0:0:0:0: [sda] Assuming drive cache: write through [64822.929250] sd 0:0:0:0: [sda] 1941441 4096-byte hardware sectors (7952 MB) [64822.929889] sd 0:0:0:0: [sda] Write Protect is off [64822.929894] sd 0:0:0:0: [sda] Mode Sense: 68 00 00 08 [64822.929897] sd 0:0:0:0: [sda] Assuming drive cache: write through [64822.929900] sda: sda1 [64822.931560] sd 0:0:0:0: [sda] Attached SCSI removable disk [64827.615005] FAT: utf8 is not a recommended IO charset for FAT filesystems, filesystem will be case sensitive! [EMAIL PROTECTED]:~$ mount | grep ipod /dev/ipod on /media/IPOD type vfat (rw,nosuid,nodev,noatime,uhelper=hal,flush,uid=1000,shortname=lower) [EMAIL PROTECTED]:~$ pmount Printing mounted removable devices: /dev/ipod on /media/IPOD type vfat (rw,nosuid,nodev,noatime,uid=1000,fmask=0022,dmask=0022,codepage=cp437,iocharset=utf8,flush) To get a short help, run pmount -h [EMAIL PROTECTED]:~$ I still need to fix it everytime I attach a device by running: pumount /dev/ipod pmount -c iso8859-1 /dev/ipod /media/IPOD Then I get it mounted with the correct iocharset option AND the utf8 flag: [EMAIL PROTECTED]:~$ mount | grep ipod /dev/ipod on /media/IPOD type vfat (rw,noexec,nosuid,nodev,quiet,shortname=mixed,uid=1000,gid=1000,umask=077,fmask=0177,dmask=0077,utf8,iocharset=iso8859-1) [EMAIL PROTECTED]:~$ pmount Printing mounted removable devices: /dev/ipod on /media/IPOD type vfat (rw,nosuid,nodev,noexec,uid=1000,gid=1000,fmask=0177,dmask=0077,codepage=cp437,iocharset=iso8859-1,shortname=mixed,quiet,utf8) To get a short help, run pmount -h [EMAIL PROTECTED]:~$ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#500540: kdebase: automounting vfat (partialy) case sensitive due to utf8 is plain wrong and dangerous
New version of pmount package was uploaded (and unblocked) recently. It deals with the very same issue. Vincent Fourmond on the debian-release list points to this bug report, see: http://lists.debian.org/debian-release/2008/10/msg00793.html -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#500540: kdebase: automounting vfat (partialy) case sensitive due to utf8 is plain wrong and dangerous
On Fri, Oct 10, 2008 at 11:31:27PM +0300, Teemu Likonen wrote: Heinrich Langos [EMAIL PROTECTED] writes: Now lets try again with more sane vfat options: # mount | grep vfat /dev/sda1 on /mnt type vfat (rw,nosuid,nodev,noatime,uhelper=hal,flush,uid=1000,shortname=lower,check=relaxed,codepage=850,iocharset=iso8859-1) shortname=lower should be used but don't take my word for it. shortname=mixed works nicely with utf8 flag, and command touch test Test teSt touches the same file three times. The main question was if it preserves the case if you do touch TeSt test TEST In my setting (shortname=lower) the resulting file was test. In your case it should be TeSt but opening test should also work. The problem with case sensitive mounting and short names is that the application doesn't know which shortname option was used. So if my program writes /media/ipod/foo.mp3 and you used shortname=win95 than the name on the filesystem will be FOO.MP3 When you mount that filesystem with iocharset=utf8 then my program WILL break as it is not be able to open the file it wrote. gnupod for example uses short filenames explicitly to avoid other problems and DOES definetly break in such a situation. I've read of other music management software that does the same thing. Long filenames are a different beast altogether... And who came up with the idea to mount vfat with utf8 anyway? It was never designed to take short utf8 names. Those are strictly 8.3 and if you try to stick utf8 characters in there, you get all kinds of length checking problems. Long names on vfat are stored in unicode anyway. So whats the big gain here? For the sake of squeezing utf8 into places where it never was ment to be, we get messed up filesystems? I admit that some of the ideas may have come from me. I have described one aspect of this issue in the kernel bug #417324: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=417324 I'm pretty confused with all these iocharset, codepage and utf8 flags but I'm certain that in Debian Etch (and its default locale settings [UTF-8] and kernel settings) filenames get converted totally wrong. Long filenames in FAT filesystem are in the form we call UTF-16 today. In default Etch system FAT's UTF-16 filenames get converted to ISO-8859-1 if the filesystem is not mounted with utf8 flag. The other direction is so that Etch's UTF-8 filenames are assumed to be in ISO-8859-1 and, since it's a single-byte encoding, every byte (even in a UTF-8 multibyte character) gets converted separately to UTF-16. This produces complete garbage of course. KDE is nice enough to use utf8 flag but someone reported that Gnome does not (or at least did not) mount with this flag. Thus it produces filenames which are unreadable in other systems (including MS Windows). I guess the change in kernel settings made you see this issue after upgrading from Etch to Lenny. The option CONFIG_FAT_DEFAULT_IOCHARSET was changed from iso-8859-1 to utf8. Ok, so in order to fix a gnome bug you broke everything else? :-) As far as I have seen in archives and related bug reports the blame for this problem gets shifted around from KDE to pmount to the kernel itself and all the way back. Everybody happily points fingers at the others. This seems to be pretty complicated. We have to make VFAT/UTF-16 -- Debian/UTF-8 conversion work somehow, and in Etch it does not work (except when KDE does the mounting). In Lenny the conversion currently works by default with just mount without any options; this is because the change in the kernel settings. But then there's the issue you reported... :-( In my experience shortname=mixed works nicely without character case problems. Please read this: http://www.nslu2-linux.org/wiki/HowTo/MountFATFileSystems If you need a short explaination of ther vfat mount options codepage, iocharset and utf8. Let me Quote: When the utf8 flag is specified along with iocharset the iocharset value only controls the character case handling - it has no effect on the encoding of the UNICODE characters as this will always use UTF-8. So, according to it the right thing would be to select the iocharset according to your language AND specifying the utf8 flag. So in my case mount ... iocharset=iso8859-1,utf8 If gnome doesn't do it right, please go ahead and fix gnome but leave the kernel default iocharset as it was. Unfortunately the right thing can't be done by default: Let me quote once more: The utf8 flag cannot be specified by default - it must be given as an explicit argument to mount I am not sure if that is still the case with the current kernel. Maybe somebody grew tired of this limitation and fixed it. Could you check that? I'm drowning in work... could you also add a bts reference to 417324 to show the dependency and reopen that bug? -henrik (Using Debian since buzz.) Wow, I'm from the Woody/Sarge generation.
Bug#500540: kdebase: automounting vfat (partialy) case sensitive due to utf8 is plain wrong and dangerous
Heinrich Langos [EMAIL PROTECTED] writes: Now lets try again with more sane vfat options: # mount | grep vfat /dev/sda1 on /mnt type vfat (rw,nosuid,nodev,noatime,uhelper=hal,flush,uid=1000,shortname=lower,check=relaxed,codepage=850,iocharset=iso8859-1) As you see it is not perfect as the TEST file gets only created as test. My guess is that shortname=mixed instead of shortname=lower should be used but don't take my word for it. shortname=mixed works nicely with utf8 flag, and command touch test Test teSt touches the same file three times. And who came up with the idea to mount vfat with utf8 anyway? It was never designed to take short utf8 names. Those are strictly 8.3 and if you try to stick utf8 characters in there, you get all kinds of length checking problems. Long names on vfat are stored in unicode anyway. So whats the big gain here? For the sake of squeezing utf8 into places where it never was ment to be, we get messed up filesystems? I admit that some of the ideas may have come from me. I have described one aspect of this issue in the kernel bug #417324: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=417324 I'm pretty confused with all these iocharset, codepage and utf8 flags but I'm certain that in Debian Etch (and its default locale settings [UTF-8] and kernel settings) filenames get converted totally wrong. Long filenames in FAT filesystem are in the form we call UTF-16 today. In default Etch system FAT's UTF-16 filenames get converted to ISO-8859-1 if the filesystem is not mounted with utf8 flag. The other direction is so that Etch's UTF-8 filenames are assumed to be in ISO-8859-1 and, since it's a single-byte encoding, every byte (even in a UTF-8 multibyte character) gets converted separately to UTF-16. This produces complete garbage of course. KDE is nice enough to use utf8 flag but someone reported that Gnome does not (or at least did not) mount with this flag. Thus it produces filenames which are unreadable in other systems (including MS Windows). I guess the change in kernel settings made you see this issue after upgrading from Etch to Lenny. The option CONFIG_FAT_DEFAULT_IOCHARSET was changed from iso-8859-1 to utf8. As far as I have seen in archives and related bug reports the blame for this problem gets shifted around from KDE to pmount to the kernel itself and all the way back. Everybody happily points fingers at the others. This seems to be pretty complicated. We have to make VFAT/UTF-16 -- Debian/UTF-8 conversion work somehow, and in Etch it does not work (except when KDE does the mounting). In Lenny the conversion currently works by default with just mount without any options; this is because the change in the kernel settings. But then there's the issue you reported... :-( In my experience shortname=mixed works nicely without character case problems. -henrik (Using Debian since buzz.) Wow, I'm from the Woody/Sarge generation. :-) -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#500540: kdebase: automounting vfat (partialy) case sensitive due to utf8 is plain wrong and dangerous
Package: kdebase Version: 4:3.5.9.dfsg.1-5 Severity: grave Justification: causes non-serious data loss I just switched from etch to lenny and now gnupod warns me about the case sensitive filesystem on my iPod: $ gnupod_check.pl gnupod_check.pl Version 0.99.8 (C) Adrian Ulrich /usr/local/bin/gnupod_check.pl: Warning: /media/IPOD is mounted case sensitive, that's bad: FAT32-iPods should be mounted case in-sensitive! (try 'mount ... -o check=relaxed') Pass 1: Checking Files in the GNUtunesDB.xml... Pass 2: Checking Files on the iPod... ..finished ... The filesystem gets automounted by KDE's mediamanager and never caused a problem when I used Etch. The kernel also complains: [490945.600269] usb 4-3: new high speed USB device using ehci_hcd and address 23 [490945.733907] usb 4-3: configuration #1 chosen from 2 choices [490945.761374] scsi18 : SCSI emulation for USB Mass Storage devices [490945.762123] usb 4-3: New USB device found, idVendor=05ac, idProduct=1262 [490945.762138] usb 4-3: New USB device strings: Mfr=1, Product=2, SerialNumber=3 [490945.762147] usb 4-3: Product: iPod [490945.762153] usb 4-3: Manufacturer: Apple Inc. [490945.762159] usb 4-3: SerialNumber: 000A27001B256A33 [490945.762731] usb-storage: device found at 23 [490945.762742] usb-storage: waiting for device to settle before scanning [490950.760450] usb-storage: device scan complete [490950.776262] scsi 18:0:0:0: Direct-Access AppleiPod 1.62 PQ: 0 ANSI: 0 [490950.824634] sd 18:0:0:0: [sda] 1941441 4096-byte hardware sectors (7952 MB) [490950.825383] sd 18:0:0:0: [sda] Write Protect is off [490950.825397] sd 18:0:0:0: [sda] Mode Sense: 68 00 00 08 [490950.825404] sd 18:0:0:0: [sda] Assuming drive cache: write through [490950.827876] sd 18:0:0:0: [sda] 1941441 4096-byte hardware sectors (7952 MB) [490950.828416] sd 18:0:0:0: [sda] Write Protect is off [490950.828429] sd 18:0:0:0: [sda] Mode Sense: 68 00 00 08 [490950.828436] sd 18:0:0:0: [sda] Assuming drive cache: write through [490950.828443] sda: sda1 [490950.830215] sd 18:0:0:0: [sda] Attached SCSI removable disk [490951.932053] FAT: utf8 is not a recommended IO charset for FAT filesystems, filesystem will be case sensitive! And it is damn right to complain! So don't give me any of the should be ignored comments that you might have googled here: http://www.linuxfromscratch.org/lfs/view/development/chapter08/fstab.html Now lets see the effects of case sensitive vfat, shall we?: # mount | grep vfat /dev/sda1 on /media/IPOD type vfat (rw,nosuid,nodev,noatime,uhelper=hal,flush,uid=1000,utf8,shortname=lower) # cd /media/IPOD /media/IPOD# touch TEST TESt TEsT touch: cannot touch `TESt': File exists touch: cannot touch `TEsT': File exists /media/IPOD# touch TESTDIRECTORY TEStDIRECTORY TEsTDIRECTORY /media/IPOD# ls -lsa TE* 0 -rwxr-xr-x 1 hlangos root 0 2008-09-28 15:33 TEsTDIRECTORY 0 -rwxr-xr-x 1 hlangos root 0 2008-09-28 15:33 TEStDIRECTORY 0 -rwxr-xr-x 1 hlangos root 0 2008-09-28 15:33 TESTDIRECTORY /media/IPOD# ls -lsa te* ls: cannot access test: No such file or directory /media/IPOD# ls -lsa ls: cannot access test: No such file or directory total 32 4 drwxr-xr-x 8 hlangos root 4096 2008-09-28 15:33 . 4 drwxr-xr-x 4 rootroot 4096 2008-09-28 15:31 .. 4 drwxr-xr-x 2 hlangos root 4096 2000-02-09 07:06 Calendars 4 drwxr-xr-x 2 hlangos root 4096 2008-04-09 15:23 Contacts 4 drwxr-xr-x 11 hlangos root 4096 2008-02-13 00:01 iPod_Control 0 -rwxr-xr-x 1 hlangos root0 2000-02-09 07:06 .metadata_never_index 4 drwxr-xr-x 2 hlangos root 4096 2008-04-09 15:25 Notes 4 drwxr-xr-x 3 hlangos root 4096 2008-01-15 23:08 Photos 4 drwxr-xr-x 2 hlangos root 4096 2000-02-09 07:06 Recordings ? -? ? ? ? ?? test 0 -rwxr-xr-x 1 hlangos root0 2008-09-28 15:33 TEsTDIRECTORY 0 -rwxr-xr-x 1 hlangos root0 2008-09-28 15:33 TEStDIRECTORY 0 -rwxr-xr-x 1 hlangos root0 2008-09-28 15:33 TESTDIRECTORY /media/IPOD# That is FUBAR ! 1. touch TEST TESt TEsT should not complain about existing files. 2. touch TESTDIRECTORY TEStDIRECTORY TEsTDIRECTORY should not create more than one file. 3. And WTF happend to test anaway? BTW: If you try to delete some of those testdirectory files you may run into more of those nasty questionmarks. I didn't dare to continue working on a filesystem in such a state but to me this looks like a corrupted file system. Hence severity grave. Now lets try again with more sane vfat options: # mount | grep vfat /dev/sda1 on /mnt type vfat (rw,nosuid,nodev,noatime,uhelper=hal,flush,uid=1000,shortname=lower,check=relaxed,codepage=850,iocharset=iso8859-1) # cd /mnt /mnt# touch TEST TESt TEsT /mnt# touch TESTDIRECTORY TEStDIRECTORY TEsTDIRECTORY /mnt# ls -lsa TE* 0 -rwxr-xr-x 1 hlangos root 0 2008-09-28 15:41 TESTDIRECTORY /mnt# ls -lsa total 32 4 drwxr-xr-x 8 hlangos root 4096 2008-09-28 15:41 . 4
